Data compression method, data decompression method, computing device, storage medium and program product

Abstract

Provided in the embodiments of the present disclosure are a data compression method, a data decompression method, a computing device, a computer storage medium and a computer program product. The data compression method comprises: determining data to be compressed that has been subjected to character coding according to a unicode coding format; according to a byte order, reading character data from the data to be compressed, and determining a first dictionary or a second dictionary hit by the character data; searching for a coding sub-space corresponding to the first dictionary or the second dictionary, and determining coded values, in corresponding coding sub-spaces, of dictionary data hit by the character data, so as to obtain a coded-value set corresponding to the data to be compressed; and on the basis of the coded-value set and the first dictionary, obtaining compressed data.

Description

[0001] Data compression method, data decompression method, computing device, storage medium and program product technical field

[0002]

[0001] The present disclosure relates to the field of data processing technology, and more particularly to a data compression method, a data decompression method, a computing device, a computer storage medium, and a computer program product.

[0003]

[0002] With the rapid development of information technology and communication technology, the amount of data generated is increasing day by day. Storing or transmitting a large amount of data requires a large cost.

[0004]

[0003] Data compression is an effective way to reduce data storage costs and improve data transmission performance. Data compression can convert the data to be compressed into data with a smaller data volume. The original data can also be obtained by decompressing the compressed data.

[0005]

[0004] Therefore, how to provide a data compression method with high compression performance becomes a technical problem that needs to be solved urgently.

[0006]

[0005] The embodiments of the present disclosure provide a data compression method, a data decompression method, a computing device, a computer storage medium, and a computer program product, which improve data compression performance.

[0007]

[0006] In a first aspect, an embodiment of the present disclosure provides a data compression method, comprising: determining data to be compressed that is character-encoded according to a Unicode encoding format; reading character data from the data to be compressed in byte order, and determining a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; searching a coding subspace corresponding to the first dictionary or the second dictionary, respectively, and determining a coding value of the dictionary data matched by the character data in the corresponding coding subspace, to obtain a set of coding value groups corresponding to the data to be compressed; and obtaining compressed data based on the coding value groups and the first dictionary.

[0008]

[0007] In a second aspect, an embodiment of the present disclosure provides a data decompression method, comprising: obtaining compressed data; obtaining a first dictionary and a coding value group from the compressed data; wherein the coding value group is constructed by searching the coding subspace corresponding to the first dictionary or the second dictionary respectively, and according to the coding values ​​of the dictionary data hit by the character data in the data to be compressed in the corresponding coding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; searching the coding subspace corresponding to the first dictionary or the second dictionary respectively, determining the character data in the corresponding coding subspace of the dictionary data hit by the coding value group, to obtain the decompressed data corresponding to the coding value group.

[0009]

[0008] In a third aspect, an embodiment of the present disclosure provides a data compression device, comprising: a data determination module, configured to determine data to be compressed that is character-encoded according to a Unicode encoding format; a data reading module, configured to read character data from the data to be compressed in byte order, and determine a first dictionary or a second dictionary in which the character data hits; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; a first search module, configured to search the encoding subspace corresponding to the first dictionary or the second dictionary, respectively, and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, so as to obtain a set of encoding value groups corresponding to the data to be compressed; and a first processing module, configured to obtain compressed data based on the encoding value groups and the first dictionary.

[0010]

[0009] In a fourth aspect, an embodiment of the present disclosure provides a data decompression device, comprising: a data acquisition module for acquiring compressed data; a second processing module for obtaining a first dictionary and a coding value group from the compressed data; wherein the coding value group is constructed by searching the coding subspace corresponding to the first dictionary or the second dictionary respectively, and according to the coding values ​​of the dictionary data hit by the character data in the data to be compressed in the corresponding coding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; a second search module is used to search the coding subspace corresponding to the first dictionary or the second dictionary respectively, and determine the character data in the corresponding coding subspace of the dictionary data hit by the coding value group, so as to obtain the decompressed data corresponding to the coding value group.

[0011]

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the data compression method provided by the embodiment of the present disclosure, or to implement the data decompression method provided by the embodiment of the present disclosure.

[0012]

[0011] In a sixth aspect, an embodiment of the present disclosure provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements the data compression method provided by the embodiment of the present disclosure, or implements the data decompression method provided by the embodiment of the present disclosure.

[0013]

[0012] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program code. When the computer program code is executed by a computer, it implements the data compression method provided by the embodiment of the present disclosure, or implements the data decompression method provided by the embodiment of the present disclosure.

[0014]

[0013] The embodiment of the present disclosure is directed to data to be compressed that is character-encoded according to a Unicode encoding format; character data is read from the data to be compressed in byte order, and a first dictionary or a second dictionary in which the character data hits is determined; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; a coding subspace corresponding to the first dictionary or the second dictionary is searched, and the coding values ​​of the dictionary data hit by the character data in the corresponding coding subspace are determined to obtain a group of coding value groups corresponding to the data to be compressed; the coding value groups and the first dictionary are serialized to obtain compressed data. The technical solution of the embodiment of the present disclosure realizes lossless compression processing of the data to be compressed that is character-encoded according to a Unicode encoding format, and combines the first dictionary constructed based on the data to be compressed and the second dictionary constructed based on the text corpus to ensure the compression rate and compression speed of data compression, thereby improving compression performance.

[0015]

[0014] These and other aspects of the present disclosure will become more readily apparent in the following description of the embodiments.

[0016]

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017]

[0016] FIG1 schematically shows a flow chart of a data compression method provided by one embodiment of the present disclosure;

[0018] FIG2 schematically shows a flow chart of a data compression method provided in a practical application according to an embodiment of the present disclosure;

[0019]

[0018] FIG3 schematically shows a flow chart of an embodiment of a data decompression method provided by an embodiment of the present disclosure;

[0020]

[0019] FIG4 schematically shows a flow chart of a data compression and decompression process in a practical application according to an embodiment of the present disclosure;

[0021]

[0020] FIG5 is a schematic structural diagram of an embodiment of a data compression device provided in an embodiment of the present disclosure;

[0022]

[0021] FIG6 is a schematic structural diagram of an embodiment of a data decompression device provided in an embodiment of the present disclosure;

[0023]

[0022] FIG7 is a schematic diagram of a structure of an embodiment of a computing device provided by an embodiment of the present disclosure.

[0024]

[0023] In order to enable people skilled in the art to better understand the solution of the present disclosure, the technical solution in the embodiment of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiment of the present disclosure.

[0025] Some of the processes described in the specification and claims of this disclosure and the accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may be executed in a different order than the order in which they appear herein or in parallel. Operation sequence numbers, such as 101 and 102, are merely used to distinguish between different operations and do not represent any specific order of execution. Furthermore, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that terms such as "first" and "second" herein are used to distinguish between different messages, devices, modules, etc., and do not represent a sequential order or limit the "first" and "second" to different types.

[0026]

[0025] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0027] During the process of developing the present disclosure, the inventors discovered that conventional compression methods, such as LZ4 (a lossless compression algorithm), Zstd (a fast lossless compression algorithm), and Gzip (a file compression algorithm), have poor compression ratios and compression / decompression speeds for character data encoded in Unicode. Zlib and LZ4 are block compression technologies that compress data blocks of a set size. Because they are unaware of the characteristics of the compressed data (such as data type and value range), they cannot achieve maximum compression.

[0028]

[0027] Therefore, in order to improve the compression performance, the inventor proposed the technical solution of the embodiment of the present disclosure after a series of studies.

[0029] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings. It is apparent that the described embodiments are only a portion of the embodiments of the present disclosure, and are not intended to be exhaustive. All other embodiments derived by those skilled in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0030]

[0029] FIG1 schematically shows a flow chart of a data compression method provided by an embodiment of the present disclosure. As shown in FIG1, the data compression method may specifically include the following steps.

[0031]

[0030] 101: Determine data to be compressed that is character-encoded according to a Unicode encoding format.

[0032] Unicode is an international standard that aims to provide a unified encoding scheme for almost all text in the world. The Unicode encoding format assigns a unique numerical value to each character in the world so that computers can understand and process these characters.

[0033]

[0032] Unicode is usually represented by "U+" followed by the hexadecimal number of the character. For example, the Unicode code point for the Latin letter A is U+004L

[0034]

[0033] In order to facilitate transmission, different encoding formats can be used to encode Unicode characters, such as UTF-8, UTF-16 and UTF-32, etc. These encoding formats encode the characters in Unicode into byte sequences for easy storage and transmission.

[0035]

[0034] UTF-8 is a variable length encoding format used to represent characters in the Unicode character set. Each Unicode character can be encoded as 1 to 4 bytes in UTF-8.

[0036]

[0035] UTF-16 is a variable-length encoding format that uses 2 or 4 bytes to represent Unicode characters. It is suitable for processing a large number of non-ASCII (American Standard Code for Information Interchange, a character encoding standard based on the Latin alphabet) characters.

[0036] UTF-32 is a fixed-length encoding format. Each Unicode character can be encoded as a fixed 4-byte representation in UTF-16, and each character occupies the same storage space.

[0037]

[0037] 102: Read character data from the data to be compressed in byte order, and determine the first dictionary or the second dictionary in which the character data hits; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on the text corpus.

[0038]

[0038] When character data is read from the data to be compressed in byte order, the characters may be read one by one according to the byte representation of the data.

[0039]

[0039] For example, in the data to be compressed in the UTF-8 encoding format, one character may occupy 1 to 4 bytes.

[0040] In a possible implementation of the present disclosure, reading character data from the data to be compressed in byte order can be implemented as follows:

[0041] First, an index variable is initialized to 0 to mark the byte position currently read. Then the data to be compressed is read in a loop until the end of the data to be compressed is reached.

[0042]

[0042] The following steps can be performed in each cycle.

[0043] a. Read the byte at the current index position.

[0044] b. Determine whether the high bit of the byte is 0. If so, it indicates that the byte is the starting byte of a single-byte character. Decode the byte into a character.

[0045] c. If the high bit of the byte begins with 110 (i.e., Ob110xxxxx), it indicates that the byte is the starting byte of a double-byte character. This byte and the next byte are decoded together into one character.

[0046]

[0046] d. If the high bit of the byte begins with 1110 (i.e., Obll10xxxx), it indicates that the byte is the starting byte of a three-byte character. This byte and the subsequent two bytes are decoded together into one character.

[0047] e. If the high bit of the byte begins with 11110 (i.e., ObllllOxxx), it indicates that the byte is the starting byte of a four-byte character. This byte and the subsequent three bytes are decoded together into one character.

[0048] f. process character data according to the decoding result.

[0049]

[0049] g. Update the index variable to point to the next byte position that needs to be read.

[0050]

[0050] After the above-mentioned loop steps are completed, a plurality of character data are sequentially read from the data to be compressed in byte order to obtain a plurality of character data.

[0051]

[0051] For each character data read, determine whether the character data hits the first dictionary or the second dictionary.

[0052] In an embodiment of the present disclosure, the first dictionary can be pre-constructed based on the data to be compressed. When constructing the first dictionary, repeated patterns, common items, or other features in the data to be compressed can be analyzed, and this information can be recorded in the first dictionary. By extracting repeated items or frequently occurring segments in the data to be compressed to construct the first dictionary, subsequent compression processing can be facilitated, achieving efficient data compression.

[0053] The second dictionary can be constructed based on a text corpus. A large-scale text corpus can be used as an input data source to construct the second dictionary, and the text in the text corpus is statistically analyzed and processed to extract common words, phrases, etc. and record them in the second dictionary.

[0054] For each character data read, the character data may be matched against dictionary data pre-written in the first dictionary and the second dictionary, respectively, and whether the character data matches the first dictionary or the second dictionary is determined based on the matching results. For example, if the first dictionary data identical to the first character data is retrieved from the first dictionary, the first character data may be determined to match the first dictionary; and if the second dictionary data identical to the second character data is retrieved from the second dictionary, the second character data may be determined to match the second dictionary. Optionally, the dictionary data in the first dictionary and the second dictionary may be encoded using the same encoding format as the character data.

[0055]

[0055] 103: Search the encoding subspace corresponding to the first dictionary or the second dictionary respectively, determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, and obtain a group of encoding value groups corresponding to the data to be compressed.

[0056]

[0056] The coding subspaces corresponding to the first dictionary and the second dictionary can be obtained by segmenting the target coding space; optionally, the dictionary data in the first dictionary and the second dictionary can be sorted from large to small according to their importance, and the dictionary data with greater importance corresponds to smaller coding values.

[0057]

[0057] In an embodiment of the present disclosure, the target code space may be an integer range consisting of zero to a specified value, and may be determined in combination with the number of characters in the first dictionary, the second dictionary, and ASCII characters. In an actual application, the target code space may be, for example, 0-16383, but the present disclosure is not limited thereto.

[0058] The first dictionary typically contains repeated patterns or common items in the data to be compressed, and the second dictionary typically contains a wide range of language information. Therefore, the code subspaces corresponding to the first and second dictionaries can be constructed by selecting different code ranges from the target code space. In this way, the code subspaces corresponding to the first and second dictionaries are subsets of the target code space.

[0059]

[0059] Optionally, the target coding space may be segmented according to different rules or features to obtain multiple different coding subspaces. The first dictionary may correspond to at least one coding subspace, and the second dictionary may correspond to at least one coding subspace.

[0060] Each dictionary data in the first dictionary and the second dictionary corresponds to a code value in the code subspace. After the code subspaces corresponding to the first dictionary and the second dictionary are determined from the target code space, a mapping relationship between the dictionary data in the first dictionary and the second dictionary and the corresponding code values ​​can be established. The dictionary data in the first dictionary and the second dictionary correspond to the code values ​​in their respective corresponding code subspaces.

[0061] In data compression, it is generally desirable to map the data to be compressed to a smaller code value to reduce the size of the data representation. Therefore, in the first dictionary and the second dictionary, the dictionary data can be sorted from most important to least important, with the code value corresponding to the most important dictionary data being smaller. The importance of the dictionary data can be related to the frequency of occurrence of the dictionary data, with the more frequently occurring dictionary data being of greater importance.

[0062]

[0062] In another possible implementation of the present disclosure, the importance of the dictionary data may also be related to the number of bytes that can be saved by the dictionary data compared to the data to be compressed, wherein the more bytes saved, the greater the importance. Accordingly, the dictionary data with greater importance may correspond to a higher encoding value. The number of bytes saved may be determined by the difference between the number of bytes of the dictionary data and the number of bytes of the character data corresponding to the dictionary data.

[0063]

[0063] In one embodiment of the present disclosure, the importance of dictionary data of the first dictionary may refer to the number of bytes saved by compression, that is, the difference between the number of bytes of the dictionary data and the number of bytes of the character data corresponding to the dictionary data; the importance of dictionary data of the second dictionary may refer to the frequency of occurrence of the dictionary data.

[0064]

[0064] Of course, the calculation method of the importance degree can also be implemented in other ways, which will be described in detail in the following embodiments.

[0065]

[0065] After reading the character data of the data to be compressed, the encoding value corresponding to the dictionary data can be found in at least one sub-encoding space of the target encoding space through the first dictionary or the second dictionary hit by each character data, thereby obtaining a group of encoding value groups corresponding to the character data contained in the data to be compressed.

[0066]

[0066] 104. Obtain compressed data based on the encoding value group and the first dictionary.

[0067] To facilitate storage or transmission, after obtaining a set of encoded value arrays corresponding to the character data contained in the data to be compressed, the encoded value arrays may be serialized to obtain compressed data. Serialization may be the process of converting a data structure or object into a format that can be stored or transmitted. There are many serialization technologies used for serialization, such as JSON (JavaScript Object Notation, a lightweight data exchange format), Java native serialization, and Varint (a variable-length integer encoding) serialization, which are not limited in this disclosure. By serializing the encoded arrays, a compact representation of the data can be achieved, storage space usage can be reduced, and data transmission and processing can be facilitated.

[0068]

[0068] In some embodiments of the present disclosure, after the encoded value group and the first dictionary are serialized, the generated serialized data can be further compressed using a traditional compression method, such as gzip, zlib, Lz4, Zstd, etc., to further improve the compression effect.

[0069]

[0069] The compressed data finally obtained can be stored in a storage medium, and can also be transmitted to other places through a network to achieve effective management and transmission of data.

[0070]

[0070] In some embodiments, the method may further include: when the character data does not hit the first dictionary and the second dictionary, determining the ASCII character that the character data hits; searching the encoding subspace corresponding to the ASCII character, and determining the encoding value of the dictionary data that the character data hits in the corresponding encoding subspace; wherein, the encoding subspace corresponding to the ASCII character is obtained by segmenting the target encoding space; the encoding value corresponding to at least part of the dictionary data in the first dictionary and the second dictionary is 1, which is the encoding value corresponding to the ASCII character.

[0071]

[0071] In actual application, the dictionary data contained in the first dictionary and the second dictionary may not contain all the character data in the data to be compressed.

[0072]

[0072] When the dictionary data in the first dictionary and the second dictionary cannot contain all the character data in the data to be compressed, that is, when the character data cannot match any one of the first dictionary and the second dictionary, the ASCII characters matched by the character data can be determined.

[0073]

[0073] ASCII characters are a basic character encoding standard used to represent English letters, numbers, punctuation marks, and some control characters. The ASCII character set contains 128 characters, each of which is represented by a 7-bit binary value ranging from 0 to 127.

[0074]

[0074] In an embodiment of the present disclosure, the encoding subspace corresponding to ASCII characters can be constructed by selecting a portion of the encoding range from the target encoding space.

[0075]

[0075] After selecting a portion of the code range for the ASCII character from the target code space, a mapping relationship between the ASCII character and the corresponding code value can be constructed. The ASCII character corresponds to the code value in its corresponding code subspace.

[0076]

[0076] In addition, as another possible implementation, the method may further include: when the character data does not match the first dictionary and the second dictionary, using the character data as the encoding value.

[0077]

[0077] When the character data does not match the first dictionary and the second dictionary, in order to ensure the integrity of the data to be compressed and to retain as much information as possible in subsequent processing, the character data may be directly used as the encoding value.

[0078]

[0078] In a possible implementation of the present disclosure, if the character data does not match the first dictionary and the second dictionary, the character data can be directly used as the encoding value.

[0079]

[0079] In some embodiments, the first dictionary may be generated in the following manner: segmenting the data to be compressed; and constructing the first dictionary based on multiple character strings obtained by segmentation.

[0080]

[0080] In a possible implementation of the present disclosure, the data to be compressed may be segmented according to a preset rule to obtain multiple character strings.

[0081]

[0081] The preset rules may include, for example, segmentation according to spaces, punctuation marks or other separators.

[0082]

[0082] In some embodiments, constructing the first dictionary based on the multiple character strings obtained by segmentation may include: counting the occurrence frequencies of the multiple character strings in the data to be compressed; calculating the importance of the multiple character strings based on the occurrence frequencies and character lengths of the multiple character strings; and selecting a first number of character strings in descending order of importance to generate the first dictionary.

[0083]

[0083] First, the data to be compressed can be traversed, and the number of times each character string appears in the data to be compressed can be counted to obtain the frequency of occurrence of each character string.

[0084]

[0084] In a possible implementation of the present disclosure, the importance of each character string can be obtained by multiplying the frequency of occurrence of the character string by the character length.

[0085]

[0085] In addition, as another possible implementation, the importance of multiple character strings can be calculated according to the occurrence frequencies and character lengths corresponding to the multiple character strings as follows:

[0086]

[0086] The importance of the multiple character strings is calculated according to the occurrence frequencies, character lengths and predetermined compressed byte numbers corresponding to the multiple character strings.

[0087]

[0087] The importance of a string can be calculated using the following formula (1). score = freq * (len - code_widh) - (len + code_widh); (1)

[0088]

[0088] In which, score can represent the importance of the character string, freq can represent the frequency of occurrence, code_widh can represent the predetermined number of compressed bytes, the predetermined number of compressed bytes can be 1, and len can represent the character length.

[0089]

[0089] In some embodiments, the method may further include: querying the encoding subspace corresponding to the first dictionary in descending order of importance to determine first compressed byte numbers corresponding to the multiple character strings; determining first compression ratios corresponding to the multiple character strings based on the first compressed byte numbers corresponding to the multiple character strings; querying the encoding subspace corresponding to the second dictionary to determine second compressed byte numbers corresponding to the multiple character strings; determining second compression ratios corresponding to the multiple character strings based on the second compressed byte numbers corresponding to the multiple character strings; and deleting character strings having a first compression ratio greater than a second compression ratio from the multiple character strings.

[0090]

[0090] The first compressed byte number may refer to the storage space occupied by the encoding value corresponding to the character data in the first dictionary, that is, the character length; the second compressed byte number may refer to the storage space occupied by the encoding value corresponding to the character data in the second dictionary, that is, the character length.

[0091]

[0091] The first compression ratio can be calculated based on the first compressed byte number and the storage space occupied by the character data itself, that is, the character length;

[0092]

[0092] The second compression ratio can be calculated based on the second compressed byte number and the storage space occupied by the character data itself.

[0093]

[0093] The first compression ratio and the second compression ratio can be calculated using the following formula (2):

[0094] CR = uncompressed size; ( 2 )

[0095]

[0094] Wherein, CR may represent the compression ratio, compressed size may represent the number of compressed bytes, and uncompressed size may represent the character length of the character data.

[0096]

[0095] The first compression ratio is greater than the second compression ratio, indicating that the first compressed byte count is greater than the second compressed byte count, that is, the storage space occupied by the encoding value determined by the first dictionary for the character string is greater than the storage space occupied by the encoding value determined by the second dictionary, indicating that the compression performance of the character string compressed using the first dictionary is weaker than the compression performance of the character string compressed using the second dictionary. Therefore, the character string with the first compression ratio greater than the second compression ratio can be deleted from the multiple character strings without being added to the first dictionary.

[0097]

[0096] In some embodiments, selecting the first number of character strings in descending order of importance to generate the first dictionary may include: recalculating the importance of the multiple character strings based on the first compressed byte counts corresponding to the multiple character strings, combined with the occurrence frequencies and character lengths corresponding to the multiple character strings; and selecting the first number of character strings in descending order of the recalculated importance to generate the first dictionary.

[0098]

[0097] Since the first compressed byte number is the actual compressed byte number of the character string, the importance of multiple character strings can be recalculated based on the first compressed byte number.

[0099]

[0098] The importance of the string can be recalculated using the following formula (3). score = freq * (len - code_1) - (len + code_1); (3)

[0100]

[0099] Wherein, score_new may represent the importance of the recalculated character string, freq may represent the frequency of occurrence, and code_1W may represent the number of first compressed bytes.

[0101] After sorting the multiple character strings from largest to smallest according to the recalculated importance, a first number of character strings with higher importance may be selected from the sorted queue to form a first dictionary. The specific value of the first number may be flexibly set by those skilled in the art based on actual application requirements and is not specifically limited in the presently disclosed embodiments.

[0102]

[0101] In some embodiments, the segmentation process of the data to be compressed can be specifically implemented as follows: reading character data from the data to be compressed in byte order and saving the character data as a character string; querying the number of occurrences of the current character string; if the number of occurrences is greater than 1, accumulating the number of occurrences; if the number of occurrences is less than or equal to 1, merging the current character string with the next character string and accumulating the number of occurrences of the merged character string.

[0103]

[0102] The following is a specific example to schematically illustrate the specific implementation method of segmenting the compressed data. It should be noted that the following example is only used to help those skilled in the art understand the implementation method of the embodiment of the present disclosure, and does not make any improper limitation to the present disclosure.

[0104]

[0103] Assume that the data to be compressed may be "aababc".

[0105]

[0104] When the data to be compressed is segmented, the character data in the data to be compressed can be read sequentially according to the byte order. In this example, a, a, b, a, b, c... can be read sequentially.

[0106] When reading character data in byte order, when a is read, the number of occurrences of the string is 0, which is less than 1. Since a is the first character, the number of occurrences of a can be accumulated at this time, and the number of occurrences of a is recorded as 1, and the next string is read. The next string is a, and the number of occurrences of the string is greater than or equal to 1, so the number of occurrences of the string can be accumulated, and the number of occurrences of a is recorded as 2. Then the next string b is read, and the number of occurrences of b is less than 1. At this time, the current string b and the previously saved string a can be merged to generate a merged string ab, and the number of occurrences of the merged string can be accumulated. At this time, the number of occurrences of the merged string ab is 1. Then read the next string. If the next string is a, accumulate the number of occurrences of the string and record the number of occurrences of a as 3. Then read the next string b, merge string a and string b to generate a merged string ab, and accumulate the number of occurrences of the merged string ab, and record the number of occurrences of the merged string as 2. Continue to read the next string c. If the number of occurrences of c is less than 1, merge string c with the previous guaranteed string ab to generate a merged string abc, and accumulate the number of occurrences of the merged string to 1.

[0107]

[0106] After the above segmentation process, the data to be compressed aababc can be segmented into a3, ab2, and abc1, wherein a3 indicates that the number of times a appears in the data to be compressed is 3, ab2 indicates that the number of times a appears in the data to be compressed is 2, and abc indicates that the number of times abc appears in the data to be compressed is 1.

[0108]

[0107] In some embodiments, the method may further include: if the currently read character data is a separator, saving at least one character data separated by the separator as a character string.

[0109]

[0108] In the embodiment of the present disclosure, the separator may include special symbols such as spaces and punctuation marks, or a specific character may be set as a separator.

[0110]

[0110] In the above example, the segmentation results of the to-be-compressed data abc / ab may be abcl and abl. Of course, the above two methods of obtaining character strings may be used in combination. For example, assuming the to-be-compressed data abc / ab, the segmentation results may be: a2, bl, cl, abcl, ab2...

[0111] In some embodiments, the second dictionary may be generated in the following manner: counting the frequency of occurrence of each character in the text corpus; sorting each character in the text corpus in descending order of frequency of occurrence; and generating the second dictionary based on the sorting result.

[0111]

[0112] In one possible implementation of the present disclosure, a text corpus can be traversed to count the number of times each character appears in the text corpus to obtain the frequency of occurrence of each character. The counted frequency of occurrence of each character can then be sorted in descending order. Based on the sorting results, a second number of characters can be selected in descending order of frequency of occurrence to generate a second dictionary. The frequency of occurrence represents the importance. Specifically, the selected first number of characters can be added to the second dictionary, and their corresponding importance can be recorded. For example, the characters and frequency of occurrence can be stored as key-value pairs, or the characters and frequency of occurrence can be saved in an ordered list to obtain the second dictionary.

[0112]

[0113] A text corpus can refer to a collection of text data that can contain text from different sources, such as news articles, novels, web content, social media posts, etc.

[0113]

[0114] In one possible implementation of the present disclosure, a subtext corpus can be obtained by filtering from a text corpus, and then a second dictionary can be generated by counting the frequency of occurrence of each character in the subtext corpus. The filtering condition can, for example, be filtering text from the text corpus for texts in the same field as the data to be compressed.

[0114]

[0115] By screening the text corpus, text related to the field of the data to be compressed can be obtained, thereby increasing the relevance of the generated second dictionary to the data to be compressed and improving compression efficiency. In addition, text unrelated to the data to be compressed can be excluded, reducing the impact of noise on the generation of the second dictionary.

[0115]

[0116] Furthermore, to further improve the compression effect, in some embodiments, obtaining compressed data based on the first dictionary and the code value group can be specifically implemented as follows: determining N consecutive code values ​​in the code value group that meet a merging condition; combining the N code values ​​and merging header identification data to obtain a combined value; wherein the header identification data is used to indicate the number of bits corresponding to each of the N code values; and the merging condition is that the number of bytes occupied by the combined code value is less than the sum of the number of bytes occupied by the N code values; and serializing the combined value, the remaining code values ​​in the code value group that do not meet the merging condition, and the first dictionary to obtain compressed data.

[0116]

[0117] By searching for N consecutive (N is 2 or 3) coded values ​​that meet the merging condition, a merged value is obtained. Since the number of bytes occupied by the merged value is less than the sum of the number of bytes occupied by the N coded values, the compression performance can be further improved.

[0117]

[0118] In some embodiments, when the Varint serialization method is used for serialization, the merging condition may be specifically: the number of bits of the N encoded values ​​is greater than or equal to 8, and the number of bits of the merged value is M times 7, where M is greater than 2.

[0118]

[0119] Varint is a variable-length integer encoding method. The basic rule of Varint encoding is that an integer value can be represented using 7 consecutive binary bits. Since 7 bits of binary data can represent the maximum value of 127, an integer less than 128 only requires one byte. Integers between 128 and 16383 require two bytes. Integers greater than 16383 require at least three bytes after Varint encoding. The most significant bit of the first byte is 1, and the remaining 7 bits plus the 7 bits of the second byte together constitute the binary representation of the integer.

[0119]

[0120] Therefore, for an 8-bit binary data, it needs to occupy 2 bytes after using Varint encoding; for a 21-bit binary data, it needs to occupy at least 3 bytes after using Varint encoding; similarly, for a 28-bit binary data, it needs to occupy at least 4 bytes after using Varint encoding.

[0120]

[0121] Therefore, when the number of bits of the N encoded values ​​is greater than or equal to 8, each encoded value occupies 2 bytes after Varint encoding, and the N encoded values ​​will occupy 2N bytes. The combined value obtained after the merging process occupies M bytes after Varint encoding, and 2N is greater than M. Optionally, when N is greater than or equal to 2, when N is 2, M can be 3; when N is 3, M can be 4, and so on.

[0121]

[122] In order to facilitate the merging operation, in some embodiments, determining the N consecutive code values ​​in the code value group that meet the merging condition can be specifically implemented as follows: querying the merged coding table corresponding to the N code values, and determining the N consecutive code values ​​in the code value group that hit any group of bit combinations in the merged coding table; according to the embodiment of the present disclosure, combining the N code values ​​and merging the header identification data to obtain the merged value can be specifically implemented as follows: searching the merged coding table for the header identification data corresponding to the bit combination; combining the N code values ​​and merging the header identification data according to the merging method in the merged coding table to obtain the merged value.

[0122]

[123] In order to further reduce the storage space occupied by the coded value group, at least one group of values ​​in the coded value group can be merged.

[0123]

[124] In which, the coded values ​​can be merged when the coded value group contains N consecutive coded values ​​that hit any group of bit combinations in the merged coding table.

[0124]

[125] In an embodiment of the present disclosure, a merged coding table may be maintained in advance, and the merged coding table records at least one set of bit position combinations that meet the merge condition. When determining N code value groups that meet the merge condition in the code value group, the code value groups may be traversed in order, and N code values ​​may be selected in turn to determine whether the N code values ​​match any set of bit position combinations in the merged coding table. Each set of bit position combinations in the merged coding table provides the possible bit position combinations corresponding to N consecutive code values ​​in the order in which they appear in the data to be compressed.

[0125]

[126] For the N code values ​​that meet the merging conditions, the N code values ​​can be merged, and then the next group of N code values ​​can be read until the code value group is traversed.

[0126]

[127] In the merge coding table, the merge method corresponding to each arrangement order and the header identification data can also be recorded.

[0127]

[128] When merging N code values, the N code values ​​can be merged according to the merging method recorded in the merged code table to generate a merged code value. In addition, the header identification data corresponding to the hit bit combination can be obtained from the merged code table, and the header identification data and the code value generated by the merger can be merged to obtain a merged value.

[0128]

[129] In an example, for example, there are three consecutive coded values ​​code1, code2, code3, where the value of code1 is 10, which is represented as 00001010 in binary, the value of code2 is 7, which is represented as 00000111 in binary, and the value of code3 is 15, which is represented as 00001111 in binary. code1 occupies 8 bits, code2 occupies 8 bits, and code3 occupies 8 bits. The three coded values ​​are serialized using Varint serialization. Each coded value will occupy 2 bytes after Varint encoding. Assuming that the three consecutive coded values ​​meet the merging conditions, the three consecutive coded values ​​can be merged according to the merging method recorded in the merge coding table.

[0129]

[130] For example, if the merged side is code], shift left by 1 and add code3, we get

[0130]

[131] Specifically: Shift code1 left by 16 bits to get 00001010 00000000000000000; shift code2 left by 8 bits to get 00000000 00000111 000000000; directly use the original binary representation of codes 00000000 0000000000001111; add the above three results to get the combined result combined_code 00001010 00000111 00001111…

[0131]

[132] Furthermore, after obtaining the merge result, it is necessary to merge the header identification data Header based on the merge result. For example, if the header representation data can be

[0000] , the final merged result can be 00000000000000000 00000000 0000101000000111 00001111.

[133] In the above example, Code 1 is represented by 8 bits, Code 2 is represented by 8 bits, and Code 3 is represented by 8 bits. The merged coded header Header consists of a fixed 4 bits

[0000] , and the combined code combined_code consists of 24 bits 00001010 00000111 00001111. The final merged code is 00000000 00000000000000000 00001010 00000111 00001111 , which is 28 bits and occupies 4 bytes. Compared with the 6 bytes required for serializing the three encoded values ​​separately, the compression performance is further improved.

[0132]

[134] In a practical application, N can be 2, then M can be 3, and the combined value is 21 bits. Then the combined coding table corresponding to the two coding values ​​can be shown in Table 1 below: Table 1:

[0133]

[135] Among them, code 1 can represent the first code value of two consecutive code values, code 2 can represent the second code value, Header can represent header identification data, combined_code can represent the result of combining code 1 and code 2, that is, the combined value, and "<<" in Table 2 represents a bit shift operation, for example, code 1 « 8, means shifting code 1 to the left by 16. The specific implementation of the merging method can be found in the above description and will not be repeated here.

[0134]

[136] In another practical application, N can be 3, M can be 4, and the combined value is 28 bits. Then the combined coding table corresponding to the three coding values ​​can be shown in Table 2 below: Table 2:

[0135]

[0137] In which, code 1 may represent the first code value among three consecutive code values, code 2 may represent the second code value, code 3 may represent the third code value, Header may represent header identification data, and combined_code may represent a combined value.

[0136]

[0138] It should be noted that Table 1 and Table 2 only provide possible implementation methods of the embodiments of the present disclosure, and the present disclosure is not limited thereto.

[0137]

[0139] In some embodiments, the coding subspace can be generated as follows: dividing the target coding space into multiple coding subspaces; determining multiple data segments obtained by segmenting the first dictionary and the second dictionary respectively; and allocating coding subspaces to the multiple data segments from the multiple coding subspaces according to the principle of allocating dictionary data with greater importance to smaller coding values.

[0138]

[0140] The target coding space can represent the usable coding range. By dividing the target coding space into multiple coding subspaces, a certain range of coding values ​​can be assigned to each coding subspace.

[0139]

[0141] According to an embodiment of the present disclosure, the target coding space can be generated by the following operations: counting the number of characters included in the first dictionary, the second dictionary and the ASII characters; and generating the target coding space based on the number of characters.

[0140]

[0142] In the embodiments of the present disclosure, the entire coding space may be evenly divided into a plurality of coding subspaces as required, but the present invention is not limited thereto. The plurality of coding subspaces may also be unevenly divided according to different requirements.

[0141]

[0143] Since the dictionary data is arranged according to importance in the dictionary, optionally, the first coding subspace may be assigned to the first data segment in the first dictionary; the second coding subspace may be assigned to the first data segment in the second dictionary, and so on. Each dictionary data in the data segment corresponds to a coding value, and a greater importance corresponds to a smaller coding value.

[0142]

[0144] Optionally, the ASII character may be divided into multiple data segments, and encoding subspaces may be allocated to the multiple data segments in multiple encoding subspaces.

[0143]

[0145] To facilitate searching, a coding space mapping table can be established based on the above allocation results. By searching the coding space mapping table, the coding subspace corresponding to the first dictionary, the second dictionary, or the ASII character can be determined.

[0146] Table 3 below schematically illustrates a coding space mapping table provided by an embodiment of the present disclosure in an actual application. Assuming that the target coding space is 0-16383, the first dictionary can store 280 dictionary data and the second dictionary can store 8192 dictionary data. Table 3

[0144]

[0147] As shown in Table 3, the target coding space can be 0-16383. Assume that the target coding space can be divided into 8 coding subspaces, each coding subspace can be a subset of the target coding space.

[0145] In Table 3, the code segment number is used to identify different code subspaces. Taking the first code subspace 0-7 as an example, it corresponds to the data segments 1-8 of the first dictionary, that is, the first to eighth dictionary data. According to the principle that the greater the importance, the smaller the code value, the code value corresponding to the first dictionary data is 0, the code value corresponding to the second dictionary data is 1..., and so on, the code value corresponding to the eighth dictionary data is 8. Similarly, for each code subspace in Table 2, the code value corresponding to the dictionary data or ASCII character in the corresponding dictionary can be determined, and examples will not be repeated here.

[0146] In Table 3, the first encoding subspace, i.e., encoding subspace 0-7 corresponding to encoding segment number 0, is assigned to data segments 1-8 of the first dictionary; the second encoding subspace, i.e., encoding subspace 8-31 corresponding to encoding segment number 1, is assigned to data segments 1-24 of the second dictionary. Similarly, the eighth encoding subspace, i.e., encoding subspace 7, is assigned to data segments 1-24 of the second dictionary.

[0147]

[0150] It should be noted that the division of the coding subspace in Table 2 and the division of the dictionary data or ASII character data segments are only one possible implementation of the present disclosure, and the present disclosure is not limited thereto.

[0148]

[0151] To facilitate understanding, FIG2 schematically shows a flow chart of a data compression method provided by an embodiment of the present disclosure in a practical application.

[0149]

[0152] As shown in FIG2 , for the data to be compressed 201, firstly, a statistical analysis can be performed on the data to be compressed to obtain a first dictionary 202.

[0150]

[0153] After generating the first dictionary 202, a second dictionary 203 may be obtained. The second dictionary 203 may be constructed based on a text corpus.

[0151]

[0154] When compressing the data to be compressed 201, character data is first read from the data to be compressed 201 in sequence according to byte order, and the first dictionary 202 or the second dictionary 203 in which the character data hits is determined. Then, the encoding subspace corresponding to the first dictionary 202 or the second dictionary 203 is searched, and the encoding value group 204 corresponding to the multiple character data is obtained.

[0152]

[0155] After obtaining the code value group 204, in order to further reduce the space occupied by the code value group 304, at least one group of code values ​​in the code value group 204 can be merged to obtain a merged code value group 205.

[0153]

[0156] Then, the combined encoding value group 205 and the first dictionary 202 can be serialized to obtain compressed data 206.

[0154]

[0157] FIG3 schematically shows a flow chart of an embodiment of a data decompression method provided by an embodiment of the present disclosure. As shown in FIG3, the data decompression method may specifically include the following steps.

[0155]

[0158] 301: Get compressed data.

[0156]

[0159] 302: Obtain a first dictionary and a set of encoded value values ​​from the compressed data.

[0157]

[0160] The encoding value group is formed by searching the encoding subspace corresponding to the first dictionary or the second dictionary, and then the encoding value of the dictionary data hit by the character data in the data to be compressed in the corresponding encoding subspace is formed; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on the text corpus.

[0158]

[0161] When the compressed data is obtained by serializing the first dictionary and the encoding value group, the compressed data may be first deserialized to obtain the first dictionary and the encoding value group.

[0159]

[0162] 303: Search the encoding subspace corresponding to the first dictionary or the second dictionary respectively, determine the character data of the dictionary data hit by the encoding value group in the corresponding encoding subspace, and obtain the decompressed data corresponding to the encoding value group.

[0160]

[0163] The compressed data may be generated by compressing the data to be compressed using the data compression method shown in FIG1, which will not be described in detail here.

[0161] In an embodiment of the present disclosure, the first dictionary may be pre-constructed based on the data to be compressed. When constructing the first dictionary, repeated patterns, common items, or other features in the data to be compressed may be analyzed, and this information may be recorded in the first dictionary. By extracting repeated items or frequently occurring segments from the data to be compressed to construct the first dictionary, subsequent compression processing may be facilitated, thereby achieving efficient data compression.

[0162]

[0165] The second dictionary can be constructed based on a text corpus. A large-scale text corpus can be used as an input data source to construct the second dictionary, and the text in the text corpus is statistically analyzed and processed to extract common words, phrases, etc., and record them in the second dictionary.

[0163]

[0166] The encoding value group can be obtained by matching the character data in the data to be compressed with the dictionary data pre-written in the first dictionary and the second dictionary, respectively, and determining whether the character data hits the first dictionary or the second dictionary based on the matching results. Then, the encoding subspace corresponding to the first dictionary or the second dictionary is searched, and the encoding value of the dictionary data hit by the character data in the data to be compressed in the corresponding encoding subspace is formed. For example, for the first character data, if the first dictionary data identical to the first character data is retrieved in the first dictionary, then it can be determined that the first character data hits the first dictionary; for the second character data, if the second dictionary data identical to the second character data is retrieved in the second dictionary, then it can be determined that the second character data hits the second dictionary. Optionally, the dictionary data in the first dictionary and the second dictionary can be encoded using the same encoding format as the character data.

[0164]

[0167] Each dictionary data in the first dictionary and the second dictionary corresponds to a code value in the code subspace. Thus, the code values ​​in the code value group can be matched with the dictionary data in the first dictionary and the second dictionary in sequence to determine the dictionary data corresponding to each code value.

[0165]

[0168] According to an embodiment of the present disclosure, searching the encoding subspaces corresponding to the first dictionary or the second dictionary to determine the character data in the corresponding encoding subspaces of the dictionary data hit by the encoding value group can be specifically implemented as follows: for any encoding value in the encoding value group, searching the encoding subspaces corresponding to the first dictionary or the second dictionary to determine whether the dictionary data is hit; if so, determining the character data in the corresponding encoding subspaces of the dictionary data; if not, searching the encoding subspaces corresponding to the ASII characters to determine the character data hit by the encoding value.

[0169] In a possible implementation of the present disclosure, in the process of searching the encoding subspaces corresponding to the first dictionary or the second dictionary to determine whether the dictionary data is hit, it can be implemented as first searching the first dictionary, and then searching the second dictionary if the dictionary data of the first dictionary is not hit.

[0166]

[0170] According to an embodiment of the present disclosure, the data decompression method further includes: determining a merged value in the coding value group; wherein the merged value is obtained by combining N consecutive coding values ​​that meet the merging conditions and merging header identification data; and splitting N coding values ​​from the merged value.

[0167]

[0171] According to an embodiment of the present disclosure, splitting the merged value to obtain N coded values ​​can be specifically implemented as follows: determining the number of bits corresponding to the N coded values ​​according to the header identification data in the merged value; and splitting the merged value to obtain N coded values ​​according to the number of bits.

[0168]

[0172] When merging N code values, the N code values ​​can be merged according to the merging method recorded in the merged code table to generate a merged code value. In addition, the header identification data corresponding to the hit bit position combination can be obtained from the merged code table, and the header identification data and the code value generated by the merging can be merged to obtain the merged value.

[0169]

[0173] Therefore, the merged value can be split by using the inverse processing of the merge method recorded in the merge coding table.

[0170]

[0174] The combined coding table can refer to Table 1 and Table 2 above, which will not be repeated here.

[0171]

[0175] In a possible implementation, the data compression method and data decompression method provided by the embodiment of the present disclosure can be applied in a database scenario, and the data to be compressed can be a data table stored in the database, or a row of data or a column of data in the data table.

[0172]

[0176] Performing data compression in a database can effectively reduce the storage space occupied by data, improve storage efficiency and transmission speed. Taking a column of data in a database table as an example, the original data can be compressed to obtain compressed data.

[0173] During the compression process, a first dictionary is first obtained by statistically analyzing the data to be compressed. The dictionary records the characters and their frequency information that appear in the data to be compressed. Simultaneously, a second dictionary, obtained based on a text corpus, is used to map the character data in the data to be compressed to a set of encoding values ​​in a target encoding space. To ensure encoding uniqueness, at least one sub-encoding space can be allocated from the target encoding space for each of the first dictionary, the second dictionary, and the ASCII characters.

[0174] After mapping the data to be compressed to the code value groups, in order to further reduce the space occupied by the code value groups, a code merging operation can be performed on at least one set of code values. This can reduce the length of the compressed data without affecting data restoration, thereby reducing the storage space and transmission bandwidth requirements.

[0175]

[0179] The above-described compression processing method can compress a column of data in a table in a database, thereby reducing storage space usage. This can improve database performance and save storage costs. Furthermore, the compressed data can be decompressed when needed to restore it to its original format, ensuring data integrity and availability.

[0176]

[0180] To facilitate understanding, FIG4 schematically shows a flow chart of a data compression / decompression process in a practical application of an embodiment of the present disclosure.

[0177]

[0181] As shown in FIG. 4, the data compression method provided by the embodiment of the present disclosure can compress original data, i.e., data to be compressed, to obtain compressed data. After the compression process, the storage space occupied by the compressed data is smaller than the storage space occupied by the data to be compressed, thereby reducing the storage space or transmission bandwidth occupied by the data.

[0178]

[0182] During the data compression process, character data in the data to be compressed is mapped to code value groups in a target code space using a first dictionary obtained through statistical analysis of the data to be compressed and a second dictionary obtained based on a text corpus. Furthermore, code space allocation is required for the target code space, where at least one sub-code space is allocated for each of the first dictionary, the second dictionary, and ASCII characters.

[0183] After mapping the character data in the data to be compressed to the code value groups, to further reduce the space occupied by the code value groups, at least one code value group in the code value groups can be coded and merged, and then serialized to obtain compressed data.

[0179]

[0184] When the compressed data needs to be decompressed, the compressed data may be first deserialized. After the compressed data is deserialized, the merged code value group may be demerged to restore the merged code value group to multiple code values.

[0180]

[0185] When demerging the merged coding value group, the merge coding table can be searched according to the number of bits corresponding to the N coding values ​​indicated by the header representation data, the merging method can be determined, and then the merging method can be inversely operated to achieve demerging of the merged coding value.

[0181] After decomposition and merging, a series of encoding values ​​can be obtained. Then, by searching the first dictionary or the second dictionary, the character data corresponding to each encoding value can be determined to achieve decompression of the compressed data. The specific implementation of the operations involved in FIG. 4 can be found in the corresponding embodiments above and will not be repeated here.

[0182]

[0187] FIG5 is a schematic structural diagram of an embodiment of a data compression device provided in an embodiment of the present disclosure. The device may include the following modules.

[0183]

[0188] The data determination module 501 is used to determine the data to be compressed that is character-encoded according to the Unicode encoding format.

[0184]

[0189] The W reading module 502 is used to read character data from the data to be compressed in sequence according to byte order, and determine the first dictionary or the second dictionary in which the character data hits; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on the text corpus.

[0185]

[0190] The first search module 503 is used to search the encoding subspace corresponding to the first dictionary or the second dictionary, determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace, and obtain a group of encoding value groups corresponding to the data to be compressed; the encoding subspaces corresponding to the first dictionary and the second dictionary are obtained by segmenting the target encoding space; the dictionary data in the first dictionary and the second dictionary are sorted from most important to least important, and the encoding value corresponding to the dictionary data with the most importance is 1.

[0186]

[0191] The first processing module 504 is used to obtain compressed data based on the encoding value group and the first dictionary.

[0187]

[0192] In some embodiments, the data compression device 500 further includes: a character determination module, used to determine the ASCII character hit by the character data when the character data does not hit the first dictionary and the second dictionary; a space search module, used to search the encoding subspace corresponding to the ASCII character, and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace; wherein, the encoding subspace corresponding to the ASCII character is obtained by segmenting the target encoding space; and the encoding value corresponding to at least part of the dictionary data in the first dictionary and the second dictionary is smaller than the encoding value corresponding to the ASCII character.

[0188]

[0193] In some embodiments, the data compression device 500 further includes: a segmentation module, configured to segment the data to be compressed; and a first dictionary construction module, configured to construct a first dictionary based on multiple character strings obtained by segmentation.

[0189]

[0194] In some embodiments, the first dictionary construction module includes: a first statistical submodule, used to count the occurrence frequencies of multiple character strings in the data to be compressed; a first calculation submodule, used to calculate the importance of multiple character strings based on the occurrence frequencies and character lengths corresponding to the multiple character strings; and a first sorting submodule, used to select a first number of character strings in descending order of importance to generate a first dictionary.

[0190]

[0195] In some embodiments, the data compression device 500 further includes: a first byte number determination submodule, used to query the encoding subspace corresponding to the first dictionary in descending order of importance to determine the first compressed byte numbers corresponding to the multiple character strings; a first compression ratio determination submodule, used to determine the first compression ratios corresponding to the multiple character strings based on the first compressed byte numbers corresponding to the multiple character strings; a second byte number determination submodule, used to query the encoding subspace corresponding to the second dictionary to determine the second compressed byte numbers corresponding to the multiple character strings; a second compression ratio determination submodule, used to determine the second compression ratios corresponding to the multiple character strings based on the second compressed byte numbers corresponding to the multiple character strings; and a deletion submodule, used to delete character strings whose first compression ratio is greater than the second compression ratio from the multiple character strings.

[0191]

[0196] In some embodiments, the first calculation submodule includes: a first calculation unit, configured to calculate the importance of multiple character strings according to the occurrence frequencies, character lengths, and predetermined compressed byte counts corresponding to the multiple character strings.

[0192]

[0197] In some embodiments, the first sorting submodule includes: a second calculation unit, used to recalculate the importance of multiple strings based on the first compressed byte numbers corresponding to the multiple strings, combined with the occurrence frequencies and character lengths corresponding to the multiple strings; a string selection unit, used to select a first number of strings in descending order of the recalculated importance to generate a first dictionary.

[0193]

[0198] In some embodiments, the segmentation module includes: a character data reading submodule, which is used to read character data from the data to be compressed in byte order and save the character data as a character string; a number query submodule, which is used to query the number of occurrences of the current character string; a number counting submodule, which is used to accumulate the number of occurrences if the number of occurrences is greater than or equal to 1; if the number of occurrences is less than 1, merge the current character string with the previously saved character string, and accumulate the number of occurrences of the merged character string.

[0194]

[0199] In some embodiments, the data compression device 500 further includes: a separation module, which is used to save at least one character data separated by the separator as a character string if the currently read character data is a separator.

[0195]

[0200] In some embodiments, the data compression device 500 further includes: a statistical module for counting the frequency of occurrence of characters in the text corpus; a sorting module for sorting the characters in the text corpus in descending order of frequency of occurrence; and a second dictionary generation module for generating a second dictionary based on the sorting result.

[0196]

[0201] In some embodiments, the first processing module includes: a value determination module, used to determine N consecutive code values ​​in the code value group that meet the merging condition; a merging module, used to combine the N code values ​​and merge header identification data to obtain a merged value; wherein the header identification data is used to indicate the number of bits corresponding to the N code values; the merging condition is that the number of bytes occupied by the merged code value is less than the sum of the number of bytes occupied by the N code values; a serialization module, used to serialize the merged value, the remaining code values ​​in the code value group that do not meet the merging condition, and the first dictionary to obtain compressed data.

[0197]

[0202] In some embodiments, when serialization is performed using Varint serialization, the merging condition is specifically: the number of bits of the N encoded values ​​is greater than or equal to 8, and the number of bits of the merged value is M times 7, where N is 2 and M is 3, or N is 3 and M is 4.

[0198]

[0203] In some embodiments, the value determination module includes: a first coding table query submodule, which is used to query the merged coding table corresponding to N coding values, and determine the consecutive N coding values ​​in the coding value group that hit any group of bit combinations in the merged coding table.

[0199]

[0204] In some embodiments, the merging module includes: a second coding table query submodule, used to search the header identification data corresponding to the bit combination in the merged coding table; a merging submodule, used to combine N coding values ​​and merge the header identification data according to the merging method in the merged coding table to obtain a merged value.

[0200]

[0205] In some embodiments, the data compression device 500 further includes: a partitioning module for partitioning the target coding space into a plurality of coding subspaces; a segmentation module for determining a plurality of data segments obtained by segmenting the first dictionary and the second dictionary respectively; and an allocation module for allocating coding subspaces to the plurality of data segments from the plurality of coding subspaces according to the allocation principle that dictionary data with greater importance corresponds to smaller coding values.

[0201] The data compression device of FIG5 can implement the data compression method of the embodiment shown in FIG1 , and its implementation principle and technical effects will not be described in detail. The specific manner in which the various modules and units of the data compression device in the above embodiment perform operations has been described in detail in the embodiment of the method and will not be elaborated on here.

[0202]

[0207] FIG6 is a schematic diagram of the structure of an embodiment of a data decompression device provided by an embodiment of the present disclosure. The device may include the following modules:

[0208] A data acquisition module 601 is used to acquire compressed data.

[0203]

[0209] The second processing module 602 is used to obtain a first dictionary and a coding value group from the compressed data; wherein the coding value group is constructed by searching the coding subspace corresponding to the first dictionary or the second dictionary, and then according to the coding value of the dictionary data hit by the character data in the data to be compressed in the corresponding coding subspace; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on the text corpus.

[0204]

[0210] The second search module 603 is used to search the encoding subspace corresponding to the first dictionary or the second dictionary, determine the character data in the corresponding encoding subspace of the dictionary data hit by the encoding value group, and obtain the decompressed data corresponding to the encoding value group.

[0205]

[0211] In some embodiments, the second search module 603 includes: a first determination submodule, which is used to search the encoding subspace corresponding to the first dictionary or the second dictionary for any encoding value in the encoding value group to determine whether the dictionary data is hit; a second determination submodule, which is used to determine the character data in the encoding subspace corresponding to the dictionary data when the dictionary data is hit; and a third determination submodule, which is used to search the encoding subspace corresponding to the ASII character when the dictionary data is not hit, to determine the character data hit by the encoding value.

[0206]

[0212] In an embodiment of the present disclosure, the data decompression device 600 further includes: a first determination module, configured to determine a merged value in a coding value group; wherein the merged value is obtained by combining N consecutive coding values ​​that meet a merging condition and merging header identification data; and a splitting module, configured to split the merged value to obtain N coding values.

[0207]

[0213] In some embodiments, the splitting module includes: a bit determination submodule, which is used to determine the number of bits corresponding to N coded values ​​according to the header identification data in the merged value; and a splitting submodule, which is used to split the merged value according to the number of bits to obtain N coded values.

[0208] The data decompression device shown in FIG6 can execute the data decompression method described in the embodiment shown in FIG3 , and its implementation principle and technical effects will not be described in detail. The specific manner in which the various modules and units in the data decompression device in the above embodiment perform operations has been described in detail in the embodiment of the method and will not be elaborated on here.

[0209]

[0215] In one possible design, the data compression device and data decompression device provided by the embodiments of the present disclosure can be implemented as a computing device. As shown in Figure 7, the computing device may include a storage component 701 and a processing component 702; the storage component 701 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 702 to implement the data compression method and data decompression method provided by the embodiments of the present disclosure.

[0210]

[0216] Of course, a computing device may also include other components, such as input / output interfaces, communication components, etc. The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0211]

[0217] In which, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0212]

[0218] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device.

[0213]

[0219] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer, it can implement the data compression method and data decompression method provided by the embodiment of the present disclosure.

[0214]

[0220] The embodiment of the present disclosure also provides a computer program product, including a computer program, which can implement the data compression method and data decompression method provided by the embodiment of the present disclosure when executed by a computer.

[0215]

[0221] The processing components in the above corresponding embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing components may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0216]

[0222] The storage component is configured to store various types of data to support operations in the device. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM).

[0217] (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Magnetic or Optical Disk.

[0218]

[0223] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0219]

[0224] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Persons of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0220] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0221]

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

Claims 1. A data compression method, comprising: Determining data to be compressed that is character-encoded according to a Unicode encoding format; Reading character data from the data to be compressed in byte order and determining a first dictionary or a second dictionary in which the character data matches; wherein the first dictionary is constructed based on the data to be compressed; and the second dictionary is constructed based on a text corpus; searching a coding subspace corresponding to the first dictionary or the second dictionary, respectively, and determining a coding value of the dictionary data matched by the character data in the corresponding coding subspace, thereby obtaining a set of coding value groups corresponding to the data to be compressed; Compressed data is obtained based on the encoding value group and the first dictionary.

2. The method according to claim 1, further comprising: If the character data does not match the first dictionary or the second dictionary, determining the ASCII character that the character data matches; Search the encoding subspace corresponding to the ASCII character and determine the encoding value of the dictionary data hit by the character data in the corresponding encoding subspace; wherein the encoding subspace corresponding to the ASCII character is obtained by segmenting the target encoding space; and the encoding values ​​corresponding to at least part of the dictionary data in the first dictionary and the second dictionary are smaller than the encoding value corresponding to the ASCII character.

3. The method according to claim 1, wherein: The first dictionary is generated in the following manner: segmenting the data to be compressed; and constructing the first dictionary based on a plurality of character strings obtained by segmentation.

4. The method according to claim 3, wherein: The step of constructing the first dictionary based on the multiple character strings obtained by segmentation includes: counting the occurrence frequencies of the multiple character strings in the data to be compressed; calculating the importance of the multiple character strings based on the occurrence frequencies and character lengths of the multiple character strings; and selecting a first number of character strings in descending order of importance to generate the first dictionary; wherein the dictionary data in the first dictionary are sorted in descending order of importance, and the dictionary data with greater importance corresponds to smaller code values.

5. The method according to claim 4, further comprising: querying the encoding subspace corresponding to the first dictionary in descending order of importance to determine first compressed byte numbers corresponding to the multiple character strings; determining first compression ratios corresponding to the plurality of character strings based on first compressed byte counts corresponding to the plurality of character strings; querying a coding subspace corresponding to the second dictionary to determine second compressed byte counts corresponding to the plurality of character strings; Determining second compression ratios corresponding to the plurality of character strings based on second compressed byte numbers corresponding to the plurality of character strings; and deleting character strings having a first compression ratio greater than a second compression ratio from the plurality of character strings.

6. The method according to claim 4, wherein: The calculating the importance of the plurality of character strings according to the occurrence frequencies and character lengths respectively corresponding to the plurality of character strings comprises: calculating the importance of the plurality of character strings according to the occurrence frequencies, character lengths and predetermined compressed byte counts respectively corresponding to the plurality of character strings; the selecting a first number of character strings in descending order of importance to generate the first dictionary comprises: recalculating the importance of the plurality of character strings according to the first compressed byte counts respectively corresponding to the plurality of character strings, in combination with the occurrence frequencies and character lengths respectively corresponding to the plurality of character strings; and selecting a first number of character strings in descending order of the recalculated importance to generate the first dictionary. Dictionary.

7. The method according to claim 3, wherein: The segmentation processing of the data to be compressed includes: reading character data from the data to be compressed in byte order and saving the character data as a character string; querying the number of occurrences of a current character string; if the number of occurrences is greater than or equal to 1, accumulating the number of occurrences; if the number of occurrences is less than 1, merging the current character string with a previously saved character string, and accumulating the number of occurrences of the merged character string.

8. The method according to claim 7, further comprising: If the currently read character data is a separator, at least one character data separated by the separator is saved as a character string.

9. The method according to claim 1, wherein: The second dictionary is generated in the following manner: counting the frequencies of occurrence of characters in the text corpus; sorting the characters in the text corpus in descending order of frequency of occurrence; and generating the second dictionary based on the sorting results; wherein the dictionary data in the second dictionary are sorted in descending order of importance, with the dictionary data with greater importance corresponding to the encoding value 1.

10. The method according to claim 1, wherein: Obtaining compressed data based on the encoding value group and the first dictionary includes: determining N consecutive encoding values ​​in the encoding value group that meet a merging condition; combining the N encoding values ​​and merging header identification data to obtain a merged value; wherein the header identification data is used to indicate the number of bits corresponding to the N encoding values ​​respectively; the merging condition is that the number of bytes occupied by the merged encoding value is less than the sum of the number of bytes occupied by the N encoding values; and serializing the merged value, the remaining encoding values ​​in the encoding value group that do not meet the merging condition, and the first dictionary to obtain compressed data.

11. The method according to claim 10, wherein: When serialization is performed using the variable-length integer encoding Varint serialization method, the merging condition is specifically: the number of bits of the N encoded values ​​is greater than or equal to 8, and the number of bits of the merged value is M times 7, where N is 2 and M is 3, or N is 3 and M is 4.

12. The method according to claim 10, wherein: Determining the N consecutive coding values ​​in the coding value group that meet the merging condition includes: querying the merged coding table corresponding to the N coding values, and determining the N consecutive coding values ​​in the coding value group that hit any group of bit position combinations in the merged coding table; combining the N coding values ​​and merging the header identification data to obtain the merged value includes: searching the merged coding table for the header identification data corresponding to the bit position combination; combining the N coding values ​​and merging the header identification data according to the merging method in the merged coding table to obtain the merged value.

13. The method according to claim 1, wherein: The coding subspace is generated as follows: the target coding space is divided into multiple coding subspaces; multiple data segments are determined by dividing the first dictionary and the second dictionary respectively; and coding subspaces are allocated to the multiple data segments from the multiple coding subspaces according to the allocation principle that the coding values ​​corresponding to the more important dictionary data are smaller.

14. A data decompression method, comprising: Get compressed data; A first dictionary and a coding value group are obtained from the compressed data; wherein the coding value group is formed by searching the coding subspace corresponding to the first dictionary or the second dictionary respectively, and according to the coding value of the dictionary data hit by the character data in the data to be compressed in the corresponding coding subspace; wherein the first dictionary is constructed based on the data to be compressed; the second dictionary is constructed based on a text corpus; searching the coding subspace corresponding to the first dictionary or the second dictionary respectively, and determining the coding value group The character data in the corresponding encoding subspace of the dictionary data in the encoding subspace is obtained to obtain the decompressed data corresponding to the encoding value group.

15. The method according to claim 14, wherein: The searching of the coding subspaces corresponding to the first dictionary or the second dictionary respectively and determining the character data in the coding subspace corresponding to the dictionary data hit by the coding value group includes: for any coding value in the coding value group, searching the coding subspaces corresponding to the first dictionary or the second dictionary respectively to determine whether the dictionary data is hit; if so, determining the character data in the corresponding coding subspace of the dictionary data; if not, searching the coding subspace corresponding to the ASII character to determine the character data hit by the coding value.

16. The method according to claim 14, wherein: Obtaining a first dictionary and a coding value group from the compressed data includes: deserializing the compressed data to obtain a first dictionary and coding values; the method further includes: determining a merged value in the coding value group; wherein the merged value is obtained by combining N consecutive coding values ​​that meet a merging condition and merging header identification data; and splitting the N coding values ​​from the merged value.

17. The method according to claim 16, wherein: The splitting from the merged value to obtain the N coded values ​​includes: determining the number of bits corresponding to the N coded values ​​respectively according to the header identification data in the merged value; and splitting from the merged value to obtain the N coded values ​​according to the number of bits.

18. A computing device, comprising a processing component and a storage component; the storage component storing one or more computer instructions; the one or more computer instructions being invoked and executed by the processing component to implement the data compression method according to any one of claims 1 to 13, or the data decompression method according to any one of claims 14 to 17.

19. A computer storage medium storing a computer program, wherein: When the computer program is executed by a computer, the data compression method according to any one of claims 1 to 13 is implemented, or the data decompression method according to any one of claims 14 to 17 is implemented.

20. A computer program product, wherein: The computer program product includes computer program code, and when the computer program code is executed by a computer, it implements the data compression method according to any one of claims 1 to 13, or implements the data decompression method according to any one of claims 14 to 17.

Images