Text compression method, module, chip, electronic device and storage medium

Abstract

A text compression method, module, chip, electronic device and storage medium, the method comprising: performing word segmentation processing on the text to be compressed, and performing word length and word frequency statistics on the words obtained after word segmentation processing, and constructing a corresponding keyword list; based on the constructed keyword list, the text to be compressed is compressed. The above scheme improves the compression rate of text compression.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a text compression method, module, chip, electronic device, and storage medium. Background Technology

[0002] Text compression is the process of encoding large amounts of text data using certain methods to achieve information compression and storage. The compressed data can be restored to its pre-compression state through decoding without losing information.

[0003] However, existing text compression methods suffer from low compression rates. Summary of the Invention

[0004] The problem addressed by this invention is to provide a text compression method, module, chip, electronic device, and storage medium to improve the compression ratio of text compression.

[0005] To address the above problems, the present invention provides a text compression method, the method comprising:

[0006] The text to be compressed is segmented into words, and the word length and frequency of the words obtained after segmentation are statistically analyzed to construct a corresponding keyword list;

[0007] Based on the constructed keyword list, the text to be compressed is compressed.

[0008] Accordingly, embodiments of the present invention also provide a text compression module, the text compression module comprising:

[0009] The list building unit is suitable for performing word segmentation on the text to be compressed, and for performing word length and word frequency statistics on the words obtained after word segmentation to build a corresponding keyword list;

[0010] The first compression unit is adapted to compress the text to be compressed based on the constructed keyword list.

[0011] Accordingly, embodiments of the present invention also provide a chip, the chip including the text compression module described in any of the above claims.

[0012] Accordingly, embodiments of the present invention also provide an electronic device, which includes the chip described above.

[0013] Accordingly, embodiments of the present invention also provide a storage medium storing one or more computer instructions, the one or more computer instructions being used to implement the text compression method described in any of the above embodiments.

[0014] Compared with the prior art, the technical solution of the present invention has the following advantages:

[0015] The above-described scheme involves segmenting the text to be compressed into words, then calculating the word length and frequency of the segmented words to construct a keyword list. The text is then compressed based on this keyword list. Because the text is segmented before compression, and the resulting words are encoded, the length of the compressed characters is increased, thus significantly improving compression efficiency. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a text compression method according to an embodiment of the present invention is shown;

[0017] Figure 2 A flowchart illustrating a method for constructing a keyword list according to an embodiment of the present invention is shown;

[0018] Figure 3 A schematic diagram of a singly linked list of keywords pointed to by a keyword pointer array is shown in an embodiment of the present invention;

[0019] Figure 4 A flowchart illustrating a method for compressing text based on a constructed keyword list according to an embodiment of the present invention is shown.

[0020] Figure 5 A flowchart illustrating a method for constructing an adjacency-joining red-black tree corresponding to each keyword of the corresponding type according to an embodiment of the present invention is shown.

[0021] Figure 6 A schematic diagram of the neighbor-join red-black tree for the keyword "call_" of the corresponding type constructed according to an embodiment of the present invention is shown.

[0022] Figure 7 This is a flowchart illustrating a method for merging and encoding keywords of corresponding types in the first compressed text according to an embodiment of the present invention;

[0023] Figure 8 A flowchart illustrating a method for compressing ASCII code values ​​in the third compressed text according to an embodiment of the present invention is shown.

[0024] Figure 9 The diagram illustrates a method for constructing a two-dimensional array token_profit and counting the occurrences of unencoded characters in the third compressed text according to an embodiment of the present invention.

[0025] Figure 10 A schematic diagram of the two-dimensional array token_profit constructed according to an embodiment of the present invention is shown;

[0026] Figure 11 This diagram illustrates the encoding slots of the two-dimensional array best_table constructed according to an embodiment of the present invention.

[0027] Figure 12 The flowchart illustrates a method for allocating the remaining encoding slots in a two-dimensional array best_table to consecutive characters that appear multiple times in a third compressed text, according to an embodiment of the present invention.

[0028] Figure 13 A schematic diagram of the structure of a text compression module according to an embodiment of the present invention is shown;

[0029] Figure 14 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0030] As can be seen from the background technology, existing text compression methods suffer from low compression rates.

[0031] Specifically, an existing text compression method uses one encoding slot to compress one, two, or three characters. Because the length of the compressed characters is relatively short, it suffers from a low compression rate.

[0032] To address the aforementioned problems, this invention provides a text compression method. First, the text to be compressed is segmented into words, and the word length and frequency of the segmented words are statistically analyzed to construct a corresponding keyword list. Then, based on the constructed keyword list, the text to be compressed is compressed. Because the text to be compressed is segmented into words before compression, and the segmented words are encoded, the length of the compressed characters increases, thus significantly improving compression efficiency.

[0033] Figure 1 A flowchart of a text compression method according to an embodiment of the present invention is shown. See also Figure 1 A text method, specifically including:

[0034] Step S110: Perform word segmentation on the text to be compressed, and perform word length and word frequency statistics on the words obtained after word segmentation to construct a corresponding keyword list;

[0035] Step S120: Compress the text to be compressed based on the constructed keyword list.

[0036] To make the above-mentioned objects, features and advantages of the embodiments of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] In step S110, the text to be compressed is segmented into words, and the word length and frequency of the words obtained after segmentation are statistically analyzed to construct a corresponding keyword list.

[0038] In fact, there is a lot of repetitive content in the symbols of the Linux kernel code. Observation reveals that the symbols in the Linux kernel code contain specific word segmentation symbols, i.e., delimiters. Based on this characteristic of the Linux kernel code symbols, in one embodiment of this invention, when the text to be compressed is Linux kernel code, the delimiters can be used to divide the text into corresponding words. The delimiters can be "-", "_", or spaces, etc.

[0039] The text to be compressed is compressed to provide a basis for assigning corresponding encoding slots to the segmented words in the future.

[0040] In one embodiment of the present invention, while performing word segmentation on the text to be compressed, the word length and word frequency of the words obtained after word segmentation are statistically analyzed based on a preset keyword data structure and keyword pointer array, and a corresponding keyword list is constructed to improve compression efficiency.

[0041] Specifically, in one embodiment of the present invention, the keyword data structure specifically includes the following elements:

[0042] The next member is a linked list pointer to the next keyword with the same hash value;

[0043] The `word` member is used to record the corresponding words;

[0044] The total_cnt member is used to record the number of times the words recorded in the word member appear.

[0045] The word_len member is used to record the length of the words recorded in the word member;

[0046] The idx_type member is used to store the encoded values ​​of the words recorded in the word member.

[0047] The `next` member facilitates the lookup of word storage locations, the `word` member is used to record information about the words described by the corresponding keyword data structure, the `total_cnt` member can count the number of occurrences of the words recorded in the `word` member, the `word_len` member is used to record the character length of the words recorded in the `word` member, and the `idx_type` member can be used to allocate encoding slots for the words recorded in the `word` member.

[0048] Using the keyword data structure to describe the words obtained after word segmentation can facilitate the calculation of word length and word frequency of the words obtained in the text to be compressed, and thus facilitate the subsequent allocation of corresponding encoding slots to the segmented words.

[0049] In a specific implementation, the keyword pointer array is used to store the keyword data structure instances of the divided words into a preset hash linked list space. Specifically, each member in the keyword pointer array has a corresponding pointer head, which points to the keyword singly linked list with the corresponding hash value, and each word in the keyword singly linked list has a pointer to the next word with the same hash value.

[0050] After setting up the corresponding keyword data structure and keyword pointer array, you can perform word length and word frequency statistics on the words obtained after word segmentation based on the preset keyword data structure and keyword pointer array to construct the corresponding keyword list. For details, please refer to [link to relevant documentation]. Figure 2 .

[0051] Figure 2 A flowchart illustrating a method for constructing a keyword list according to an embodiment of the present invention is shown. Please refer to... Figure 2 A method for constructing a keyword list may include the following steps:

[0052] Step S1101: Traverse the text to be compressed and obtain the current word reached during the traversal.

[0053] In practice, the text to be compressed can be traversed sequentially, and words in the text can be identified using delimiters as boundaries.

[0054] The words in the compressed text are traversed in order to facilitate the counting of word lengths and occurrences.

[0055] Step S1102: Calculate the hash value corresponding to the current word.

[0056] As mentioned earlier, the hash value of a word is used as the index of the keyword pointer array, so that the hash list space can be addressed using this index to find the keyword data structure instance corresponding to the word.

[0057] In practice, different hash functions can be used to calculate the hash value of words. Those skilled in the art can set them according to actual needs, and no restrictions are imposed here.

[0058] Step S1103: Using the calculated hash value as an index, determine whether there is a keyword data structure instance of the current word in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array; if the determination result is yes, step S1104 can be executed; otherwise, step S1105 can be executed.

[0059] In practice, the calculated hash value is the pointer head in the keyword pointer array. The corresponding keyword singly linked list can be found through this pointer head. Subsequently, the current word can be compared one by one with the word members of each keyword data structure instance in the found keyword singly linked list to determine whether the keyword data structure instance of the current word exists in the corresponding keyword singly linked list.

[0060] See Figure 3 For each word obtained from the segmentation in the text to be compressed, the keyword data structure instances of words with the same hash value are stored in the same keyword linked list.

[0061] For example, assuming the hash value of the word "override" is 2, the keyword data structure instance for "override" will be inserted into the linked list of keywords formed by the second pointer in the keyword pointer array. The words "find," "event," and "notifier" all have the same hash value of 357, and are all inserted into the linked list pointed to by pointer 357. Subsequent occurrences of a word with a hash value of 357 only require comparison with the words in the linked list pointed to by pointer 357.

[0062] Therefore, by using the keyword pointer array, when storing the keyword data structure instances of words in the text to be compressed, the comparison between words can be limited to the corresponding keyword singly linked list, rather than being compared with all the words that have already been stored. This can significantly reduce the number of word query comparisons and thus improve compression efficiency.

[0063] Step S1104: Increase the preset first value by the total_cnt member in the corresponding keyword data structure instance.

[0064] In practice, when it is determined that the keyword data structure instance of the current word exists in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array, the total_cnt member of the current word data structure instance can be incremented by a preset first value to accumulate the number of times the current word appears in the text to be compressed. The first value can be set according to actual needs and is not limited here.

[0065] Step S1105: Construct a corresponding keyword data structure instance for the current word, and insert the created keyword data structure instance into the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array.

[0066] In practical implementation, when the keyword data structure instance for the current word does not exist in the linked list pointed to by the corresponding pointer in the keyword pointer array, a corresponding keyword data structure instance can be created for the current word and inserted into the linked list pointed to by the corresponding pointer in the keyword pointer array, so as to accumulate the occurrence count of the current word. Specifically, in the created keyword data structure instance for the current word, the value of the total_cnt member is the first value. That is, when the keyword data structure instance is initialized, the value of the total_cnt member is the first value. When the current word appears again, the value is accumulated based on the first value, so that the occurrence count of the current word in the text to be compressed can be accumulated.

[0067] Step S1106: Determine whether the text to be compressed has been traversed completely; if the determination result is yes, the operation can be ended; otherwise, step S1107 can be executed.

[0068] Step S1107: Obtain the next word as the current word to be traversed, and restart step S1102.

[0069] In specific implementation, when the words in the text to be compressed have not been traversed, the next word can be obtained according to the delimiter, and the obtained next word can be used as the current word to be traversed. Then, the execution can be restarted from step S1102 until all the words in the text to be compressed have been traversed. Then, the keyword list corresponding to the text to be compressed can be obtained.

[0070] However, in practical applications, not every word obtained after word segmentation has compression value. For example, when the length of the segmented words is less than or equal to the length of the compression code, or when the words appear infrequently, compressing these types of words not only fails to save storage space but also further reduces compression efficiency.

[0071] Therefore, in one embodiment of the present invention, corresponding encoding conditions are set to filter words in the constructed keyword list that do not meet the compression conditions, so as to further improve the compression efficiency.

[0072] Specifically, the encoding conditions are:

[0073] n×L>(2+L) (1)

[0074] Where n represents the frequency of word occurrence, and L represents the symbol length of word occurrence.

[0075] By setting the above encoding conditions, words that appear infrequently or whose length is less than the preset encoding length can be removed from the constructed keyword list, thereby improving compression efficiency. Specifically, by traversing all the keyword singly linked lists pointed to by the pointers in the keyword pointer array, the keyword data structure instances that do not meet the above encoding conditions are found, and they are removed from the corresponding keyword singly linked lists.

[0076] By performing steps S1101 to S1107 above, corresponding keyword data structure instances can be created for the words obtained after word segmentation in the text to be compressed, and stored in the keyword singly linked list pointed to by the keyword pointer array. In the future, by querying the keyword data structure instance of the word, the information on the frequency of occurrence of the word can be obtained, and the word can be compressed based on the information on the frequency of occurrence of the word.

[0077] Step S120 is executed to compress the text to be compressed based on the constructed keyword list.

[0078] As mentioned earlier, the constructed keyword list, i.e., the keyword data structure instance of words stored in the hash linked list space, records information about the words appearing in the text to be compressed and their frequency. Therefore, based on the constructed keyword list, information about the words appearing in the text to be compressed and their frequency can be obtained. Furthermore, based on the frequency information, the words in the text to be compressed can be compressed and encoded. For details, please refer to [link to relevant documentation]. Figure 4 .

[0079] Figure 4 This diagram illustrates a method for compressing text based on a constructed keyword list, according to an embodiment of the present invention. See also... Figure 4 A method for compressing the text to be compressed based on a constructed keyword list, according to an embodiment of the present invention, may specifically include:

[0080] Step S1201: Traverse the text to be compressed and obtain the current word reached during the traversal.

[0081] In practice, the text to be compressed is traversed so that the corresponding encoding slots can be assigned to the words appearing in the text to be compressed based on the keyword data structure instances stored in the keyword list.

[0082] Step S1202: Calculate the hash value of the current word.

[0083] In practice, the hash function used to calculate the hash value of the current word is the same as the hash function used when constructing the keyword list.

[0084] Step S1203: Using the calculated hash value as an index, determine whether there is a corresponding keyword data structure instance in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer data; if the determination result is yes, step S1204 can be executed; otherwise, step S1206 can be executed.

[0085] In practice, the calculated hash value is used as an index to determine whether there is a corresponding keyword data structure instance in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer data, so as to provide a basis for allocating encoding slots for the current word in the future.

[0086] Step S1204: Combine the current word encountered during traversal with the separator at the corresponding position to form a keyword of the corresponding type. Based on the value of the idx_type member in the keyword data structure instance of the current word encountered during traversal, determine whether the keyword of the corresponding type has been assigned an encoding slot. If the determination result is yes, step S1206 can be executed; otherwise, step S1205 can be executed.

[0087] In practice, the keyword list does not include information about delimiters in the text to be compressed. During actual compression, delimiters can be merged with one of the words preceding or following them to further improve the compression rate and facilitate subsequent word merging operations.

[0088] Specifically, based on the positional relationship between the separator and the words, merging the separator with the words appearing before and after it yields the following four types of keywords:

[0089] The first type of keywords includes the corresponding words and the separators following the words;

[0090] The second type of keywords includes the corresponding separator and the words following the separator;

[0091] The third type of keywords includes the corresponding words and the separators before and after the words;

[0092] The fourth type of keywords only includes the corresponding words.

[0093] Specifically, when a word in the keyword list appears at the beginning of a symbol, or when the preceding word has already merged with the separator preceding it, the word can merge with the separator that appears after it to form the first type of keyword; when a word in the keyword list appears at the end of a symbol, and the preceding word has no compression value, the word can merge with the separator that appears before it to form the second type of keyword; when both words before and after a word in the keyword list have no compression value, the word can merge with the separators that appear before and after it to form the third type of keyword; when a word in the keyword list appears at the end of a symbol, and the preceding word has compression value, and the separator preceding the word has already merged with the word preceding it, the word itself constitutes the fourth type of keyword.

[0094] After the current word is traversed and its preceding and following separators match to form a keyword of one of the four types mentioned above, a coding slot can be assigned to the merged keyword of the corresponding type.

[0095] As mentioned earlier, in the keyword data structure instance, the idx_type member is used to store the encoding value of the corresponding word. During initialization, the idx_type member is set to its default value; after the corresponding word has been assigned an encoding slot, the idx_type member contains information about the assigned encoding slot.

[0096] Therefore, when there is a keyword data structure instance of the current word that is traversed in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer data, it can be determined whether the keyword of the corresponding type has been assigned an encoding slot by querying the information recorded in the idx_type member.

[0097] Step S1205: Assign encoding slots to the corresponding type of keywords, record the corresponding encoding values ​​in the idx_type member of the corresponding keyword data structure instance, and create the corresponding final character data structure instance for the corresponding type of keywords.

[0098] In specific implementation, when it is determined that the corresponding type of keyword has not been assigned a coding slot and there is an empty coding slot, a coding slot is assigned to the corresponding type of keyword, and the corresponding coding value is recorded in the idx_type member of the corresponding keyword data structure instance as an identifier that the corresponding type of keyword has been assigned a coding slot for subsequent querying.

[0099] When a corresponding encoding slot is assigned to a keyword of the corresponding type, a corresponding final character data structure instance is created for the keyword of the corresponding type to record the encoding information of the finally generated keyword of the corresponding type for subsequent decompression query.

[0100] In one embodiment of the present invention, the final character data structure corresponding to the final character data structure instance includes the following elements:

[0101] The Str member is used to record information about keywords of the corresponding type;

[0102] The Len member is used to record the character length of the corresponding keyword.

[0103] The cnt member is used to record the number of occurrences of the corresponding keyword.

[0104] It should be noted that other methods can also be used to record the information of the corresponding type of keywords generated in the end. Those skilled in the art can choose according to actual needs, and the present invention does not impose any restrictions.

[0105] Step S1206: Obtain the encoding value recorded in the idx_type member of the corresponding keyword data structure instance, and replace the corresponding keyword in the text to be compressed with the encoding value recorded in the idx_type member of the corresponding keyword data structure instance.

[0106] In specific implementation, when the corresponding encoding slot has been assigned to the corresponding type of keyword, the encoding value recorded in the idx_type member of the corresponding keyword data structure instance can be obtained, and the corresponding type of keyword in the text to be compressed can be replaced with the encoding value recorded in the idx_type member of the corresponding keyword data structure instance, thereby compressing the corresponding type of keyword appearing in the text to be compressed.

[0107] Step S1207: Determine whether the text to be compressed has been traversed completely; if the determination result is yes, the operation can be ended; otherwise, step S1208 can be executed.

[0108] Step S1208: Obtain the next word in the text to be compressed as the current word to be traversed, and restart from step S1202.

[0109] In specific implementation, when the text to be compressed has not been traversed completely, the next word can be obtained as the current word to be traversed, and the execution can be restarted from step S1202 until the text to be compressed is traversed completely, and the corresponding first compressed text is obtained.

[0110] In this application, strings appearing line by line in Linux kernel code symbols are treated as a single symbol. Observation reveals that each symbol contains multiple words, and these words are arranged in the same order, as shown below:

[0111] 00000000 t trace_event_raw_event_timer_class

[0112] 00000000 t trace_event_raw_event_timer_start

[0113] 00000000 t trace_event_raw_event_timer_expire_entry

[0114] 00000000 t trace_event_raw_event_hrtimer_init

[0115] 00000000 t trace_event_raw_event_hrtimer_start

[0116] 00000000 t trace_event_raw_event_hrtimer_expire_entry

[0117] 00000000 t trace_event_raw_event_hrtimer_class

[0118] 00000000 t trace_event_raw_event_itimer_state

[0119] As shown above, there are multiple symbols that begin with "trace_event_raw_event_". Therefore, in this embodiment of the invention, multiple words that appear repeatedly in the text to be compressed, including those appearing in sequence, are merged and encoded to further improve the compression rate.

[0120] For details, please continue to see Figure 1 After obtaining the first compressed text, it may also include:

[0121] Step S130: Merge and encode the large encoded characters in the first compressed text to obtain the second compressed text.

[0122] In one embodiment of the present invention, an adjacency-joining red-black tree can be constructed for each of the large coded words appearing in the first compressed text, and the large coded words in the first compressed text can be merged and encoded based on the constructed adjacency-joining red-black tree.

[0123] To facilitate the construction of the adjacency-joining red-black tree for each of the large encoded words, the final word data structure instance created for the corresponding keyword type also includes the following members:

[0124] The node_cnt member is used to record the number of different keywords of the corresponding type that are adjacent to the keyword of the corresponding type.

[0125] Specifically, a pre-defined merged word data structure is used to construct an adjacency-merging red-black tree for keywords of the corresponding type, in order to statistically analyze the other words of the corresponding type adjacent to the keywords of the corresponding type and their frequency of occurrence. For details, please refer to [link to documentation]. Figure 5 .

[0126] In one embodiment of the present invention, the merged word data structure includes the following members:

[0127] The entry member is used to represent the root node of the adjacency-join red-black tree;

[0128] The code member is used to record the encoded value of the keyword of the corresponding type that is adjacent to the root node of the red-black tree and recorded in the entry member.

[0129] The `count` member is used to record the number of occurrences of keywords of the corresponding type that are adjacent to the red-black tree nodes recorded in the `entry` member.

[0130] Figure 5 A flowchart illustrating a method for constructing an adjacency-joining red-black tree for each of the large coded words, according to an embodiment of the present invention, is shown. See also... Figure 5 A method for constructing an adjacency-join red-black tree for each of the large coded words may specifically include:

[0131] Step S1301: Scan the first compressed text to obtain the two adjacent large encoded characters that are currently being traversed.

[0132] In specific implementation, the first compressed text is obtained by compressing keywords of the corresponding type in the text to be compressed. That is, the first compressed text includes the encoding of the corresponding type of keywords and other unencoded character information. For ease of description, the encoding value corresponding to the corresponding type of keywords is referred to as a large encoded character in this application.

[0133] The inventors of this application have discovered that when consecutive large coded characters appear in the first compressed text and the number of occurrences of these consecutive large coded characters is greater than or equal to a preset merging threshold, merging these consecutive large coded characters can further improve the compression rate. Therefore, the first compressed text can be scanned in a sequential manner from beginning to end to query whether two adjacent large coded characters exist in the first compressed text, providing a basis for subsequently merging and encoding the two adjacent large coded characters found in the query.

[0134] Step S1302: Using the tree member in the final word data structure corresponding to the first large encoded word as the root, determine the corresponding adjacency-joining red-black tree.

[0135] In a specific implementation, when it is determined that there are two adjacent large encoded words in the first compressed text, the large encoded word that is arranged first among the two adjacent large encoded words can be called the first large encoded word and the large encoded word that is arranged second is called the second large encoded word. The tree member in the final word data structure corresponding to the first large encoded word can be queried, and the corresponding adjacent union red-black tree can be obtained by taking the tree member as the root.

[0136] Step S1303: In the determined adjacency-merging red-black tree, query the merged word data structure instance whose key value is the last large encoded word; if it is determined that there is a corresponding merged word data structure instance, step S1304 can be executed; otherwise, step S1305 can be executed.

[0137] In practice, each node in the obtained adjacency-merging red-black tree includes key-value information of keywords of the corresponding type that are adjacent to the keyword of the corresponding type. Therefore, by querying the determined adjacency-merging red-black tree to determine whether there is a merged character data structure instance with the key value of the next encoded character, it is possible to determine whether the merged character data structure instance of the next large encoded character exists in the adjacency-merging red-black tree of the previous large encoded character, so as to subsequently count the occurrence frequency of the two adjacent large encoded characters in the first compressed text.

[0138] Step S1304: Increase the count member of the corresponding merged word data structure instance by a preset second value.

[0139] In practical implementation, when a merged word data structure instance with the key value of the next large encoded word exists in the determined adjacency merged red-black tree, it indicates that the two adjacent large encoded words have already appeared before. At this time, the count member of the merged word data structure instance can be incremented by a preset second value to accumulate the occurrence counts of the two adjacent large encoded words. The second value can be set according to actual needs and is not limited here.

[0140] Step S1305: Create a corresponding merged word data structure instance and insert the created merged word data structure instance into the adjacent merged red-black tree of the previous large encoded word.

[0141] In specific implementation, when there is no corresponding merged word data structure instance in the determined adjacency merged red-black tree, a corresponding merged word data structure instance is created and inserted into the adjacency merged red-black tree of the previous large coded word, so as to count the occurrence times of the current two consecutive large coded words traversed in the first compressed text.

[0142] In the corresponding merged word data structure instance created during initialization, the `entry` member is the value of the node in the allocated adjacency merged red-black tree, the `code` member is the encoded value of the next large encoded word, and the value of the `count` member can be the default initial value. Subsequently, when the two adjacent large encoded words appear again during traversal, the occurrence count of the two adjacent large encoded words can be counted by accumulating the value of the `count` member.

[0143] Step S1306: Determine whether the first compressed text has been traversed completely; if the determination result is yes, the operation can be ended; otherwise, step S1307 can be executed.

[0144] Step S1307: Obtain the two next adjacent large encoded words in the first compressed text as the two currently adjacent large encoded words to be traversed, and restart execution from step S1302.

[0145] In specific implementation, when the first compressed text has not been traversed, the next two adjacent large encoded words are taken as the two currently adjacent large encoded words to be traversed, and the execution starts again from step S1302 to count the number of occurrences of adjacent large encoded words in the first compressed text until the first compressed text has been completely traversed.

[0146] Please see Figure 6 Taking the previously encoded word "call_" as an example, we search for the final word data structure instance with the Str member "call_" in the final word data structure. When the final word data structure instance with the Str member "call_" is found, we retrieve the value of the tree member from the final word data structure instance of "call_", and using the tree member in the final word data structure instance corresponding to "call_" as the root, we can retrieve the adjacency-join-merge red-black tree of "call_". Table 1 shows the information of the adjacency-join-merge red-black tree of "call_".

[0147] Table 1

[0148] key count first string->next string 120 1 call_->get_ 164 1 call->set 199 1 call_->table 403 1 call_->handler 423 20 call_->function_ 935 74 call_->chain 969 6 call_->rcu_ 1228 6 call_->usermodehelper_ 1234 3 call_->usermodehelper 1663 4 call_->function 2154 3 call_->srcu 2252 4 call_->base 3022 1 call_->proto 3112 1 call_->nested 11246 3 call_->netdevice_ 11306 3 call -> netevent

[0149] As shown in Table 1, there are 16 adjacent words for "call_". Among them, the adjacent word "function_" appears 20 times after "call_", and "chain" appears 74 times after "call_", both of which have compression value, while the word "nested" appears only once after "call_".

[0150] It should be noted that the depth of the adjacency-join red-black tree for each corresponding keyword type is not the same. For example, the adjacency-join red-black tree for words like "get", "set", and "put" can have up to thousands of nodes.

[0151] After performing steps S1301 to S1307 above, creating corresponding adjacency-joining red-black trees for two adjacent large encoded words in the first compressed text, the adjacent large encoded words in the first compressed text can be merged according to the created adjacency-joining tree. For details, please refer to [link to relevant documentation]. Figure 7 .

[0152] Figure 7 This diagram illustrates a method for merging and encoding keywords of corresponding types in the first compressed text, according to an embodiment of the present invention. (See also...) Figure 7 A method for merging and encoding keywords of corresponding types based on a constructed adjacency-joining red-black tree may specifically include the following steps:

[0153] Step S1311: Take the first compressed text as the current text to be merged, and traverse the current text to be merged to obtain the two adjacent large encoded words that are traversed to the current text.

[0154] In a specific implementation, the first compressed text is taken as the current text to be merged. By traversing the current text to be merged, the adjacency union red-black tree of the first large codeword among the two adjacent large codewords appearing in the first compressed text is obtained.

[0155] It should be noted that during the first round of merging encoding, the current text to be merged is the first compressed text; after the first round of merging encoding, that is, starting from the second round of merging encoding, the subsequent current text to be merged is the text obtained after the previous round of merging encoding.

[0156] In practice, the symbols in the current text to be merged can be traversed in a forward-to-back order to query for two adjacent large coded characters, so as to be merged in subsequent steps.

[0157] Step S1312: In the adjacency merge red-black tree of the first large codeword among the two currently adjacent large codewords, find the merged word data structure instance whose key value is the second large codeword.

[0158] As mentioned earlier, each red-black tree node in the adjacency-merging red-black tree of the corresponding keyword type records information about other keywords of the corresponding type adjacent to the keyword of that type and their frequency of occurrence. Therefore, the merged word data structure instance with the key value of the last large encoded word is queried in the adjacency-merging red-black tree of the corresponding keyword type to obtain the frequency of occurrence information recorded in the count member of the merged word data structure instance, so as to determine whether to perform merge encoding in subsequent steps based on the frequency of occurrence information recorded in the count member.

[0159] Step S1313: Determine whether the value of the count member in the found merged word data structure instance is greater than or equal to the preset merging threshold; if the determination result is yes, step S1314 can be executed; otherwise, step S1316 can be executed.

[0160] In practical implementation, the value of the `count` member in the merged character data structure instance is compared with a preset merging threshold to determine whether the two currently adjacent large encoded characters have merging value, thereby further improving the compression ratio. The merging threshold can be set according to actual needs and is not limited here.

[0161] Step S1314: Allocate new merged encoding slots for the two currently adjacent large encoded words and create corresponding final word data structure instances.

[0162] In specific implementation, when it is determined that the value of the count member in the found merged word data structure instance is greater than or equal to the preset merging threshold, a new merged encoding slot is allocated to the two currently adjacent large encoded words, so as to merge the original two encoded values ​​with fixed length into one encoded value with the same fixed length, thereby further improving the compression ratio.

[0163] While allocating new merged encoding slots for the two currently adjacent large encoded words, corresponding final word data structure instances are created for the two currently adjacent large encoded words to record the merged encoding information of the two currently adjacent large encoded words for use in subsequent decompression.

[0164] Step S1315: Replace the two adjacent large coded characters in the current text to be merged with the corresponding merging coded slot coded values.

[0165] In practice, after allocating corresponding merging encoding slots to the two currently adjacent large encoded characters, all the two currently adjacent large encoded characters in the current text to be merged can be replaced with the encoding values ​​of the allocated merging encoding slots to achieve further compression of the current text to be merged.

[0166] Step S1316: Determine whether the current text to be merged has been traversed completely; if the result is yes, step S1318 can be executed; otherwise, step S1317 can be executed.

[0167] Step S1317: Obtain the two next adjacent large coded words in the current text to be merged as the two next adjacent large coded words, and restart from step S1312.

[0168] In specific implementation, when the current text to be merged has not been traversed, the next two adjacent large encoded characters in the current text to be merged are obtained as the two adjacent large encoded characters, and the execution starts again from step S1312 until the current text to be merged is traversed and a round of large encoded character merging and encoding operation is completed.

[0169] Step S1318: Determine whether the preset stop merging condition is met; if the determination result is yes, the operation can be ended; otherwise, step S1319 can be executed.

[0170] In practice, the number of words included in a text to be compressed that appear multiple times in the same order may be greater than two. Therefore, the above-mentioned word merging operation can be performed in multiple rounds.

[0171] Specifically, taking the string "trace_event_raw_event_" appearing in the text to be compressed as an example, when performing one round of merging encoding on the current text to be merged, "trace_" and "event_" are merged; then, another round of merging is performed, merging the merged "trace_event_" with the following "raw_"; subsequently, the next round of merging is performed, merging the merged "trace_event_raw_" with the following "event_". This process is repeated multiple times to further improve the compression ratio.

[0172] In theory, the number of rounds of word merging, i.e., the threshold for the number of merging rounds, should not exceed the maximum number of words contained in a single symbol in the text to be compressed. The maximum number of words contained in a single symbol in the text to be compressed can be obtained during the process of constructing the keyword list.

[0173] Furthermore, when merging and encoding keywords of the same type, new encoding slots need to be allocated to the merged words. Therefore, when the encoding slots are full, that is, when all encoding slots have been allocated, merging cannot continue.

[0174] Therefore, the stop merging condition set in this embodiment of the invention is any one of the following three:

[0175] (1) The number of rounds to be merged is greater than the corresponding round number threshold;

[0176] (2) The count member values ​​of the merged word data structure instances in the adjacent-joining red-black tree of the corresponding type of keywords in the first compressed text are all less than the merging threshold.

[0177] (3) The coding slots have been allocated.

[0178] Step S1319: Take the next text to be merged as the current text to be merged, and execute step S1311 to start execution again.

[0179] When any of the above conditions for stopping merging are met, the operation of merging and encoding keywords of the corresponding type can be stopped, and the second compressed text can be obtained.

[0180] Please continue reading Figure 1 In one embodiment of the present invention, to further improve the compression ratio, after obtaining the second compressed text, the method further includes:

[0181] Step S140: Replace the other unencoded symbols in the second compressed text with their corresponding ASCII code values ​​to obtain the corresponding third compressed text.

[0182] In practical applications, after the above steps S110 to S140, the words and corresponding delimiters in the text to be compressed have been replaced with the corresponding large-encoded characters, but there are still some uncompressed symbols.

[0183] Therefore, in order to further improve the compression ratio, the uncompressed symbols in the second compressed text can be replaced with their corresponding ASCII code values ​​to obtain the corresponding third compressed text.

[0184] Please continue reading Figure 1 To further improve the compression ratio, after obtaining the corresponding third compressed text, the method may further include:

[0185] Step S150: Compress the ASCII code values ​​in the third compressed text using the Linux kernel symbol table compression algorithm.

[0186] In this application, the ASCII code values ​​corresponding to the uncompressed symbols are used as small codewords to distinguish them from the aforementioned large codewords. The remaining small codewords in the third compressed text are then compressed using the Linux kernel symbol table compression algorithm to further improve the compression ratio. For details, please refer to [link to relevant documentation]. Figure 8 .

[0187] Figure 8 This diagram illustrates a method for compressing ASCII values ​​in the third compressed text according to an embodiment of the present invention. Please refer to... Figure 8 A method for compressing ASCII code values ​​in the third compressed text may specifically include the following steps:

[0188] Step S151: Construct a two-dimensional array token_profit to count the occurrences of unencoded characters in the third compressed text.

[0189] In one embodiment of the present invention, a method for counting the occurrences of unencoded characters in the third compressed text by constructing a two-dimensional array token_profit is described in detail below. Figure 9 .

[0190] See Figure 9 The method for constructing a two-dimensional array token_profit to count the occurrences of unencoded characters in the third compressed text in this embodiment of the invention may specifically include:

[0191] Step S1511: Traverse the third compressed text and obtain the two currently adjacent characters.

[0192] Step S1512: Using the ASCII codes of the two adjacent characters traversed to the current position as row and column indices respectively, increase the occurrence count of the corresponding member in the two-dimensional array token_profit by a preset third value.

[0193] See Figure 10 In the two-dimensional array token_profit, the values ​​on the horizontal and vertical axes represent the ASCII codes of the corresponding characters, and the letters below the values ​​on the horizontal and vertical axes represent the characters corresponding to the ASCII codes. The member values ​​in the two-dimensional array token_profit represent the number of times the letter combinations on the horizontal and vertical axes appear, i.e., the optimization values.

[0194] For example, the optimized value for the two consecutive characters "le" represented in row 101 and column 108 is 83, indicating that the consecutive characters "le" appear 83 times; the optimized value for the two consecutive characters "ki" represented in row 105 and column 107 is 217, indicating that the two consecutive characters "ki" appear 217 times in the third compressed text. Furthermore, blank squares in the two-dimensional array token_profit indicate that the corresponding character combinations did not appear in the third compressed text.

[0195] Step S1513: Determine whether the third compressed text has been traversed completely; if the determination result is yes, the operation can be ended; otherwise, step S1514 can be executed.

[0196] Step S1514: Obtain the next two adjacent characters as the current two adjacent characters to be traversed, and restart execution from step S1512.

[0197] In specific implementation, when it is determined that the third compressed text has not been traversed completely, the next two adjacent characters can be obtained from the third compressed text in sequence as the current two adjacent characters to be traversed, and the execution can be restarted from step S1512 until all uncoded characters in the third compressed text have been traversed completely.

[0198] By executing steps S1511 to S1514, the number of occurrences of consecutive characters in the third compressed text can be recorded in a two-dimensional array token_profit. This token_profit can then be used to further compress the small coded characters in the third compressed text.

[0199] Execute step S152 to construct a two-dimensional array best_table and a two-dimensional length array best_table_len.

[0200] In a specific implementation, a two-dimensional array best_table and a two-dimensional length array best_table_len are constructed to allocate encoding slots for unencoded characters in the third compressed text in the subsequent process.

[0201] In this embodiment of the invention, the two-dimensional array best_table is an unsigned character type two-dimensional array used to record the information of the encoding slot and the two consecutive characters that are assigned to the encoding slot; the two-dimensional array best_table_len is used to record the information of the encoding slot to be assigned.

[0202] It should be noted that the values ​​of the members in the two-dimensional array `best_table_len` can be divided into three types: First, when the number of characters corresponding to the encoding slot is 1, the value of the member in the two-dimensional array `best_table_len` is set to the preset fourth value; second, when the number of characters corresponding to the encoding slot is 2, the value of the member in the two-dimensional array `best_table_len` is set to the preset fifth value; third, when the encoding slot has not been allocated, the value of the member in the two-dimensional array `best_table_len` is set to the preset sixth value. Therefore, by examining the information in the values ​​of the members in the two-dimensional array `best_table_len`, it is possible to determine whether the corresponding encoding has been assigned to the corresponding character, and to determine whether the character corresponding to the assigned encoding slot is a single character or a string consisting of two characters.

[0203] In step S153, in the two-dimensional array best_table, a corresponding encoding slot is assigned to each unencoded single character appearing in the third compressed text.

[0204] In practical applications, after constructing the two-dimensional array token_profit, the third compressed text is traversed once more. The ASCII code value of the character appearing in the third compressed text is used as the index, and the corresponding encoding slot in the two-dimensional array best_table is assigned to the corresponding character. The value of the corresponding member in the best_table_len array is set to the fourth value to indicate that the encoding slot has been occupied and the corresponding number of characters is 1.

[0205] See Figure 11 In practical applications, the characters included in the C language naming convention are "0" to "9", "A" to "Z", and "a" to "z". Based on the ASCII code values ​​of these characters, the corresponding encoding slots in the two-dimensional array `best_table` are allocated according to the ASCII code value of each character.

[0206] Step S154: In the two-dimensional array best_table, assign corresponding encoding slots to consecutive characters that appear multiple times in the third compressed text.

[0207] In practice, after step S163 is completed, the remaining encoding slots in the two-dimensional array best_table can be allocated to consecutive characters that appear multiple times in the third compressed text. See [link to relevant documentation] for details. Figure 12 .

[0208] Figure 12 A flowchart illustrating a method for allocating remaining encoding slots in a two-dimensional array `best_table` to consecutive characters that appear multiple times in a third compressed text, according to an embodiment of the present invention, is shown. See also... Figure 12 A method for allocating the remaining encoding slots in a two-dimensional array `best_table` to consecutive characters that appear multiple times in a third compressed text, specifically including:

[0209] Step S1541: Obtain the maximum optimized value from the two-dimensional array token_profit.

[0210] In practice, the optimized values ​​of the members in the two-dimensional array `token_profit` represent the number of times the corresponding two consecutive characters appear in the third compressed text. Therefore, the larger the optimized value in the two-dimensional array `token_profit`, the more frequently the corresponding two characters appear. Prioritizing the allocation of encoding slots to these two characters helps improve the compression ratio.

[0211] Step S1542: Determine whether the obtained maximum optimized value is greater than the preset optimization threshold; if the determination result is yes, step S1543 can be executed; otherwise, the operation can be terminated.

[0212] In practice, the optimization threshold can be set according to actual needs and is not limited here. It is understood that when there are sufficient encoding slots, a larger optimization threshold will result in fewer two characters that can be optimized; conversely, a smaller threshold will result in more two characters that can be optimized.

[0213] Step S1543: Set Ca and Cb to two consecutive characters corresponding to the row index and column index of the maximum optimized value, respectively.

[0214] In practice, Ca and Cb are used to record the characters corresponding to the row and column indices of the largest optimized value in the two-dimensional array token_profit.

[0215] Step S1544: Traverse the two-dimensional array best_table in reverse order to obtain the currently available coding slot.

[0216] In practical applications, some encoding slots in the two-dimensional array `best_table` have already been allocated to "0" to "9", "A" to "Z", and "a" to "z". The encoding slots allocated to "0" to "9", "A" to "Z", and "a" to "z" are located at the beginning of the two-dimensional array `best_table`. In other words, the free encoding slots in the two-dimensional array `best_table_len` are located at the end of the two-dimensional array `best_table`. Therefore, traversing the two-dimensional array `best_table` in reverse order from back to front is beneficial for quickly finding free encoding slots, thereby helping to further improve compression efficiency.

[0217] Step S1545: In the two-dimensional array best_table_len, mark the current free encoding slot as allocated, and set the corresponding member in the two-dimensional array best_table to the two consecutive characters corresponding to Ca and Cb.

[0218] In practice, by marking the currently free encoding slots as allocated in the two-dimensional array best_table_len, and setting the corresponding members in the two-dimensional array best_table to the two consecutive characters corresponding to Ca and Cb, the encoding slots in the two-dimensional array best_table can be allocated to the two consecutive characters corresponding to Ca and Cb.

[0219] Step S1546: Replace the two consecutive characters corresponding to Ca and Cb in the third compressed text with the encoding value corresponding to the currently free encoding slot traversed.

[0220] In practice, after allocating corresponding encoding slots for the two consecutive characters corresponding to Ca and Cb, the two consecutive characters corresponding to Ca and Cb in the third compressed text can be replaced with the encoding value corresponding to the currently free encoding slot to compress the two consecutive characters corresponding to Ca and Cb in the third compressed text.

[0221] Step S1547: Set the optimized values ​​of the corresponding members Ca and Cb in the two-dimensional array token_profit to preset values, so as to serve as compression tokens for the two consecutive characters corresponding to Ca and Cb.

[0222] In specific implementation, after allocating the corresponding encoding slots to the two consecutive characters corresponding to the member with the largest optimized value in the two-dimensional array token_profit, the optimized value of the member is set to a preset value. Specifically, the optimized value of the member is reduced to the preset value as a compression marker for the two consecutive characters corresponding to Ca and Cb. Thus, the optimized values ​​of the members in the two-dimensional array token_profit can be traversed in descending order of optimized value.

[0223] Step S1548: Determine whether there are any free coding slots in the two-dimensional array best_table; if the result is yes, step S1541 can be executed; otherwise, the operation can be terminated.

[0224] In practice, when there are free encoding slots in the two-dimensional array best_table, the execution can restart from step S1542 to continue allocating encoding slots to the two consecutive characters corresponding to the members in the two-dimensional array token_profit; otherwise, the encoding slots in the two-dimensional array best_table_len have been exhausted, and the operation can be terminated.

[0225] Accordingly, this embodiment of the invention also provides a text compression module.

[0226] Figure 13 A schematic diagram of the structure of a text compression module according to an embodiment of the present invention is shown. See also Figure 13 A text compression module may include a list construction unit 1301 and a first compression unit 1302, wherein:

[0227] The list construction unit 1302 is adapted to perform word segmentation on the text to be compressed to obtain corresponding words, and to perform word length and word frequency statistics on the words obtained after word segmentation to construct a corresponding keyword list. In one embodiment of the present invention, the list construction unit 1302 is adapted to divide the text to be compressed using delimiters to obtain multiple corresponding words.

[0228] The first compression unit 1302 is adapted to compress the text to be compressed based on the constructed keyword list.

[0229] In one embodiment of the present invention, the list construction unit 1301 is adapted to perform word length and word frequency statistics on the words obtained after word segmentation based on a preset keyword data structure and a keyword pointer array, and construct a corresponding keyword list. The keyword data structure may include: a `next` member, used to point to a hash singly linked list pointer of the next word with the same hash value; a `word` member, used to record the corresponding word; a `total_cnt` member, used to record the occurrence count of the corresponding word; a `word_len` member, used to record the symbol length of the corresponding word; and an `idx_type` member, used to record the encoding value of the corresponding word.

[0230] Specifically, the list construction unit 1301 is adapted to traverse the text to be compressed, obtain the current word reached during traversal; calculate the hash value corresponding to the current word; use the calculated hash value as an index to determine whether there is a keyword data structure instance of the current word in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array; when it is determined that the current word exists in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array, the total_cnt member of the corresponding keyword data structure instance is increased by a preset first value; when it is determined that the current word does not exist in the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array, a corresponding keyword data structure instance is constructed for the current word, and the created keyword data structure instance is inserted into the keyword singly linked list pointed to by the corresponding pointer in the keyword pointer array; obtain the next word as the current word reached during traversal, and restart the execution from the step of calculating the hash value of the current word until the traversal of the text to be compressed is completed.

[0231] In another embodiment of the present invention, the first compression unit 1302 is adapted to traverse the text to be compressed and obtain the current word encountered during traversal; combine the current word encountered with the separator at the corresponding position to form a keyword of the corresponding type; determine whether the keyword of the corresponding type has been allocated an encoding slot based on the value of the idx_type member in the keyword data structure instance corresponding to the current word; when it is determined that the keyword of the corresponding type has not been allocated an encoding slot and there is an empty encoding slot, allocate an encoding slot for the keyword of the corresponding type and record the corresponding encoding value in the idx member of the corresponding keyword data structure instance. In the _type member; when the corresponding keyword of the same type has been assigned an encoding slot, obtain the encoding value recorded in the idx_type member of the corresponding keyword data structure instance; replace the corresponding keyword of the same type in the text to be compressed with the encoding value recorded in the idx_type member of the corresponding keyword data structure instance; obtain the next word in the text to be compressed as the current word to be traversed, and restart the step of combining the current word to be traversed with the separator at the corresponding position to form the corresponding keyword, until the text to be compressed is traversed completely, and obtain the corresponding first compressed text.

[0232] In a specific implementation, the text compression module may further include an instance construction unit 1303, wherein:

[0233] The instance construction unit 1303 is adapted to construct a final character data structure instance of the corresponding type of keyword based on a preset final character data structure after allocating encoding slots for the corresponding type of keyword, so as to record the encoding information of the corresponding type of keyword.

[0234] In one embodiment of the present invention, the final word data structure includes: a Str member for recording information of the corresponding type of keyword; a Len member for recording the character length of the corresponding type of keyword; a cnt member for recording the number of occurrences of the corresponding type of keyword; a node_cnt member for recording the number of different corresponding type of keywords adjacent to the corresponding type of keyword; and a tree member for recording the adjacency union red-black tree of the corresponding type of keyword.

[0235] In a specific implementation, the text compression module may further include a second compression unit 1304, wherein:

[0236] The second compression unit 1304 is adapted to merge and encode the large coded characters in the first compressed text after obtaining the first compressed text, so as to obtain the second compressed text.

[0237] In one embodiment of the present invention, the second compression unit 1304 is adapted to construct an adjacency-joining red-black tree for each of the large coded words in the first compressed text based on a preset merged word data structure; and to merge and encode the large coded words in the first compressed text based on the constructed adjacency-joining red-black tree.

[0238] Specifically, the second compression unit 1304 is adapted to scan the first compressed text to obtain the two currently adjacent large encoded words traversed; the two currently adjacent large encoded words include the preceding large encoded word and the following large encoded word; using the tree member in the final word data structure corresponding to the preceding large encoded word as the root, a corresponding adjacency merging red-black tree is determined; in the determined adjacency merging red-black tree, a merged word data structure instance with the key value of the following large encoded word is queried; when it is determined that a corresponding merged word data structure instance exists, the count member in the corresponding merged word data structure instance is incremented by a preset second value; when it is determined that no corresponding merged word data structure instance exists, a corresponding merged word data structure instance is created, and the created merged word data structure instance is inserted into the adjacency merging red-black tree of the preceding large encoded word; the next two adjacent large encoded words in the first compressed text are obtained, and the step of determining the corresponding adjacency merging red-black tree with the tree member in the final word data structure corresponding to the preceding large encoded word as the root is restarted until the first compressed text traversal is completed.

[0239] Specifically, the second compression unit 1304 is adapted to take the first compressed text as the current text to be merged, and traverse the current text to be merged to obtain the two currently adjacent large encoded words; in the adjacency merge red-black tree of the first large encoded word among the two currently adjacent large encoded words, search for a merged word data structure instance whose key value is the second large encoded word; when it is determined that the value of the count member in the found merged word data structure instance is greater than or equal to a preset merging threshold, allocate new merged encoding slots for the two currently adjacent large encoded words, and create corresponding final word data structure instances; replace the two currently adjacent large encoded words in the current text to be merged with the encoding values ​​of the corresponding merged encoding slots. The process involves: obtaining the two next adjacent large encoded characters in the current text to be merged; re-executing the step of searching for the merged character data structure instance with the key value of the next large encoded character in the adjacency merge red-black tree of the first large encoded character in the current two adjacent large encoded characters; and continuing this process until the current text to be merged is traversed completely to obtain the next text to be merged. The process then determines whether a preset merging stop condition has been met. If the merging stop condition has not been met, the next text to be merged is taken as the current text to be merged, and the process of traversing the current text to obtain the two next adjacent large encoded characters is restarted until the merging stop condition is met to obtain the second compressed text. The preset merging stop condition is any one of the following: the number of merging rounds is greater than the corresponding round threshold; the value of the count member of the merged character data structure instance in the adjacency merge red-black tree of the corresponding keyword type in the first compressed text is less than the merging threshold; and the encoding slots have been fully allocated.

[0240] In this embodiment of the invention, the corresponding type of keywords may include: a first type of keywords, including the corresponding word and the separator after the word; a second type of keywords, including the corresponding separator and the word after the separator; a third type of keywords, including the corresponding word and the separators before and after the word; and a fourth type of keywords, including only the corresponding word.

[0241] In a specific implementation, the text compression module may further include a deletion unit 1305, wherein:

[0242] The deletion unit 1305 is adapted to delete keywords that do not meet preset encoding conditions from the keyword list before assigning corresponding encoding slots to the keywords in the keyword list. In one embodiment of the present invention, the encoding conditions include: n×L>(2+L); where n represents the frequency of occurrence of the word, and L represents the symbol length of the word.

[0243] In a specific implementation, the text compression module may further include a third compression unit 1306, wherein:

[0244] The third compression unit 1306 is adapted to replace other unencoded symbols in the second compressed text with their corresponding ASCII code values ​​after obtaining the second compressed text, so as to obtain the corresponding third compressed text.

[0245] In a specific implementation, the text compression module may further include a fourth compression unit 1307, wherein:

[0246] The fourth compression unit 1307 is adapted to compress the ASCII code values ​​in the third compressed text using a Linux kernel symbol table compression algorithm after obtaining the third compressed text.

[0247] Accordingly, embodiments of the present invention also provide a chip, the chip including the aforementioned text compression module. The text compression module is described in detail in the foregoing sections and will not be repeated here.

[0248] Accordingly, embodiments of the present invention also provide an electronic device, which includes the aforementioned chip. The text compression module is described in detail in the foregoing sections and will not be repeated here.

[0249] An optional hardware structure for the electronic device provided in this embodiment of the invention can be as follows: Figure 14 As shown, it includes: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0250] In this embodiment of the invention, the number of processor 01, communication interface 02, memory 03 and communication bus 04 is at least one, and processor 01, communication interface 02 and memory 03 communicate with each other through communication bus 04.

[0251] Communication interface 02 can be an interface for a communication module used for network communication, such as the interface of a GSM module.

[0252] Processor 01 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0253] Memory 03 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0254] The memory 03 stores one or more computer instructions, which are executed by the processor 01 to implement the auxiliary graphics configuration method of the present invention.

[0255] It should be noted that the electronic device implemented above may also include other devices (not shown) that may not be essential to understanding the content disclosed in the embodiments of the present invention; given that these other devices may not be essential to understanding the content disclosed in the embodiments of the present invention, the embodiments of the present invention will not describe them one by one.

[0256] This invention also provides a storage medium storing one or more computer instructions, which are used to implement the static identifier region detection method provided in this invention.

[0257] The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise stated, the elements or features described are optional. Individual elements or features may be practiced without combination with other elements or features. Furthermore, embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some constructions of any embodiment may be included in another embodiment and may be replaced by corresponding constructions of another embodiment. It will be apparent to those skilled in the art that claims in the appended claims that are not expressly referenced to each other may be combined to form embodiments of the present invention, or may be included as new claims in amendments made after the filing of this application.

[0258] Embodiments of the present invention can be implemented by various means, such as hardware, firmware, software, or combinations thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0259] In firmware or software configuration, embodiments of the present invention can be implemented in the form of modules, processes, functions, etc. Software code can be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and can send data to and receive data from the processor via various known means.

[0260] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is accorded the widest scope consistent with the principles and novel features disclosed herein.

[0261] Accordingly, embodiments of the present invention also provide a storage medium storing one or more computer instructions for implementing the text compression method described above. The text compression method is detailed in the foregoing description and will not be repeated here.

[0262] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A text compression method, characterized in that, include: The text to be compressed is segmented into words, and the word length and frequency of the segmented words are statistically analyzed to construct a corresponding keyword list. This includes: constructing a keyword pointer array, which has multiple members, each member having a pointer head pointing to a keyword singly linked list indexed by the hash value of the word; for each word obtained from the segmentation of the text to be compressed, the corresponding keyword singly linked list is searched using the word's hash value as the pointer head in the keyword pointer array, and each keyword structure instance in the keyword singly linked list has a pointer head pointing to the next word with the same hash value; when a keyword data structure instance of a word exists in the found keyword singly linked list, the word occurrence count in the keyword data structure instance is increased by a preset first value; otherwise, a corresponding keyword data structure instance is constructed for the word and inserted into the keyword singly linked list, and the keyword data structure instance also includes the word's frequency information; Based on the constructed keyword list, the text to be compressed is compressed.

2. The text compression method according to claim 1, characterized in that, The step of segmenting the text to be compressed includes: The text to be compressed is divided using delimiters to obtain multiple corresponding words.

3. The text compression method according to claim 1, characterized in that, The keyword data structure includes: The next member is a pointer to the hash singly linked list of words that have the same hash value; The `word` member is used to record the corresponding words; The total_cnt member is used to record the number of times the corresponding word appears; The word_len member is used to record the symbol length of the corresponding word; The idx_type member is used to record the encoding value of the corresponding word.

4. The text compression method according to claim 3, characterized in that, The compression of the text to be compressed based on the constructed keyword list includes: Iterate through the compressed text and get the current word reached during the iteration; Combine the current word encountered during the traversal with the corresponding separator to form keywords of the corresponding type; Based on the value of the idx_type member in the keyword data structure instance corresponding to the current word, determine whether the keyword of the corresponding type has been assigned an encoding slot; When it is determined that the keyword of the corresponding type has not been assigned a coding slot and there is an empty coding slot, a coding slot is assigned to the keyword of the corresponding type, and the corresponding coding value is recorded in the idx_type member of the corresponding keyword data structure instance; When the corresponding keyword has been assigned an encoding slot, retrieve the encoding value recorded in the idx_type member of the corresponding keyword data structure instance; Replace the corresponding keywords in the text to be compressed with the encoded value of the idx_type member record in the corresponding keyword data structure instance; The next word in the text to be compressed is obtained as the current word to be traversed, and the step of combining the current word to be traversed with the delimiter at the corresponding position to form the corresponding keyword is executed again until the text to be compressed is traversed and the corresponding first compressed text is obtained.

5. The text compression method according to claim 4, characterized in that, After assigning encoding slots to the corresponding keyword types, the following is also included: Based on the preset final character data structure, an instance of the final character data structure of the corresponding type of keyword is constructed to record the encoding information of the corresponding type of keyword.

6. The text compression method according to claim 5, characterized in that, The final word data structure includes: The Str member is used to record information about keywords of the corresponding type; The Len member is used to record the character length of the corresponding keyword. The cnt member is used to record the number of occurrences of the corresponding keyword type; The node_cnt member is used to record the number of different keywords of the corresponding type that are adjacent to the keyword of the corresponding type; The tree member is used to record information about the neighbor-join red-black tree of the corresponding keyword type.

7. The text compression method according to claim 6, characterized in that, Using the encoded values ​​of the corresponding keywords in the first compressed text as large encoded words, after obtaining the first compressed text, the method further includes: The large encoded characters in the first compressed text are merged and encoded to obtain the second compressed text.

8. The text compression method according to claim 7, characterized in that, The step of merging and encoding large encoded characters in the first compressed text includes: Based on the preset merged character data structure, construct an adjacency-joining red-black tree for each of the large encoded characters in the first compressed text; Based on the constructed neighbor-joining red-black tree, the large coded words in the first compressed text are merged and encoded.

9. The text compression method according to claim 8, characterized in that, The merged word data structure includes: The entry member is used to represent the root node of the adjacency-join red-black tree; The `code` member is used to encode the keyword of the corresponding type that is adjacent to the root node. The `count` member is used to record the number of times the corresponding keyword of the same type appears in the adjacent strings.

10. The text compression method according to claim 9, characterized in that, The construction of an adjacency-joining red-black tree for each large coded character in the first compressed text, based on a preset merged character data structure, includes: The first compressed text is scanned to obtain the two currently adjacent large encoded words; the two currently adjacent large encoded words include the preceding large encoded word and the following large encoded word; Using the tree member in the final word data structure corresponding to the first large encoded word as the root, determine the corresponding adjacency-joining red-black tree; In the determined adjacency-joining red-black tree, search for the merged word data structure instance whose key value is the latter large encoded word; When it is determined that a corresponding merged word data structure instance exists, the count member in the corresponding merged word data structure instance is incremented by a preset second value; When it is determined that there is no corresponding merged word data structure instance, a corresponding merged word data structure instance is created, and the created merged word data structure instance is inserted into the adjacent merged red-black tree of the previous large encoded word; Obtain the two next adjacent large encoded words in the first compressed text, and restart the process from the step of determining the corresponding adjacent red-black tree with the tree member in the final word data structure corresponding to the previous large encoded word as the root, until the first compressed text is traversed.

11. The text compression method according to claim 10, characterized in that, The process of merging and encoding large coded words in the first compressed text based on the constructed adjacency-joining red-black tree includes: The first compressed text is taken as the current text to be merged, and the current text to be merged is traversed to obtain the two adjacent large encoded words that are traversed to the current text. In the adjacency merge red-black tree of the first large codeword among the two currently adjacent large codewords, find the merged word data structure instance whose key value is the second large codeword; When it is determined that the value of the count member in the found merged word data structure instance is greater than or equal to the preset merging threshold, a new merged encoding slot is allocated to the two currently adjacent large encoded words, and the corresponding final word data structure instance is created. Replace the two adjacent large coded characters in the current text to be merged with the corresponding merging coded slot coded values; The next two adjacent large encoded words in the current text to be merged are obtained as the current two adjacent large encoded words. The step of searching for the merged word data structure instance with the key value of the next large encoded word in the adjacency merge red-black tree of the current two adjacent large encoded words is restarted until the current text to be merged is traversed and the next text to be merged is obtained. Determine whether the preset stop merging condition has been met; When it is determined that the stop merging condition has not been met, the next text to be merged is taken as the current text to be merged, and the step of traversing the current text to be merged and obtaining the two adjacent large coded words is restarted until the stop merging condition is met, and the second compressed text is obtained.

12. The text compression method according to claim 11, characterized in that, The preset stop merging condition is any one of the following: The number of merged rounds exceeds the corresponding round number threshold; The count member values ​​of the merged word data structure instances in the adjacent-joining red-black tree of the corresponding keyword type in the first compressed text are all less than the merging threshold. The coding slots have been allocated.

13. The text compression method according to any one of claims 4 to 12, characterized in that, The corresponding keyword types include: The first type of keywords includes the corresponding words and the separators following the words; The second type of keywords includes the corresponding separator and the words following the separator; The third type of keywords includes the corresponding words and the separators before and after the words; The fourth type of keywords only includes the corresponding words.

14. The text compression method according to claim 13, characterized in that, Before assigning corresponding coding slots to the keywords in the keyword list, the process also includes: Remove keywords from the keyword list that do not meet the preset encoding conditions.

15. The text compression method according to claim 14, characterized in that, The encoding conditions include: ； in, Indicates the frequency of word occurrence. Indicates the length of the word in characters.

16. The text compression method according to claim 11, characterized in that, After obtaining the second compressed text, it also includes: Replace the other unencoded symbols in the second compressed text with their corresponding ASCII code values ​​to obtain the corresponding third compressed text.

17. The text compression method according to claim 16, characterized in that, After obtaining the third compressed text, the process also includes: The ASCII values ​​in the third compressed text are compressed using the Linux kernel symbol table compression algorithm.

18. A text compression device, characterized in that, include: The list construction unit is suitable for performing word segmentation on the text to be compressed, and for performing word length and word frequency statistics on the words obtained after word segmentation to construct a corresponding keyword list. This includes: constructing a keyword pointer array, which includes multiple members, each member having a pointer head pointing to a keyword singly linked list indexed by the hash value of the word; for each word obtained from the segmentation of the text to be compressed, using the word's hash value as the pointer head in the keyword pointer array, searching for the corresponding keyword singly linked list, where each word's keyword structure instance in the keyword singly linked list has a pointer head pointing to the next word with the same hash value; when a keyword data structure instance of a word exists in the found keyword singly linked list, the word's occurrence count in the keyword data structure instance is increased by a preset first value; otherwise, a corresponding keyword data structure instance is constructed for the word and inserted into the keyword singly linked list, where the keyword data structure instance also includes word frequency information; The first compression unit is adapted to compress the text to be compressed based on the constructed keyword list.

19. The text compression device according to claim 18, characterized in that, The list construction unit is adapted to divide the text to be compressed using delimiters to obtain multiple corresponding words.

20. The text compression device according to claim 18, characterized in that, The keyword data structure includes: The next member is a pointer to the hash singly linked list of words that have the same hash value; The `word` member is used to record the corresponding words; The total_cnt member is used to record the number of times the corresponding word appears; The word_len member is used to record the symbol length of the corresponding word; The idx_type member is used to record the encoding value of the corresponding word.

21. The text compression device according to claim 20, characterized in that, The first compression unit is adapted to traverse the text to be compressed, obtain the current word reached during traversal; combine the current word reached during traversal with the delimiter at the corresponding position to form keywords of the corresponding type; and determine whether the keywords of the corresponding type have been allocated encoding slots based on the value of the idx_type member in the keyword data structure instance corresponding to the current word. When it is determined that the corresponding keyword type has no assigned encoding slot and there is a free encoding slot, an encoding slot is assigned to the corresponding keyword type, and the corresponding encoding value is recorded in the idx_type member of the corresponding keyword data structure instance; when the corresponding keyword type has been assigned an encoding slot, the encoding value recorded in the idx_type member of the corresponding keyword data structure instance is obtained; the corresponding keyword type in the text to be compressed is replaced with the encoding value recorded in the idx_type member of the corresponding keyword data structure instance; the next word in the text to be compressed is obtained as the current word to be traversed, and the step of combining the current word to be traversed with the separator at the corresponding position to form the corresponding keyword type is restarted until the text to be compressed is traversed completely, and the corresponding first compressed text is obtained.

22. The text compression device according to claim 21, characterized in that, Also includes: The instance construction unit is adapted to construct a final character data structure instance of the corresponding type of keyword based on a preset final character data structure after allocating encoding slots for the corresponding type of keyword, so as to record the encoding information of the corresponding type of keyword.

23. The text compression device according to claim 22, characterized in that, The final word data structure includes: The Str member is used to record information about keywords of the corresponding type; The Len member is used to record the character length of the corresponding keyword. The cnt member is used to record the number of occurrences of the corresponding keyword type; The node_cnt member is used to record the number of different keywords of the corresponding type that are adjacent to the keyword of the corresponding type; The tree member is used to record the neighbor-join red-black tree of the corresponding keyword type.

24. The text compression device according to claim 23, characterized in that, Also includes: The second compression unit is adapted to use the encoded values ​​of the corresponding keywords in the first compressed text as large encoded words. After obtaining the first compressed text, the large encoded words in the first compressed text are merged and encoded to obtain the second compressed text.

25. The text compression device according to claim 24, characterized in that, The second compression unit is adapted to construct an adjacency-joining red-black tree for each of the large coded words in the first compressed text based on a preset merged word data structure; Based on the constructed neighbor-joining red-black tree, the large coded words in the first compressed text are merged and encoded.

26. The text compression device according to claim 25, characterized in that, The merged word data structure includes: The entry member is used to represent the root node of the adjacency-join red-black tree; The `code` member is used to encode the keyword of the corresponding type that is adjacent to the root node. The `count` member is used to record the number of times the corresponding keyword of the same type appears in the adjacent strings.

27. The text compression apparatus according to claim 26, characterized in that, The second compression unit is adapted to scan the first compressed text to obtain the two currently adjacent large encoded words that are traversed; the two currently adjacent large encoded words include the preceding large encoded word and the following large encoded word; the corresponding adjacency union red-black tree is determined with the tree member in the final word data structure corresponding to the preceding large encoded word as the root; In the determined adjacency-joining red-black tree, search for the merged word data structure instance whose key value is the latter large encoded word; When it is determined that a corresponding merged word data structure instance exists, the count member in the corresponding merged word data structure instance is incremented by a preset second value; when it is determined that no corresponding merged word data structure instance exists, a corresponding merged word data structure instance is created, and the created merged word data structure instance is inserted into the adjacency merged red-black tree of the previous large encoded word. Obtain the two next adjacent large encoded words in the first compressed text, and restart the process from the step of determining the corresponding adjacent red-black tree with the tree member in the final word data structure corresponding to the previous large encoded word as the root, until the first compressed text is traversed.

28. The text compression device according to claim 27, characterized in that, The second compression unit is adapted to take the first compressed text as the current text to be merged, and to traverse the current text to be merged to obtain the two adjacent large encoded words that are currently traversed; in the adjacency merge red-black tree of the two adjacent large encoded words, to search for the merged word data structure instance whose key value is the second large encoded word. When it is determined that the value of the count member in the found merged character data structure instance is greater than or equal to the preset merge threshold, a new merged encoding slot is allocated to the two currently adjacent large encoded characters, and a corresponding final character data structure instance is created; the two currently adjacent large encoded characters in the current text to be merged are replaced with the encoding value of the corresponding merged encoding slot. The next two adjacent large encoded words in the current text to be merged are obtained as the current two adjacent large encoded words. The step of searching for the merged word data structure instance with the key value of the next large encoded word in the adjacency merge red-black tree of the current two adjacent large encoded words is restarted until the current text to be merged is traversed and the next text to be merged is obtained. It is then determined whether the preset merging stop condition has been met. When it is determined that the stop merging condition has not been met, the next text to be merged is taken as the current text to be merged, and the step of traversing the current text to be merged and obtaining the two adjacent large coded characters is restarted until the stop merging condition is met, thus obtaining the second compressed text.

29. The text compression device according to claim 28, characterized in that, The preset stop merging condition is any one of the following: The number of merged rounds exceeds the corresponding round number threshold; The count member values ​​of the merged word data structure instances in the adjacent-joining red-black tree of the corresponding keyword type in the first compressed text are all less than the merging threshold. The coding slots have been allocated.

30. The text compression apparatus according to any one of claims 21 to 29, characterized in that, The corresponding keyword types include: The first type of keywords includes the corresponding words and the separators following the words; The second type of keywords includes the corresponding separator and the words following the separator; The third type of keywords includes the corresponding words and the separators before and after the words; The fourth type of keywords only includes the corresponding words.

31. The text compression device according to claim 30, characterized in that, Also includes: The deletion unit is adapted to delete keywords that do not meet preset encoding conditions from the keyword list before assigning corresponding encoding slots to the keywords in the keyword list.

32. The text compression device according to claim 31, characterized in that, The encoding conditions include: ； in, Indicates the frequency of word occurrence. The length of the symbol representing a word.

33. The text compression device according to claim 28, characterized in that, Also includes: The third compression unit is adapted to replace other unencoded symbols in the second compressed text with their corresponding ASCII code values ​​after obtaining the second compressed text, so as to obtain the corresponding third compressed text.

34. The text compression device according to claim 33, characterized in that, Also includes: The fourth compression unit is adapted to compress the ASCII code values ​​in the third compressed text using a Linux kernel symbol table compression algorithm after obtaining the third compressed text.

35. A chip, characterized in that, The chip includes a text compression device as described in any one of claims 18 to 34.

36. An electronic device, characterized in that, Includes the chip as described in claim 35.

37. A storage medium, characterized in that, The storage medium stores one or more computer instructions, which are used to implement the text compression method as described in any one of claims 1-17.

Images