A method for quickly compressing local archive-related data

Abstract

The present invention relates to the technical field of data processing, and in particular to a method for quickly compressing local archive-related data, comprising: dividing all other words of any word into clusters with high similarity and clusters with low similarity; obtaining the importance of each word in the text according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity and the frequency of each word; obtaining the probability value of any two words appearing at the same time in the text according to the frequency of any two words appearing at the same time in the text and the number of all words; obtaining the frequency weight index of each word in the text according to the frequency of any two words appearing at the same time in the text, the probability value and the frequency weight coefficient of each word; adjusting the frequency of each word appearing according to the frequency weight index of each word in the text, and constructing a Huffman tree for data compression. The present invention improves the compression efficiency of text data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for quickly compressing local archive-related data. Background Art

[0002] When dealing with the backup and data compression of local document files, Huffman coding is usually considered to optimize storage space. Huffman coding is a prefix coding method based on the frequency of symbol occurrence. It can assign shorter codes to frequently occurring symbols to achieve data compression. Since locally archived files are often closely related in content, Huffman coding does not take into account the contextual relevance of text content and the existence of words with similar meanings but different words in the document. Relying only on symbol frequency is not enough to capture the importance of the text, and thus cannot achieve the optimal compression effect. Summary of the invention

[0003] The present invention provides a method for quickly compressing local archive-related data to solve the problem that Huffman coding does not take into account the contextual relevance of text content and the situation that there are words with similar meanings but different words in a document, and only relies on symbol frequency, which is insufficient to capture the importance of the text, and thus cannot achieve the optimal compression effect.

[0004] A local archive-related data rapid compression method of the present invention adopts the following technical solution:

[0005] An embodiment of the present invention provides a method for quickly compressing local archive-related data, the method comprising the following steps:

[0006] Get several words in the text and the frequency of each word;

[0007] Obtain the semantic similarity between any two words in the text, and divide all other words of any word into clusters with high similarity and clusters with low similarity; obtain the importance of each word in the text according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity, as well as the frequency of occurrence of each word;

[0008] According to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of each word in the cluster with high similarity, the number of all words and the importance of each word in the text, the frequency weight coefficient of each word in the text is obtained; the frequency of simultaneous occurrence of any two words in the text is obtained; according to the frequency of simultaneous occurrence of any two words in the text and the number of all words, the probability value of simultaneous occurrence of any two words in the text is obtained; according to the frequency of simultaneous occurrence of any two words in the text, the probability value and the frequency weight coefficient of each word, the frequency weight index of each word in the text is obtained;

[0009] The frequency of each word in the text is adjusted according to the frequency weight index of each word, and a Huffman tree is constructed for data compression.

[0010] Preferably, the obtaining of the semantic similarity between any two words in the text comprises the following specific steps:

[0011] The cosine similarity measurement method is used to obtain the semantic similarity between the vth word and each other word.

[0012] Preferably, the step of dividing all other words of any one word into clusters with high similarity and clusters with low similarity comprises the following specific steps:

[0013] Among the semantic similarities between the vth word and all other words, all other words whose similarities are greater than a preset judgment threshold b constitute a cluster with high similarity to the vth word, and all other words whose similarities are less than or equal to the preset judgment threshold b constitute a cluster with low similarity to the vth word.

[0014] Preferably, the importance of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity and the frequency of occurrence of each word, and the specific steps include the following:

[0015] According to the sum of the occurrence frequencies of each word in the cluster with high similarity to any word in the text and the number of all words in the text, the proportion of words in the cluster with high similarity to any word in the text is obtained;

[0016] The importance of each word in the text is obtained based on the semantic similarity between any word in the text and each word in the clusters with high and low similarity, the proportion of words in the clusters with high similarity in the text, and the frequency of occurrence of each word.

[0017] Preferably, the importance of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity, the proportion of words in the cluster with high similarity in the text, and the frequency of occurrence of each word, including the specific formula as follows:

[0018]

[0019] In the formula, H v Indicates the importance of the vth word in the text; It represents the average value of the semantic similarity between all words in the cluster with the highest similarity between the vth word and the vth word in the text; N represents the average value of the semantic similarity between all words in the cluster with the lowest similarity between the vth word and the vth word in the text; v,i Indicates the frequency of occurrence of the i-th word in the cluster with high similarity to the v-th word in the text; n v N represents the number of all words in the cluster with high similarity to the vth word in the text; all Represents the number of all words in the text; exp() is an exponential function with a natural constant as the base; is the proportion of words in the text that are in the cluster with high similarity to the vth word in the text.

[0020] Preferably, the frequency weight coefficient of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of occurrence of each word in the cluster with high similarity, the number of all words and the importance of each word in the text, and the specific formula included is as follows:

[0021]

[0022] Where W v Represents the frequency weight coefficient of the vth word in the text; N v,i represents the frequency of the i-th word in the cluster with the highest similarity to the v-th word in the text; a represents the initial weight coefficient of the frequency of the v-th word in the text; n v Represents the number of all words in the cluster with high similarity to the vth word in the text; C v,i Indicates the semantic similarity between the vth word in the text and the i-th word in the cluster with the highest similarity; softmax() is the normalization function; H v Indicates the importance of the vth word in the text; the initial weight coefficient of the frequency of the vth word in the text is a preset first threshold.

[0023] Preferably, the obtaining of the frequency of simultaneous occurrence of any two words in the text comprises the following specific steps:

[0024] In the text, take one word as one unit, set a linear sliding window with N units along the writing direction of the text, and slide with one unit as the step until the whole text is traversed. The number of linear sliding windows that contain both the vth word and the sth word is recorded as the frequency of simultaneous occurrence of the vth word and the sth word, where N is the preset number of units.

[0025] Preferably, the probability value of any two words appearing at the same time in the text is obtained according to the frequency of any two words appearing at the same time in the text and the number of all words, and the specific formula included is as follows:

[0026]

[0027] Where P v,s represents the probability value of the vth word and the sth word appearing at the same time in the text; M v,s Indicates the frequency of the vth word and the sth word appearing together in the text; N is the preset number of units; N all Represents the number of all words in the text.

[0028] Preferably, the frequency weight index of each word in the text is obtained according to the frequency, probability value and frequency weight coefficient of each word appearing at the same time of any two words in the text, and the specific formula included is as follows:

[0029]

[0030] Where Y v represents the frequency weight index of the vth word in the text; z represents the initial frequency weight index of the vth word in the text; M v,s represents the frequency of the vth word and the sth word appearing together in the text; m represents the number of all words in the text except the vth word; P v,s N represents the probability value of the vth word and the sth word appearing at the same time in the text; v represents the frequency of the vth word in the text; W v represents the frequency weight coefficient of the vth word in the text; softmax() is the normalization function; N all Represents the number of all words in the text; the initial weight index of the vth word frequency in the text is the preset second threshold.

[0031] Preferably, the step of adjusting the frequency of occurrence of each word according to the frequency weight index of each word in the text and constructing a Huffman tree for data compression includes the following specific steps:

[0032] Multiply each word frequency weight index by the frequency of occurrence of each word to obtain the weighted frequency of occurrence of each word; based on the weighted frequency of occurrence of each word, use Huffman coding to construct a Huffman tree for data compression to obtain the compressed code of each word.

[0033] The beneficial effect of the technical solution of the present invention is that when archiving text, because there are often a large number of repeated words or phrases between related documents, these repeated elements are of high importance in historical texts. Through Huffman coding, words of high importance will be assigned shorter codes. In the subsequent document coding process, the use of short codes for words of high importance will further improve compression efficiency and achieve rapid compression. According to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of each word in the cluster with high similarity, the number of all words and the importance of each word in the text, the frequency weight coefficient of each word in the text is obtained; the frequency of any two words appearing at the same time in the text is obtained; according to the frequency of any two words appearing at the same time in the text and the number of all words, the probability value of any two words appearing at the same time in the text is obtained; according to the frequency of any two words appearing at the same time in the text and the number of all words, the frequency weight index of each word in the text is obtained. When using Huffman to encode words in a text, the similarity between words is taken into account. The weight index of each word is obtained based on the semantic similarity between each word and other words and the frequency of occurrence of the current word. Finally, a Huffman tree is established to achieve fast compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 The present invention is a flowchart of the steps of a method for quickly compressing local archive-related data. DETAILED DESCRIPTION

[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a local archive-related data rapid compression method proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0037] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0038] The following is a detailed description of a specific scheme of a method for quickly compressing local archive-related data provided by the present invention in conjunction with the accompanying drawings.

[0039] See also Figure 1 , which shows a flowchart of a method for quickly compressing local archive-related data provided by an embodiment of the present invention, the method comprising the following steps:

[0040] Step S001: Obtain a number of words in a text and the frequency of occurrence of each word.

[0041] The text data of the document is obtained through programming code, and the Stanford NLP library is used to decompose the text data of the document into separate words or phrases, and the frequency of occurrence of each word is obtained.

[0042] Among them, the Stanford NLP library is a well-known technology, and the specific method will not be introduced here.

[0043] What needs to be explained is that the text file that needs to be compressed is determined, and the text data in the file is determined. First, the irrelevant characters, punctuation marks, and special symbols in the text data are deleted, and then the Stanford NLP library is used to decompose the text data into separate words or phrases. If the text is in English, the English text needs to be converted into uniform uppercase and lowercase. The number of times each word appears in the document is counted for subsequent text analysis. Among them, all words constitute a text.

[0044] Get several words in the text and the frequency of each word.

[0045] Step S002: Obtain the semantic similarity between any two words in the text, and divide all other words of any word into clusters with high similarity and clusters with low similarity; obtain the importance of each word in the text according to the semantic similarity between any word in the text and each word in the clusters with high similarity and clusters with low similarity and the frequency of occurrence of each word.

[0046] It should be noted that the acquired words are annotated using the Stanford NLP library, the frequency of occurrence of each word is counted, and the word is added to the basic dictionary. The format of the basic dictionary is {"word":"frequency of occurrence"}.

[0047] It should be noted that when using Huffman coding, since words with high frequency are often more important in the text, it is generally determined that words with relatively high frequency use short codes, and words with relatively low frequency use long codes.

[0048] It should be noted that since similar but different words may appear in the text and the frequency of such words is low, but similar words appear frequently, Huffman coding does not consider the semantic relationship between words, including similarity, and therefore may not be able to fully utilize the semantic information in the text to optimize the coding. Therefore, it is necessary to determine the similarity between words and correct the frequency weight of the current word when performing Huffman coding based on the similarity between words.

[0049] It should be noted that similar or related words may appear in the text. When words similar to the current word appear or words with similar meanings appear at a high frequency, it means that the current word is also of great importance. The distribution of similar words can be determined, and the preliminary importance of the current word can be determined based on the similarity between similar words and the distribution and frequency of the current word.

[0050] The cosine similarity measurement method is used to obtain the semantic similarity between any two words in the text;

[0051] It should be noted that cosine similarity is a commonly used method to measure the similarity between two vectors. It is usually used to measure the semantic similarity between words in a text, and its value range is generally between -1 and 1.

[0052] For any word, calculate the similarity between any word and other words; obtain the semantic similarity between the vth word and each other word;

[0053] In the semantic similarity between the vth word and all other words, the preset judgment threshold b is 0.4, and all other words with similarities greater than the preset judgment threshold b constitute a cluster with high similarity to the vth word, and all other words with similarities less than or equal to the preset judgment threshold b constitute a cluster with low similarity to the vth word.

[0054] Among them, the cosine similarity measurement method is a well-known technology, and the specific method will not be introduced here.

[0055] What needs to be explained is that the importance of the current word is determined based on the distribution of the similarity between the current word and other words. After clustering the similarity between other words in the text and the current word using a clustering algorithm, if the words in the high-similarity internal cluster have a higher similarity with the current word, it means that the current word has similar words in the text, and the greater the frequency of similar words, the higher the importance of the current word. At the same time, if the similarity index in the low-similarity internal cluster of the current word is also relatively high, it means that the current word may be a core word in the text, and most words in the text are related to the current word.

[0056] It should be noted that, for any word, the text importance of the current word in the text is determined according to the occurrence frequency and similarity of the text in the cluster with high similarity and the cluster with low similarity.

[0057] The importance of the vth word in the text is calculated as follows:

[0058]

[0059] In the formula, H v Indicates the importance of the vth word in the text; It represents the average value of the semantic similarity between all words in the cluster with the highest similarity between the vth word and the vth word in the text; N represents the average value of the semantic similarity between all words in the cluster with the lowest similarity between the vth word and the vth word in the text; v,i Indicates the frequency of occurrence of the i-th word in the cluster with high similarity to the v-th word in the text; n v N represents the number of all words in the cluster with high similarity to the vth word in the text; all Represents the number of all words in the text; exp() is an exponential function with a natural constant as the base; is the proportion of words in the text that are in the cluster with high similarity to the vth word in the text.

[0060] What needs to be explained is that Indicates the proportion of words in the text with high similarity clusters, represents the average value of all similarities in the cluster with low similarity of the vth word, Indicates the cluster with low similarity and the cluster with high similarity of the vth word. The similarity difference of this cluster is relatively small; It represents the cluster with low similarity and the cluster with high similarity of the vth word. When the similarity difference of this cluster is relatively small, the similarity in the cluster with high similarity is also relatively high.

[0061] At this point, the importance of each word in the text is obtained.

[0062] Step S003: Obtain the frequency weight coefficient of each word in the text according to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of occurrence of each word in the cluster with high similarity, the number of all words and the importance of each word in the text; obtain the frequency of simultaneous occurrence of any two words in the text; obtain the probability value of simultaneous occurrence of any two words in the text according to the frequency of simultaneous occurrence of any two words in the text and the number of all words; obtain the frequency weight index of each word in the text according to the frequency of simultaneous occurrence of any two words in the text, the probability value and the frequency weight coefficient of each word.

[0063] It should be noted that the weight of the frequency of the current word in the text is adjusted according to the importance of the current word in the text and the words with high similarity to the current word to determine the weight index of the current word. Generally, the initial weight of the frequency of a word is 1.

[0064] The first threshold a is preset to 1, and the initial weight coefficient of the frequency of the vth word in the text is set to the preset first threshold a.

[0065] The frequency weight coefficient of the vth word in the text is calculated as follows:

[0066]

[0067] Where W v Represents the frequency weight coefficient of the vth word in the text; N v,i represents the frequency of the i-th word in the cluster with the highest similarity to the v-th word in the text; a represents the initial weight coefficient of the frequency of the v-th word in the text; n v Represents the number of all words in the cluster with high similarity to the vth word in the text; C v,i Indicates the semantic similarity between the vth word in the text and the i-th word in the cluster with the highest similarity; softmax() is the normalization function; H v Indicates the importance of the vth word in the text.

[0068] It should be noted that the above analysis only considers a single word to determine its importance in the text. Since words have fixed collocations in the text, especially in specific fields, their proper nouns may need to be used in combination, and the probability of multiple proper nouns appearing at the same time is very high. Generally, there are some words in the text whose word meaning similarity may not be very high, but the correlation and dependency between word meanings are very strong. Therefore, it is necessary to determine the coexisting words based on the contextual relationship in the text, determine the semantic correlation between the two based on the frequency of their co-occurrence, and then improve the frequency weight index of each word.

[0069] Taking the vth word and the sth word in the text as an example, in the text, take one word as a unit, set a linear sliding window with N units along the writing direction of the text, and slide with one unit as a step until the entire text is traversed. The number of linear sliding windows that contain both the vth word and the sth word is recorded as the frequency of simultaneous occurrence of the vth word and the sth word.

[0070] In this embodiment, N is a preset number of units, and N is 2, which is taken as an example for description.

[0071] The probability value of the vth word and the sth word appearing at the same time in the text is calculated as follows:

[0072]

[0073] Where P v,s represents the probability value of the vth word and the sth word appearing at the same time in the text; M v,s Indicates the frequency of the vth word and the sth word appearing together in the text; N is the preset number of units; N all Represents the number of all words in the text.

[0074] It should be noted that the frequency weight indexes of the two words are determined based on the above-mentioned co-occurrence probability values ​​of the two words and the difference between the single occurrence probabilities of the two words. If the probability of the two words appearing together is less different from the probability of the single words appearing, it means that the semantic correlation between the two words is very strong and its frequency weight index needs to be improved.

[0075] The second threshold z is preset to 1, and the initial weight index of the vth word frequency in the text is set to the preset second threshold; this is used as an example for description.

[0076] The calculation method of the frequency weight index of the vth word in the text is as follows:

[0077]

[0078] Where Y v represents the frequency weight index of the vth word in the text; z represents the initial frequency weight index of the vth word in the text; M v,s represents the frequency of the vth word and the sth word appearing together in the text; m represents the number of all words in the text except the vth word; P v,s N represents the probability value of the vth word and the sth word appearing at the same time in the text; v represents the frequency of the vth word in the text; W v represents the frequency weight coefficient of the vth word in the text; softmax() is the normalization function; N all Represents the number of all words in the text.

[0079] At this point, the frequency weight index of each word in the text is obtained.

[0080] Step S004: adjusting the frequency of occurrence of each word according to the frequency weight index of each word in the text, and constructing a Huffman tree for data compression.

[0081] According to the frequency weight index of each word in the text, the frequency of each word is adjusted, and the frequency weight index of each word is multiplied by the frequency of each word to obtain the weighted frequency of each word; according to the weighted frequency of each word, Huffman coding is used to construct a Huffman tree for data compression to obtain the compressed code of each word. Words with a large weighted frequency will obtain a shorter code, while words with a small weighted frequency will obtain a longer code.

[0082] It should be noted that the frequency weight index of each word reflects the importance of the word in the text. Each word in the document is replaced with its corresponding Huffman code. Since the words with higher frequency have shorter codes, the overall document length will be reduced, thereby achieving data compression.

[0083] It should be noted that in the scenario of file archiving, especially when the archived files are a series of related documents, there are often a large number of repeated words or phrases between related documents, and these repeated elements are of high importance in historical texts. Through Huffman coding, these high-frequency words will be given shorter codes, which not only reduces storage space during the initial encoding, but also in the subsequent document encoding process, due to the repetitiveness of these words, the use of short codes will further improve compression efficiency. This method can not only reduce the demand for storage space, but also speed up data transmission.

[0084] At this point, this embodiment is completed.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for quickly compressing local archive-related data, characterized in that: The method comprises the following steps: Get several words in the text and the frequency of each word; Obtain the semantic similarity between any two words in the text, and divide all other words of any word into clusters with high similarity and clusters with low similarity; obtain the importance of each word in the text according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity, as well as the frequency of occurrence of each word; According to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of each word in the cluster with high similarity, the number of all words and the importance of each word in the text, the frequency weight coefficient of each word in the text is obtained; the frequency of simultaneous occurrence of any two words in the text is obtained; according to the frequency of simultaneous occurrence of any two words in the text and the number of all words, the probability value of simultaneous occurrence of any two words in the text is obtained; according to the frequency of simultaneous occurrence of any two words in the text, the probability value and the frequency weight coefficient of each word, the frequency weight index of each word in the text is obtained; Adjust the frequency of each word according to the frequency weight index of each word in the text, and construct a Huffman tree for data compression; The importance of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity and the frequency of occurrence of each word, and the specific steps include the following: According to the sum of the occurrence frequencies of each word in the cluster with high similarity to any word in the text and the number of all words in the text, the proportion of words in the cluster with high similarity to any word in the text is obtained; The importance of each word in the text is obtained based on the semantic similarity between any word in the text and each word in the clusters with high and low similarity, the proportion of words in the clusters with high similarity in the text, and the frequency of each word. The importance of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the clusters with high similarity and low similarity, the proportion of words in the cluster with high similarity in the text, and the frequency of occurrence of each word. The specific formula included is as follows: In the formula, H v Indicates the importance of the vth word in the text; It represents the average value of the semantic similarity between all words in the cluster with the highest similarity between the vth word and the vth word in the text; N represents the average value of the semantic similarity between all words in the cluster with the lowest similarity between the vth word and the vth word in the text; v,i Indicates the frequency of occurrence of the i-th word in the cluster with high similarity to the v-th word in the text; n v N represents the number of all words in the cluster with high similarity to the vth word in the text; all Represents the number of all words in the text; exp() is an exponential function with a natural constant as the base; is the proportion of words in the text that are in the cluster with high similarity to the vth word in the text.

2. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The specific steps of obtaining the semantic similarity between any two words in the text are as follows: The cosine similarity measurement method is used to obtain the semantic similarity between the vth word and each other word.

3. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The method of dividing all other words of any word into clusters with high similarity and clusters with low similarity includes the following specific steps: Among the semantic similarities between the vth word and all other words, all other words whose similarities are greater than a preset judgment threshold b constitute a cluster with high similarity to the vth word, and all other words whose similarities are less than or equal to the preset judgment threshold b constitute a cluster with low similarity to the vth word.

4. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The frequency weight coefficient of each word in the text is obtained according to the semantic similarity between any word in the text and each word in the cluster with high similarity, the frequency of each word in the cluster with high similarity, the number of all words and the importance of each word in the text, including the specific formula as follows: Where W v Represents the frequency weight coefficient of the vth word in the text; N v,i represents the frequency of occurrence of the i-th word in the cluster with high similarity to the v-th word in the text; a represents the initial weight coefficient of the frequency of the v-th word in the text; n v Represents the number of all words in the cluster with high similarity to the vth word in the text; C v,i Indicates the semantic similarity between the vth word in the text and the i-th word in the cluster with the highest similarity; softmax() is the normalization function; H v Indicates the importance of the vth word in the text; the initial weight coefficient of the frequency of the vth word in the text is a preset first threshold.

5. A method for quickly compressing local archive-related data according to claim 1, characterized in that: The specific steps of obtaining the frequency of co-occurrence of any two words in the text are as follows: In the text, take one word as one unit, set a linear sliding window with N units along the writing direction of the text, and slide with one unit as the step until the whole text is traversed. The number of linear sliding windows that contain both the vth word and the sth word is recorded as the frequency of simultaneous occurrence of the vth word and the sth word, where N is the preset number of units.

6. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The probability value of any two words appearing at the same time in the text is obtained according to the frequency of any two words appearing at the same time in the text and the number of all words, including the specific formula as follows: Where P v,s represents the probability value of the vth word and the sth word appearing at the same time in the text; M v,s Indicates the frequency of the vth word and the sth word appearing together in the text; N is the preset number of units; N all Represents the number of all words in the text.

7. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The frequency weight index of each word in the text is obtained according to the frequency, probability value and frequency weight coefficient of any two words appearing at the same time in the text, and the specific formula included is as follows: Where Y v represents the frequency weight index of the vth word in the text; z represents the initial frequency weight index of the vth word in the text; M v,s represents the frequency of the vth word and the sth word appearing together in the text; m represents the number of all words in the text except the vth word; P v,s N represents the probability value of the vth word and the sth word appearing at the same time in the text; v W represents the frequency of the vth word in the text; v represents the frequency weight coefficient of the vth word in the text; softmax() is the normalization function; N all Represents the number of all words in the text; the initial weight index of the vth word frequency in the text is the preset second threshold.

8. A method for rapidly compressing local archive-related data according to claim 1, characterized in that: The specific steps of adjusting the frequency of each word in the text according to the frequency weight index of each word and constructing a Huffman tree for data compression are as follows: Multiply each word frequency weight index by the frequency of occurrence of each word to obtain the weighted frequency of occurrence of each word; based on the weighted frequency of occurrence of each word, use Huffman coding to construct a Huffman tree for data compression to obtain the compressed code of each word.

Images