A text secondary compression method, device and equipment and computer readable storage medium

Abstract

The application provides a text secondary compression method and device, equipment and computer readable storage medium, a text file is expressed into a corresponding code number sequence through a word dictionary; the code number sequence is compressed into a first digital sequence representing the semantics of the text file through a pre-training model; the occurrence probability of a target prediction number in the first digital sequence is predicted based on the similarity of the first digital sequence; and the first digital sequence is secondarily compressed through an arithmetic coding algorithm for the probability distribution of the occurrence probability. Through the implementation of the application scheme, after the text file is preprocessed, the text file is compressed into a digital sequence representing the semantics through a pre-training model, the simplicity of the pre-training model representing the semantics realizes the first compression, and the second compression is realized through a digital prediction algorithm. The combination of the compact form of semantic representation and the two compression methods of digital prediction can obtain a higher compression ratio.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a method, apparatus, device and computer-readable storage medium for secondary text compression. Background Technology

[0002] Traditional data compression methods utilize the uneven distribution of character occurrence probabilities, encoding high-probability characters with longer codes and low-probability characters with shorter codes to achieve overall file compression. However, the order in which characters appear in a file generally carries deeper meaning, and character-level frequency statistical compression significantly limits the compression ratio. Summary of the Invention

[0003] This application provides a text secondary compression method, apparatus, device, and computer-readable storage medium, which can at least solve the compression ratio limitation of character statistical frequency compression in related technologies.

[0004] The first aspect of this application provides a text secondary compression method, including:

[0005] The text file is represented as a sequence of corresponding encoded numbers using a word segmentation dictionary;

[0006] The pre-trained model compresses the encoded number sequence into a first number sequence representing the semantics of the text file;

[0007] Based on the similarity of the first number sequence, predict the probability of the occurrence of the target predicted number in the first number sequence;

[0008] Based on the probability distribution of the occurrence probability, the first number sequence is compressed twice using an arithmetic coding algorithm.

[0009] A second aspect of this application provides a text secondary compression apparatus, comprising:

[0010] The encoding module is used to represent text files into corresponding encoded number sequences using a word segmentation dictionary;

[0011] The first compression module is used to input the encoded number sequence into the pre-trained model and compress it into a first number sequence representing the semantics of the text file;

[0012] The prediction module is used to predict the probability of the occurrence of a target predicted number in the first number sequence based on the similarity of the first number sequence.

[0013] The second compression module is used to perform secondary compression on the first number sequence using an arithmetic coding algorithm based on the probability distribution of the occurrence probability.

[0014] A third aspect of this application provides an electronic device, characterized in that it includes a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and the processor executes the computer program in accordance with the steps of the text secondary compression method provided in the first aspect of this application.

[0015] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the text secondary compression method provided in the first aspect of this application.

[0016] As can be seen from the above, according to the method, apparatus, device, and computer-readable storage medium provided in this application, a text file is represented as a corresponding encoded number sequence through a word segmentation dictionary; the encoded number sequence is compressed into a first numerical sequence representing the semantics of the text file through the pre-trained model; based on the similarity of the first numerical sequence, the probability of occurrence of the target predicted number in the first numerical sequence is predicted; and the first numerical sequence is further compressed using an arithmetic coding algorithm based on the probability distribution of the occurrence probability. Through the implementation of this application, after preprocessing the text file, the text file is compressed into a numerical sequence representing semantics through a pre-trained model. The simplicity of the semantic representation represented by the pre-trained model achieves the first compression, and the second compression is performed through a numerical prediction algorithm. The combination of the compact semantic representation and numerical prediction compression achieves a higher compression ratio. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the basic process of the text secondary compression method provided in the first embodiment of this application;

[0018] Figure 2 A schematic diagram illustrating the semantic representation of the BERT model provided in the first embodiment of this application;

[0019] Figure 3 A flowchart illustrating the arithmetic decoding process provided in the first embodiment of this application.

[0020] Figure 4 A schematic diagram illustrating the semantic decoding process of the BERT model provided in the first embodiment of this application;

[0021] Figure 5 A detailed flowchart illustrating the text secondary compression method applied to a text secondary compression system provided in the second embodiment of this application;

[0022] Figure 6 A schematic diagram of the program modules of the text secondary compression device provided in the third embodiment of this application;

[0023] Figure 7This is a schematic diagram of the structure of an electronic device provided in the fourth embodiment of this application. Detailed Implementation

[0024] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] To address the issue that character-level compression in related technologies severely limits the compression ratio, the first embodiment of this application provides a secondary text compression method, such as... Figure 1 This is a basic flowchart of the text secondary compression method provided in this embodiment. The text secondary compression method includes the following steps:

[0026] Step 101: Represent the text file as a sequence of corresponding encoded numbers using a word segmentation dictionary.

[0027] Specifically, existing text compression technologies typically compress text characters directly, leveraging the uneven probability of character occurrence. Characters with high probability are encoded with longer codes, while those with low probability are encoded with shorter codes, thus compressing the entire file. In this embodiment, before compressing the text file, it is preprocessed. This preprocessing primarily involves encoding the text into a numerical form that a pre-trained model can handle. This includes Chinese dictionary construction, text segmentation, and text encoding. After preprocessing, the text is represented as a numerical sequence, which can be used as input to the pre-trained model. The pre-trained model then represents the text as a semantic vector.

[0028] In one optional implementation of this embodiment, the step of representing a text file as a corresponding sequence of encoded words using a word segmentation dictionary includes: encoding the words that may appear in the text file into a dictionary in sequence; segmenting the text file according to the dictionary; and obtaining the sequence of encoded words of the text file by looking up the number of each word in the dictionary after segmentation.

[0029] Specifically, in this embodiment, for the preprocessing of text files, first, the words that may appear in the Chinese text are encoded into a dictionary and encoded in order. The total length of the dictionary is N. For example, {1, appear; 2, contain; 3, of; 4, in; 5, has; 6, more; 7, profound; 8, a; 9, character; 10, meaning; 11, order; 12, file; 13, compress;...}; Secondly, according to the constructed dictionary, the text to be compressed is segmented. An open-source segmenter can be used, such as Hanlp, jieba, etc. For example, if the text to be compressed is 'The order of character appearance in a file contains a more profound meaning', after segmentation, it becomes 'a|file|in|character|appear|of|order|contain|has|more|profound|meaning'; Finally, the segmented sentence is encoded into a numerical vector, and each segment in the sentence looks up the corresponding number in the dictionary. For example, the segmentation 'a|file|in|character|appear|of|order|contain|has|more|profound|meaning' is encoded to [8, 12, 4, 9, 1, 3, 11, 2, 5, 6, 7, 10]. For the Chinese text file to be compressed, the entire text is segmented, and the numbers corresponding to each word in the dictionary are found to obtain the encoded number sequence value vector of the entire text, with a length of H. The encoded number sequence of the text to be compressed is S=(S1, S2,...., S H ).

[0030] Step 102: Compress the encoded number sequence into a first digital sequence representing the semantics of the text file through a pre-trained model.

[0031] Specifically, in this embodiment, the pre-trained model can use the BERT model, or the ERNIE model and the larger T5 model. The pre-trained model BERT (Bidirectional Encoder Representation from Transformers, bidirectional encoding identification of transformers) is trained through a large amount of text, and the common sense of human language is included in the trained model. In practical applications, one can train the pre-trained model using a large amount of corpus by oneself, or download the already trained pre-trained model from an open-source source. For texts in a specific field, the general pre-trained model can also be fine-tuned using the texts in that field so that the model can better capture the knowledge in the specific field. When the text file needs to be compressed, the encoded number sequence encoded for the text file is input into the BERT model, and the first compression is performed through the BERT model to obtain the digital sequence representing the semantics of the text file.

[0032] In one optional implementation of this embodiment, the step of compressing the encoded number sequence into a first numerical sequence representing the semantics of a text file using a pre-trained model includes: after obtaining the encoded number sequence, controlling the pre-trained model to add placeholders to the encoded number sequence; converting the corresponding numbers in the encoded number sequence into vectors of a preset numerical length; wherein the vector corresponding to the placeholder is a representation vector of the number; and obtaining the first numerical sequence representing the semantics of the text file by concatenating the representation vectors.

[0033] Specifically, in this embodiment, such as Figure 2 This is a flowchart illustrating the semantic representation process of the BERT model provided in this embodiment. A sentence is input into the pre-trained BERT model, and a placeholder is added to the beginning of the sentence using word embeddings. The placeholder is represented by cls. The representation vector obtained after the placeholder is processed by the pre-trained model is the semantic representation of the entire sentence. The pre-trained model converts each word in the sentence into a fixed-length semantic vector, representing the semantic meaning of that word in the sentence. The vector corresponding to the placeholder serves as the semantic representation of the entire sentence. The output vector corresponding to the placeholder is used as the representation vector C = (c1, c2, ..., c...). d For example, C = [0.23, -1.23, 3.23, ..., 2.123] represents the components c of vector C. i ∈C is a b-digit number, meaning the vector C consists of d*b numerical values. For the entire text file, after word segmentation, the length is H. A BERT semantic representation embedding operation is performed on L words, meaning each L word is represented by a semantic vector, and the length of each L-word semantic vector is d*b. The entire text can be divided into H / L segments. For each segment, this network is used to obtain a semantic representation. Therefore, the entire text can be represented as M = d*b*H / L numbers, or the semantic representation of the entire text is S = S1S2...S... i ...S M =0193....783673..1534. The semantic representation of the text is obtained through the BERT model. The semantic representation captures the basic semantic information and the resulting representation is shorter. It is equivalent to compressing the original text file, transforming the original text file into a compact number sequence through semantic representation. There is a high degree of similarity and correlation within the number sequence.

[0034] Step 103: Based on the similarity of the first number sequence, predict the probability of the occurrence of the target predicted number in the first number sequence.

[0035] In this embodiment, the high similarity and correlation within a numerical sequence are utilized to predict the probability of the next number using a prediction probability model. This prediction probability model includes, but is not limited to, Markov models, which predict the probability of the next number based on the previous few numbers. An l-order probability model is employed, meaning the probability P(x) of the (l+1)th number is predicted using the preceding l numbers. i |x i-1...i-l Based on the similarity of the first number sequence, predicting the probability of the target predicted number appearing in the first number sequence can make the prediction probability more accurate.

[0036] In one optional implementation of this embodiment, after the step of predicting the probability of occurrence of the target predicted number in the first number sequence based on the similarity of the first number sequence, the method further includes: creating a frequency table of the occurrence of numbers in the second number sequence based on the second number sequence that has been predicted; and updating the occurrence probability through the frequency table.

[0037] Specifically, in this embodiment, prediction and compression are a reciprocal process. That is, the (l+1)th number is predicted using the first l numbers, and the (l+1)th number is compressed. This process is repeated sequentially, dynamically using the already compressed sequence to update the predicted frequency of the next number. Table 1 shows the frequency table of predicted numbers in the number sequence provided in this embodiment. For example, if l = 8, then as shown in Table 1, the predicted probability is updated using the frequency table.

[0038]

[0039] Where, n s The frequency of occurrences is given by the number of occurrences already shown. δ is a smoothing parameter, typically set to a very small value, such as 0.01.

[0040] Table 1

[0041]

[0042] By recursively utilizing the prediction concept model, for Ml numbers (the first l numbers are used as seed values, which do not need to be predicted and are directly stored), the predicted probability values ​​for each corresponding number (0 to 9, a total of 10 numbers) can be given, expressed as:

[0043]

[0044] The sum of the probabilities of the Ml numbers is expressed as:

[0045]

[0046] Step 104: Based on the probability distribution of occurrence probability, the first number sequence is compressed twice using an arithmetic coding algorithm.

[0047] In one optional implementation of this embodiment, the step of performing secondary compression on the first digital sequence using an arithmetic coding algorithm includes: determining the sequence value to be encoded and the corresponding probability interval in the first digital sequence according to the probability distribution; selecting the smallest integer in the probability interval as the arithmetic coding value of the sequence value to be encoded based on binary conversion; and completing the secondary compression of the first digital sequence by concatenating all arithmetic coding values.

[0048] Specifically, in this embodiment, for relatively large files, encoding the entire Ml words into a single arithmetic code would be too cumbersome. Therefore, it is necessary to divide the file into segments for encoding. For the Ml words to be encoded, the file is divided into segments of X words each, i.e., D = (Ml) / X segments. Arithmetic encoding is then performed on each segment.

[0049] K1=(S l+1 ,S l+2 ,...,S l+X ),K2=(S l+X+1 ,S l+X+2 ,S l+2X ),....,K D =(S l+(D-1)X+1 ,S l+(D-1)X+2 ,S l+DX For the i-th segment, the number to be encoded is: The corresponding cumulative probability is: C i =(C l+(i - 1)X+1 C l+(i - 1)X+2 ,...,C l+iX ), that is, for K i The j-th number You can find its corresponding probability interval, for example The corresponding probability interval is

[0050] For each number, the encoding interval is divided into 0 to 9 according to the size of the probability interval. For K i For each number in the array, find the corresponding probability interval. Choose a large integer B (for example, B = 2). 64 ).

[0051] The encoding range for the first number is:

[0052] The encoding range for the second number is:

[0053]

[0054] And so on, the code for the Xth number is:

[0055] Based on the memory resource consumption after binary conversion, we select a relatively short integer from this range, which is K. i Arithmetic coding. For D segments, apply arithmetic coding sequentially, and then combine the K values ​​of all segments. i By concatenating the two parts, the entire text is encoded. The arithmetic encoding process described above can be understood as a secondary compression of the text file. By designing two compression processes, the semantics and numerical arrangement patterns of the text can be captured respectively, thus achieving a relatively high compression ratio.

[0056] It should be noted that since the text file is compressed twice, decoding the compressed file requires two corresponding decoding steps. The entire decoding process consists of two steps: arithmetic decoding and BERT semantic decoding. Arithmetic decoding uses the compressed integers to decode the semantic vector of the entire text. BERT semantic decoding uses the BERT model to decode the corresponding original text using the semantic vector.

[0057] In one optional implementation of this embodiment, after the step of performing secondary compression on the first digit sequence using an arithmetic coding algorithm based on the probability distribution of occurrence probability, the method further includes: updating the prediction probability formula by updating the frequency table; predicting the digit distribution probability of the first digit sequence according to the prediction probability formula; solving for the corresponding digits based on the digit distribution probability using an arithmetic decoding algorithm; and dividing all digits into different semantic representation vectors according to the semantic representation of the first digit sequence.

[0058] Specifically, in this embodiment, such as Figure 3 This is a flowchart illustrating the arithmetic decoding process provided in this embodiment. The arithmetic decoding process is similar to arithmetic encoding. First, using the directly stored l numbers as seeds, starting from the (l+1)th word, the frequency table is dynamically updated, the prediction probability formula is updated, and the probability formula is used step by step to predict the distribution probability of the numbers. The corresponding numbers are then solved step by step using arithmetic decoding.

[0059] For the first number to be decoded, the prediction range for the 10 numbers from 0 to 9 is... if The first number is found to be k, and the corresponding upper and lower limits become... The prediction range for the second number is:

[0060]

[0061] if The second number is thus found to be v, and the corresponding upper and lower limits become... By analogy, these Ml consecutive numbers can be solved. Adding the l numbers directly stored, we obtain M numbers representing semantics. These M numbers are divided into semantic vectors of length d*b, resulting in a total of M / (d*b) semantic representation vectors. Each semantic vector is a vector C = (c1, c2, ..., c...) of length d. d Next, BERT decoding is performed on each semantic vector to obtain the corresponding original text.

[0062] Furthermore, in an optional embodiment of this example, after the step of dividing all numbers into different semantic representation vectors according to the semantic representation of the first number sequence, the method further includes: during the decoding of different semantic representation vectors based on the recurrent neural network structure, for the first loop step state in the recurrent neural network structure, updating the first loop step state by inputting the semantic representation vector and the second loop step state, and generating a corresponding encoded number sequence; searching for the corresponding dictionary according to the encoded number sequence and obtaining the text file through decoding.

[0063] Specifically, in this embodiment, BERT semantic decoding adopts a recurrent structure. The semantic vector C obtained from the preceding algorithm is then decoded using a recurrent network structure. For example... Figure 4 This is a flowchart illustrating the semantic decoding process of the BERT model provided in this embodiment. For each loop step, the semantic vector and the state h of the previous loop step are used as input to update the state h of this loop step. The corresponding word segmentation code is generated using the state h of this loop step. The original text can be decoded by looking up the corresponding dictionary using the word segmentation code.

[0064] Based on the embodiments of the above application, a text file is represented as a corresponding sequence of encoded numbers using a word segmentation dictionary; the sequence of encoded numbers is compressed into a first sequence of numbers representing the semantics of the text file using a pre-trained model; based on the similarity of the first sequence of numbers, the probability of the occurrence of the target predicted number in the first sequence of numbers is predicted; and the first sequence of numbers is further compressed using an arithmetic coding algorithm based on the probability distribution of the occurrence probability. Through the implementation of this application's solution, after preprocessing the text file, the text file is compressed into a sequence of numbers representing semantics using a pre-trained model. The simplicity of the semantic representation represented by the pre-trained model achieves the first compression, and the second compression is performed using a number prediction algorithm. The combination of the compact form of semantic representation and the two compression methods of number prediction achieves a higher compression ratio.

[0065] Figure 5 The method described in the second embodiment of this application is a refined text secondary compression method, which includes:

[0066] Step 501: Represent the text file as a sequence of corresponding encoded numbers using a word segmentation dictionary.

[0067] Step 502: After the pre-trained model receives the encoded number sequence, control the pre-trained model to add placeholders to the encoded number sequence.

[0068] Step 503: Convert the corresponding word segmentation numbers in the encoded number sequence based on the preset dictionary into vectors of preset number length.

[0069] Specifically, in this embodiment, the corresponding word segmentation number based on the preset dictionary is the corresponding word segmentation number of the dictionary constructed by preprocessing the text file. In the converted vector, the vector corresponding to the placeholder is the representation vector of the corresponding word segmentation number.

[0070] Step 504: Obtain the numerical sequence representing the semantics of the text file by concatenating the various representation vectors.

[0071] Step 505: Based on the similarity of the number sequences, predict the probability of the target predicted number appearing in the number sequence.

[0072] Step 506: Determine the sequence value of the number to be encoded and the corresponding probability interval in the number sequence based on the probability distribution of the occurrence probability.

[0073] Step 507: Based on binary conversion, select the smallest integer in the probability interval as the arithmetic code value of the sequence value to be encoded.

[0074] Step 508: Secondary compression of the digital sequence is completed by concatenating all arithmetic code values.

[0075] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.

[0076] According to the text secondary compression method provided in this application, the text file is represented as a corresponding sequence of encoded numbers using a word segmentation dictionary. After the pre-trained model obtains the sequence of encoded numbers, placeholders are added to the sequence. The corresponding word segmentation numbers in the sequence based on the preset dictionary are converted into vectors of a preset length. A sequence of numbers representing the semantics of the text file is obtained by concatenating the representation vectors. Based on the similarity of the number sequences, the probability of the target predicted number in the number sequence is predicted. The value of the number sequence to be encoded and its corresponding probability interval are determined according to the probability distribution of the occurrence probability. Based on binary conversion, the smallest integer in the probability interval is selected as the arithmetic code value of the number sequence value to be encoded. Secondary compression of the number sequence is completed by concatenating all arithmetic code values. Through the implementation of this application, after the text file is preprocessed, it is compressed into a sequence of numbers representing the semantics using a pre-trained model. The simplicity of the semantic representation by the pre-trained model achieves the first compression, and secondary compression is performed by the number prediction algorithm. The combination of the compact form of semantic representation and the two compression methods of number prediction achieves a higher compression ratio.

[0077] Figure 6 A text secondary compression apparatus is provided in the third embodiment of this application. This text secondary compression apparatus can be used to implement the text secondary compression method in the foregoing embodiments. For example... Figure 6 As shown, the text secondary compression device mainly includes:

[0078] Encoding module 601 is used to represent a text file into a corresponding sequence of encoded numbers using a word segmentation dictionary;

[0079] The first compression module 602 is used to input the encoded number sequence into the pre-trained model and compress it into a first number sequence representing the semantics of the text file;

[0080] The prediction module 603 is used to predict the probability of the occurrence of a target predicted number in the first number sequence based on the similarity of the first number sequence.

[0081] The second compression module 604 is used to perform secondary compression on the first number sequence using an arithmetic coding algorithm based on the probability distribution of the occurrence probability.

[0082] In one optional implementation of this embodiment, the encoding module is specifically used to: encode the words that may appear in the text file into a dictionary in sequence; segment the text file according to the dictionary; and obtain the encoding sequence of the text file by looking up the number of each word in the dictionary after segmentation.

[0083] Furthermore, in an optional implementation of this embodiment, the first compression module is specifically used to: after obtaining the encoded number sequence, control the pre-trained model to add placeholders to the encoded number sequence; convert the corresponding number in the encoded number sequence into a vector of a preset number length; wherein, the vector corresponding to the placeholder is a representation vector of the number; and obtain a first number sequence representing the semantics of the text file by concatenating the representation vectors.

[0084] In an optional embodiment of this invention, the text secondary compression device further includes a creation module and an update module. The creation module is used to create a frequency table of the occurrences of numbers in the second number sequence based on the second number sequence after prediction. The update module is used to update the occurrence probabilities using the frequency table.

[0085] In one optional implementation of this embodiment, the second compression module is specifically used to: determine the sequence value of the number to be encoded and the corresponding probability interval in the first digital sequence according to the probability distribution; select the smallest integer in the probability interval as the arithmetic code value of the sequence value to be encoded based on binary conversion; and complete the secondary compression of the first digital sequence by concatenating all arithmetic code values.

[0086] In an optional embodiment of this invention, the text secondary compression device further includes a decoding module and a partitioning module. The updating module is further configured to update the prediction probability formula by updating the frequency table. The prediction module is further specifically configured to predict the digit distribution probability of the first digit sequence according to the prediction probability formula. The decoding module is configured to solve for the corresponding digits based on the digit distribution probability using an arithmetic decoding algorithm. The partitioning module is configured to partition all digits into different semantic representation vectors according to the semantic representation of the first digit sequence.

[0087] Furthermore, in an optional embodiment of this example, the decoding module is also used to: during the decoding of different semantic representation vectors based on the recurrent neural network structure, update the first loop step state by inputting the semantic representation vector and the second loop step state for the first loop step state in the recurrent neural network structure, and generate a corresponding encoding number sequence.

[0088] It should be noted that the text secondary compression methods in the first and second embodiments can be implemented based on the text secondary compression device provided in this embodiment. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the security function identification device described in this embodiment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0089] According to the text secondary compression device provided in this application, a text file is represented as a corresponding encoded number sequence using a word segmentation dictionary; the encoded number sequence is compressed into a first numerical sequence representing the semantics of the text file using a pre-trained model; based on the similarity of the first numerical sequence, the probability of the occurrence of the target predicted number in the first numerical sequence is predicted; and based on the probability distribution of the occurrence probability, the first numerical sequence is compressed a second time using an arithmetic coding algorithm. Through the implementation of this application, after preprocessing the text file, the pre-trained model compresses the text file into a numerical sequence representing semantics. The simplicity of the semantic representation represented by the pre-trained model achieves the first compression, and the numerical prediction algorithm performs the second compression. The combination of the compact semantic representation and the numerical prediction compression achieves a higher compression ratio.

[0090] Figure 7 An electronic device is provided in the fourth embodiment of this application. This electronic device can be used to implement the text secondary compression method in the foregoing embodiments, and mainly includes:

[0091] The system includes a memory 701, a processor 702, and a computer program 703 stored on the memory 701 and executable on the processor 702. The memory 701 and the processor 702 are connected via communication. When the processor 702 executes the computer program 703, it implements the text secondary compression method described in the foregoing embodiments. The number of processors can be one or more.

[0092] The memory 701 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 701 is used to store executable program code, and the processor 702 is coupled to the memory 701.

[0093] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 7 The memory in the illustrated embodiment.

[0094] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the text secondary compression method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0096] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0097] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0098] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0099] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0100] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0101] The above is a description of the text secondary compression method, apparatus, device, and computer-readable storage medium provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for secondary text compression, characterized in that, include: Encode the words that may appear in the text file into a dictionary in order; The text file is segmented according to the dictionary; The encoding sequence of the text file is obtained by finding the number of each word in the dictionary after word segmentation; After obtaining the encoded number sequence, the pre-trained model is controlled to add placeholders to the encoded number sequence; The corresponding number in the encoded number sequence is converted into a vector of a preset number length; wherein the vector corresponding to the placeholder is the representation vector of the number; A first numerical sequence representing the semantics of the text file is obtained by concatenating the representation vectors. Based on the similarity of the first number sequence, predict the probability of the occurrence of the target predicted number in the first number sequence; Based on the probability distribution of the occurrence probability, the first number sequence is compressed twice using an arithmetic coding algorithm.

2. The text secondary compression method according to claim 1, characterized in that, After the step of predicting the probability of the occurrence of the target predicted number in the first number sequence based on the similarity of the first number sequence, the method further includes: Create a frequency table of the numbers appearing in the second number sequence based on the completed prediction; The occurrence probability is updated using the frequency table.

3. The text secondary compression method according to claim 1, characterized in that, The step of performing secondary compression on the first digital sequence using an arithmetic coding algorithm includes: The sequence values ​​of the numbers to be encoded in the first number sequence and the corresponding probability intervals are determined based on the probability distribution. Based on binary conversion, the smallest integer in the probability interval is selected as the arithmetic code value of the sequence value to be encoded; The second compression of the first number sequence is completed by concatenating all the arithmetic code values.

4. The text secondary compression method according to claim 2, characterized in that, After the step of performing secondary compression on the first digit sequence using an arithmetic coding algorithm based on the probability distribution of the occurrence probability, the method further includes: The prediction probability formula is updated by updating the frequency table accordingly. Predict the digit distribution probability of the first digit sequence according to the prediction probability formula; The corresponding number based on the probability distribution of the given number is obtained by using an arithmetic decoding algorithm; Based on the semantic representation of the first number sequence, all numbers are divided into different semantic representation vectors.

5. The text secondary compression method according to claim 4, characterized in that, After the step of dividing all numbers into different semantic representation vectors based on the semantic representation of the first number sequence, the method further includes: In the process of decoding different semantic representation vectors based on a recurrent neural network structure, for the first loop step state in the recurrent neural network structure, the first loop step state is updated by inputting the semantic representation vector and the second loop step state, and the corresponding encoding number sequence is generated. The corresponding dictionary is found based on the encoded number sequence, and the text file is obtained by decoding.

6. A text secondary compression device, characterized in that, include: An encoding module is used to encode the words that may appear in a text file into a dictionary in sequence; and to segment the text file according to the dictionary. The encoding sequence of the text file is obtained by finding the number of each word in the dictionary after word segmentation; The first compression module is used to, after obtaining the encoded number sequence, control the pre-trained model to add placeholders to the encoded number sequence; convert the corresponding number in the encoded number sequence into a vector of a preset number length; wherein the vector corresponding to the placeholder is the representation vector of the number; and obtain a first number sequence representing the semantics of the text file by concatenating the representation vectors. The prediction module is used to predict the probability of the occurrence of a target predicted number in the first number sequence based on the similarity of the first number sequence. The second compression module is used to perform secondary compression on the first number sequence using an arithmetic coding algorithm based on the probability distribution of the occurrence probability.

7. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images