High-efficiency lossless text compression method based on large language model

Abstract

The invention discloses a method for performing efficient lossless text compression based on a large language model, which relates to the technical field of text compression and comprises the following steps of: preprocessing to-be-compressed text data to obtain an initial token sequence; selecting and loading a pre-trained big language model, analyzing the current context information by using the big language model, obtaining a probability value of each token, forming a probability distribution list, and recording a ranking sequence number of the current token; the context window is moved backwards by one token, and context information is updated until all the tokens are processed; and recording the ranking sequence numbers of all the tokens in sequence to form a sequence number list so as to obtain a final text compression result. According to the method, the compression rate is remarkably increased, the requirements for storage space and transmission bandwidth are reduced, the high processing speed is kept, and the method is suitable for real-time compression and decompression of large-scale text data.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and more specifically, to a method for performing efficient lossless text compression based on a large language model. Background Art

[0002] With the rapid development of the Internet industry, the amount of information in the network is growing exponentially. As one of the most important forms of information dissemination on the Internet, text information occupies an important position in data exchange and storage. However, the huge amount of text data not only consumes a lot of storage resources, but also causes problems of inefficiency and increased delay in the data transmission process. Although traditional text compression methods, such as Huffman coding and arithmetic coding, have improved the compression ratio to a certain extent, their compression efficiency and speed are still limited when facing the growing amount of data. In recent years, the application of large language models in the field of natural language processing has made significant progress, providing new ideas for text compression.

[0003] The above disclosed technical solutions have at least the following technical problems: the compression efficiency of the traditional prediction method based on statistical models is not high enough, and the prediction results are not accurate enough. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a method for efficient lossless text compression based on a large language model, which performs text prediction through a large language model to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for efficient lossless text compression based on a large language model comprises the following steps: preprocessing text data to be compressed to obtain an initial token sequence; selecting and loading a pre-trained large language model, initializing model parameters and extracting current context information according to the size of a context window; analyzing the current context information using the large language model, obtaining a probability value of each token, forming a probability distribution list, and recording the ranking sequence number of the current token; moving the context window backward by one token, updating the context information, until all tokens are processed; recording the ranking sequence numbers of all tokens in order to form a sequence number list; selecting any lossless compression algorithm to compress the ranking sequence, compressing it and outputting it to obtain a final text compression result.

[0006] In a preferred embodiment, the text data to be compressed is preprocessed to obtain an initial token sequence, specifically: the text data to be compressed is obtained, and the text data is denoised and standardized; the denoised and standardized text data is dynamically divided into a number of tokens based on an adaptive word segmentation and tokenization strategy; a unique identifier is assigned to each token code to obtain an initial token sequence.

[0007] In a preferred embodiment, the selecting and loading of a pre-trained large language model, initializing model parameters and extracting current context information according to the context window size are specifically as follows: selecting a pre-trained large language model according to a specific application scenario; loading a pre-trained large language model based on a deep learning framework and a corresponding library; setting model parameters of the pre-trained large language model, wherein the model parameters include but are not limited to a learning rate, a batch size, and a context window size; parsing the input text data into tokens using a word segmenter; extracting the text before the current token from the input text as context according to the context window size; if the context window size exceeds the length of the input text, gradually processing the text by using a sliding window technique, combining the content in the current window with the content in the previous window; and combining the current token and the context information into a matrix form as an input sequence.

[0008] In a preferred embodiment, the probability value of each token is obtained, a probability distribution list is formed, and the ranking number of the current token is recorded, specifically: the integer encoding of the tokens is converted into a high-dimensional vector and input into a large language model; the high-dimensional vector is processed by a multi-layer neural network to capture the dependency relationship and context information between words; in each layer, a self-attention mechanism is used to calculate the relationship between each token and other tokens, and the context of the entire input sequence is analyzed; the probability distribution of each token is calculated by a probability mass function, and all tokens are sorted according to the calculated probability distribution; the position of the current token in the sorted list is obtained, and its ranking number is recorded; the sorted tokens and their corresponding probability values ​​are formed into a probability distribution list and the probability distribution list and the ranking number of the current token are output.

[0009] In a preferred embodiment, the context window is moved backward by one token and the context information is updated until all tokens are processed, specifically: a pointer is initialized by tracking the position of the token currently being processed; an iterator is used to traverse from the start position to the end position of the sequence; in each iteration, the current token is extracted from the input sequence according to the position of the current pointer, and the current token is added to the context window; the required tasks are performed using the information of the current context window, and the pointer is moved backward by one position to point to the next token; and the loop is stopped when the pointer reaches the end of the sequence.

[0010] In a preferred embodiment, the ranking serial numbers of all tokens are recorded in order to form a serial number list. Specifically, in each iteration, the serial number of the current token is added to the serial number list. The serial number can be a continuous integer or a number generated according to a specific rule.

[0011] In a preferred embodiment, the method of selecting any lossless compression algorithm to compress the ranking sequence and outputting the final text compression result is specifically as follows: establishing an input and output stream with the ranking sequence; constructing a hash function to count all character codes represented by two bytes, and recording the number of occurrences of each character; constructing a minimum heap based on the hash function and sorting the character statistics to obtain a Huffman tree; encoding the characters of the child nodes of the Huffman tree according to the principle of left 0 and right 1, and writing the encoding results into an array corresponding to the array used for counting characters; performing pre-order traversal and post-order traversal on the Huffman tree to obtain a pre-order sequence and a post-order sequence; writing the pre-order sequence and the post-order sequence into the first position of the compressed file in the form of an object; re-reading the text to be compressed, obtaining each character and calling the hash function to obtain a new code for the character, converting the code into a number and writing it into a byte, and writing the byte into the compressed file; repeating the above-mentioned process, when the text to be compressed is read in and all characters are processed, a compressed file is obtained.

[0012] The technical effects and advantages of the method for efficient lossless text compression based on a large language model of the present invention are as follows: 1. The present invention uses a large language model to predict text, which is an important innovation in traditional text compression technology. This prediction method based on deep learning is more efficient and accurate than the traditional prediction method based on statistical models. The present invention not only significantly improves the compression rate, reduces the demand for storage space and transmission bandwidth, but also maintains a high processing speed, and is suitable for real-time compression and decompression of large-scale text data.

[0013] 2. The present invention achieves lossless compression by combining the predictive power of a large language model. Lossless compression means that the original text information is not lost during the compression and decompression process, which is crucial for application scenarios that require data integrity. At the same time, the present invention can further improve the speed of compression and decompression by optimizing the algorithm and model parameters to achieve efficient performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The present invention is a schematic diagram of the structure of a method for efficient lossless text compression based on a large language model. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1, Figure 1 The present invention provides a method for efficient lossless text compression based on a large language model, comprising the following steps: S1, preprocess the compressed text data to obtain the initial token sequence; S2,,selects and loads a pre-trained large language model, initializes the model parameters and extracts the current context information according to the context window size; S3, use the large language model to analyze the current context information, obtain the probability value of each token, form a probability distribution list, and record the ranking number of the current token; S4, move the context window back one token and update the context information until all tokens are processed; S5, record the ranking numbers of all tokens in order to form a number list; S6, select any lossless compression algorithm to compress the ranking sequence and output the final text compression result.

[0017] S1, preprocess the compressed text data to obtain the initial token sequence; In natural language processing (NLP), "token" usually refers to the smallest unit or element into which text data is segmented. These tokens can be words, phrases, punctuation marks, numbers, or any other text fragments with specific meanings. Specifically, tokens are the basic units obtained after preprocessing of text. They represent the various components of the text and are used for subsequent tasks such as text analysis, processing, or compression. After the text data to be compressed is segmented into several tokens, each token will be processed separately, such as encoding to generate the initial token sequence.

[0018] The text data to be compressed is preprocessed to obtain an initial token sequence, specifically: Obtaining text data to be compressed, and performing denoising and standardization on the text data; Dynamically split the denoised and standardized text data into several tokens based on the adaptive word segmentation and tokenization strategy; Assign a unique identifier to each token encoding to obtain the initial token sequence.

[0019] S2, select and load the pre-trained large language model, initialize the model parameters and extract the current context information according to the context window size; The big language model is based on deep learning technology, especially the Transformer architecture. It learns language patterns, grammar, and semantics by processing massive amounts of text data, thereby being able to understand and generate models of human language.

[0020] Large Parameters: Large language models often contain billions or more parameters, which enables them to learn rich language features and patterns.

[0021] Massive data training: These models are pre-trained on large amounts of text data to capture the complexity and diversity of language.

[0022] Generality: Large language models can provide general language representation for various natural language processing tasks, such as text classification, sentiment analysis, machine translation, etc.

[0023] The pre-trained large language models can be Tongyi Qianwen, Doubao large model, etc.

[0024] Let s represent a characters. Suppose there is a dictionary containing D entries , we can convert s into a series of tokens through the parsing process ,in There is a one-to-one correspondence between s and x, so compressing s is equivalent to compressing x.

[0025] Each Treated as a random variable The result of an instantiation of is an element extracted from the domain X. This process can be regarded as an encoding mechanism, through which the original text s is effectively converted into a sequence of tokens.

[0026] The method selects and loads a pre-trained large language model, initializes model parameters, and extracts current context information according to the context window size, specifically: Select a pre-trained large language model based on the specific application scenario; Load pre-trained large language models based on deep learning frameworks and corresponding libraries; Setting model parameters of a pre-trained large language model, the model parameters including but not limited to a learning rate, a batch size, and a context window size; Use a tokenizer to parse the input text data into tokens; According to the context window size, extract the text before the current token from the input text as context; If the context window size exceeds the length of the input text, the text is processed step by step using a sliding window technique, combining the content of the current window with the content of the previous window; Combine the current token and context information into a matrix form as the input sequence.

[0027] S3, use the large language model to analyze the current context information, obtain the probability value of each token, form a probability distribution list, and record the ranking number of the current token; The large language model can be viewed as a predictor, and when the window size is set to M, it operates as follows: at the i-th moment, the model receives the first M tokens as input, that is, , and calculate the next tag accordingly Based on these probabilities, a PMF vector can be constructed ,Right now:

[0028] Here D is the size of the dictionary X, the vector Sort by descending order of their element values. In addition, define is a permutation whose scope is the set of integers , and satisfy .

[0029] Specifically, represents the ranking of token j at time i. We define the ranking of the input sequence at time i as token The ranking at the i-th moment is denoted as

[0030] The specific calculation formula of the probability mass function is:

[0031] in, .

[0032] The probability value of each token is obtained to form a probability distribution list, and the ranking number of the current token is recorded, specifically: Convert the integer encoding of tokens into a high-dimensional vector and input it into a large language model; Process high-dimensional vectors through multi-layer neural networks to capture the dependencies and contextual information between words; In each layer, a self-attention mechanism is used to calculate the relationship of each token with other tokens and analyze the context of the entire input sequence; Calculate the probability distribution of each token through the probability mass function, and sort all tokens according to the calculated probability distribution; Get the position of the current token in the sorted list and record its ranking number; The sorted tokens and their corresponding probability values ​​form a probability distribution list and output the probability distribution list and the ranking number of the current token.

[0033] S4, move the context window back one token and update the context information until all tokens are processed; The context window is moved back by one token, and the context information is updated until all tokens are processed. Specifically: Initialize the pointer by tracking the position of the token currently being processed; Use an iterator to traverse from the beginning of the sequence to the end; In each iteration, according to the current pointer position, the current token is extracted from the input sequence and added to the context window; Perform the required task using the information in the current context window and move the pointer back one position to point to the next token; When the pointer reaches the end of the sequence, stop the loop.

[0034] S5, record the ranking numbers of all tokens in order to form a number list; In each iteration, the sequence number of the current token (usually an integer starting from 0 or 1) is added to the sequence number list. The sequence number can be a consecutive integer or a number generated according to a specific rule.

[0035] S6, select any lossless compression algorithm to compress the ranking sequence and output the final text compression result.

[0036] token The codeword length is set to The codeword length here is chosen to satisfy Kraft's inequality, which means that there exists a prefix-free code whose codeword length follows the above definition. Kraft's inequality ensures that a given set of codeword lengths can form a valid prefix-free code. The performance of this compression scheme can be measured by the compression ratio, which is defined as follows:

[0037] in In the uncompressed state The number of bits in the sequence, and NT is the total number of tokens in the entire sequence. This expression quantifies the effectiveness of the compression scheme relative to the original data.

[0038] The method of selecting any lossless compression algorithm to compress the ranking sequence and outputting the final text compression result is as follows: Establish input and output flows with the ranking sequence; Construct a hash function to count all character codes represented by two bytes and record the number of occurrences of each character; Construct a minimum heap based on the hash function and sort the character statistics to get the Huffman tree; Encode the characters of the Huffman tree leaf nodes according to the principle of left 0 and right 1, and write the encoding results into the array corresponding to the array used when counting characters; Perform pre-order traversal and post-order traversal on the Huffman tree to obtain the pre-order sequence and post-order sequence; Write the pre-order sequence and post-order sequence to the first position of the compressed file in the form of objects; Re-read the text to be compressed, get each character and call the hash function to get the new code of the character, convert the code into a number and write it into a byte, and write the byte into the compressed file; Repeat the above process, and when the text to be compressed is read in and all characters are processed, a compressed file is obtained.

[0039] Using the Huffman tree to compress the ranking sequence and output the final text compression result has the following benefits: Highly efficient compression rate: Frequency-based encoding: Huffman tree constructs encoding based on the frequency of characters (or symbols) in a sequence. Characters with higher frequencies are assigned shorter codes, while characters with lower frequencies are assigned longer codes. This frequency-based encoding method can minimize the length of the encoded data and improve the compression ratio.

[0040] Lossless compression: Huffman coding is a lossless compression algorithm, which means that the original data can be completely restored after decompression without any information loss. This is especially important for application scenarios that require data integrity.

[0041] Greedy algorithm: The construction of Huffman tree is based on the greedy algorithm, which is relatively simple and easy to implement. It can quickly generate the optimal code, making the compression process efficient and easy to operate.

[0042] Automated processing: In practical applications, computer programs can be used to automatically count character frequencies, construct Huffman trees, and generate codes, thereby reducing the complexity and time cost of manual operations.

[0043] Data transmission: In some network transmission protocols, Huffman trees can be used to optimize the transmission of data packets, reduce bandwidth usage, and increase data transmission speed.

[0044] Reduce storage space: Huffman coding can significantly reduce the storage space required for ranking sequences, which is especially important for application scenarios with limited storage space.

[0045] Speed ​​up data transmission: The compressed data is smaller in size, so it can be transmitted to the target location faster, improving the efficiency of data transmission.

[0046] In summary, using Huffman tree to compress ranking sequences and output the final text compression results has multiple benefits, such as high compression rate, simple construction process, wide application scenarios, and optimized data storage and transmission. These benefits make Huffman coding an important tool in the field of data compression.

[0047] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0048] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0049] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0050] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0051] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0052] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for efficient lossless text compression based on a large language model, characterized in that: The following steps are involved: Preprocess the compressed text data to obtain the initial token sequence; Select and load a pre-trained large language model, initialize the model parameters and extract the current context information according to the context window size; Use the large language model to analyze the current context information, obtain the probability value of each token, form a probability distribution list, and record the ranking number of the current token; Move the context window back by one token and update the context information until all tokens are processed; Record the ranking numbers of all tokens in order to form a number list; Select any lossless compression algorithm to compress the ranking sequence and output it to obtain the final text compression result.

2. The method for efficient lossless text compression based on a large language model according to claim 1, characterized in that: The text data to be compressed is preprocessed to obtain an initial token sequence, specifically: Obtaining text data to be compressed, and performing denoising and standardization on the text data; Dynamically split the denoised and standardized text data into several tokens based on the adaptive word segmentation and tokenization strategy; Assign a unique identifier to each token encoding to obtain the initial token sequence.

3. The method for efficient lossless text compression based on a large language model according to claim 2, characterized in that: The method selects and loads a pre-trained large language model, initializes model parameters, and extracts current context information according to the context window size, specifically: Select a pre-trained large language model based on the specific application scenario; Load pre-trained large language models based on deep learning frameworks and corresponding libraries; Setting model parameters of a pre-trained large language model, the model parameters including but not limited to a learning rate, a batch size, and a context window size; Use a tokenizer to parse the input text data into tokens; According to the context window size, extract the text before the current token from the input text as context; If the context window size exceeds the length of the input text, the text is processed step by step using a sliding window technique, combining the content of the current window with the content of the previous window; Combine the current token and context information into a matrix form as the input sequence.

4. The method for efficient lossless text compression based on a large language model according to claim 3, characterized in that: The probability value of each token is obtained to form a probability distribution list, and the ranking number of the current token is recorded, specifically: Convert the integer encoding of tokens into a high-dimensional vector and input it into a large language model; Process high-dimensional vectors through multi-layer neural networks to capture the dependencies and contextual information between words; In each layer, a self-attention mechanism is used to calculate the relationship of each token with other tokens and analyze the context of the entire input sequence; Calculate the probability distribution of each token through the probability mass function, and sort all tokens according to the calculated probability distribution; Get the position of the current token in the sorted list and record its ranking number; The sorted tokens and their corresponding probability values ​​form a probability distribution list and output the probability distribution list and the ranking number of the current token.

5. The method for efficient lossless text compression based on a large language model according to claim 4, characterized in that: The context window is moved back by one token, and the context information is updated until all tokens are processed. Specifically: Initialize the pointer by tracking the position of the token currently being processed; Use an iterator to traverse from the beginning to the end of the sequence; In each iteration, according to the current pointer position, the current token is extracted from the input sequence and added to the context window; Perform the required task using the information in the current context window and move the pointer back one position to point to the next token; When the pointer reaches the end of the sequence, stop the loop.

6. The method for efficient lossless text compression based on a large language model according to claim 5, characterized in that: The ranking numbers of all tokens are recorded in order to form a list of numbers, specifically: In each iteration, the sequence number of the current token is added to the sequence number list. The sequence number can be a consecutive integer or a number generated according to a specific rule.

7. The method for efficient lossless text compression based on a large language model according to claim 6, characterized in that: The method of selecting any lossless compression algorithm to compress the ranking sequence and outputting the final text compression result is as follows: Establish input and output flows with the ranking sequence; Construct a hash function to count all character codes represented by two bytes and record the number of occurrences of each character; Construct a minimum heap based on the hash function and sort the character statistics to get the Huffman tree; Encode the characters of the Huffman tree leaf nodes according to the principle of left 0 and right 1, and write the encoding results into the array corresponding to the array used when counting characters; Perform pre-order traversal and post-order traversal on the Huffman tree to obtain the pre-order sequence and post-order sequence; Write the pre-order sequence and post-order sequence to the first position of the compressed file in the form of objects; Re-read the text to be compressed, get each character and call the hash function to get the new code of the character, convert the code into a number and write it into a byte, and write the byte into the compressed file; Repeat the above process, and when the text to be compressed is read in and all characters are processed, a compressed file is obtained.

Images