Text compression method, text decompression method, model training method, device and equipment

Abstract

The invention relates to the technical field of text processing, in particular to a text compression method, a text decompression method, a model training method, devices and equipment.The text compression method comprises the steps that a to-be-compressed target text is obtained, and the to-be-compressed target text is input into a feature extraction module of a text compression model; using a deep learning technology to capture semantic and syntactic features of the text, and expressing the extracted key information in the form of an embedded vector sequence; and a q-former quantization processing module is used for carrying out quantization processing on the embedded vector sequence, and the continuous embedded vector sequence is converted into a discrete token sequence, so that effective compression of text information is realized. By adopting a form of combining a feature extraction module and a q-former structure, the redundancy of text data can be remarkably reduced, and meanwhile, the high-quality restoration capability of an original text is kept.

Description

Technical Field

[0001] The present application relates to the technical field of text processing, and in particular to a text compression method, a text decompression method, a model training method, a device and equipment. Background Art

[0002] With the rapid development of information technology, the amount of global data has exploded. In this context, text data, as an important form of information transmission and knowledge storage, its processing and compression technology has become a hot topic in research.

[0003] Conventional text compression techniques, such as Huffman coding and Lempel-Ziv-Welch (LZW) coding, achieve compression by analyzing the frequency of occurrence of characters in the text, constructing optimal prefix codes or through string matching. For example, Huffman coding achieves compression by constructing a binary tree based on character frequency and assigning a unique variable-length code to each character. The LZW algorithm achieves compression by constructing a string dictionary and replacing repeated strings with indexes in the dictionary.

[0004] However, with the rapid increase in the amount of text data and the complexity of language structure, conventional text compression technology has a low compression rate and poor decompression effect. Summary of the invention

[0005] The purpose of this application is to provide a text compression method, a text decompression method, a model training method, a device and equipment, which can improve the compression rate and improve the decompression effect.

[0006] In a first aspect, a text compression method is provided, comprising:

[0007] Get the target text to be compressed;

[0008] Acquire a text compression model, wherein the text compression model includes: a feature extraction module and a q-former quantization processing module;

[0009] The target text to be compressed is input into the text compression model, and the feature extraction module is used to perform feature extraction according to the target text to be compressed to obtain an embedded vector sequence; the embedded vector sequence is quantized by the q-former quantization processing module to obtain a token sequence to achieve text compression.

[0010] In a preferred example, the present application may be further configured as follows: the text compression model further includes: a preprocessing module;

[0011] Before extracting features using the feature extraction module according to the target text to be compressed to obtain the embedded vector sequence, the method further includes:

[0012] Using the preprocessing module to preprocess the target text to be compressed to obtain a processed text;

[0013] Correspondingly, the feature extraction module is used to extract features according to the target text to be compressed to obtain an embedded vector sequence, including:

[0014] The feature extraction module is used to extract features from the processed text to obtain an embedded vector sequence.

[0015] In a preferred example, the present application can be further configured as follows: the feature extraction module includes: GPT or XLNet.

[0016] In a preferred example, the present application may be further configured as follows: the q-former quantization processing module is used to quantize the embedded vector sequence to obtain a token sequence to achieve text compression, including:

[0017] Acquire a quantization granularity; based on the quantization granularity, use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

[0018] In a second aspect, a text decompression method is provided, comprising:

[0019] Obtaining a token sequence and a request language, wherein the token sequence is obtained by the text compression method according to any one of the first aspects;

[0020] Get the large language model for decompression;

[0021] The token sequence and the request language are input into the large language model, so as to perform text restoration on the token sequence using the large language model according to the request language to obtain an output text.

[0022] In a third aspect, a model training method is provided, including:

[0023] Get multiple training original texts, request language, and token sequences corresponding to each of the multiple training original texts;

[0024] Acquire a training model, wherein the training model includes a compression training model and a large language training model, wherein the compression training model includes: a feature extraction training module and a q-former quantization processing training module;

[0025] Input the original training text into the compression training model, perform feature extraction using the feature extraction training module according to the original training text, and obtain a training embedded vector sequence; perform quantization processing on the training embedded vector sequence using the q-former quantization processing training module, and obtain a training token sequence;

[0026] Inputting the training token sequence and the request language into the large language training model, so as to perform text restoration on the training token sequence using the large language training model according to the request language to obtain a training output text;

[0027] The training model is iteratively trained according to the training output text and the training original text to obtain a text processing model, wherein the text processing model includes: a text compression model and a large language model, wherein the text compression model is used to compress the text, and the large language model is used to decompress a token sequence obtained from the compressed text.

[0028] In a fourth aspect, a text compression device is provided, comprising:

[0029] The first acquisition module is used to acquire the target text to be compressed; acquire a text compression model, wherein the text compression model includes: a feature extraction module and a q-former quantization processing module;

[0030] The compression module is used to input the target text to be compressed into the text compression model, so as to use the feature extraction module to perform feature extraction according to the target text to be compressed to obtain an embedded vector sequence; and use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

[0031] In a fifth aspect, a text decompression device is provided, comprising:

[0032] A second acquisition module is used to acquire a token sequence and a request language, wherein the token sequence is obtained by the text compression method according to any one of the first aspects; and to acquire a large language model for decompression;

[0033] The decompression module is used to input the token sequence and the request language into the large language model, so as to perform text restoration on the token sequence using the large language model according to the request language to obtain an output text.

[0034] In a sixth aspect, a model training device is provided, comprising:

[0035] The third acquisition module is used to acquire a plurality of training original texts, a request language, and a token sequence corresponding to each of the plurality of training original texts; acquire a training model, wherein the training model includes a compression training model and a large language training model, wherein the compression training model includes: a feature extraction training module and a q-former quantization processing training module;

[0036] The training module is used to input the original training text into the compression training model, so as to use the feature extraction training module to perform feature extraction according to the original training text to obtain a training embedded vector sequence; use the q-former quantization processing training module to quantize the training embedded vector sequence to obtain a training token sequence; input the training token sequence and the request language into the large language training model, so as to use the large language training model to perform text restoration on the training token sequence according to the request language to obtain a training output text; iteratively train the training model according to the training output text and the original training text to obtain a text processing model, wherein the text processing model includes: a text compression model and a large language model, wherein the text compression model is used to compress the text, and the large language model is used to decompress the token sequence obtained from the compressed text.

[0037] In a seventh aspect, an electronic device is provided, including:

[0038] one or more processors;

[0039] Memory;

[0040] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the method shown in any possible implementation of the first aspect, or, perform operations corresponding to the method shown in an implementation of the second aspect, or, perform operations corresponding to the method shown in an implementation of the third aspect.

[0041] In an eighth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, at least one program, a code set or an instruction set is loaded by a processor and executes operations corresponding to the method shown in any possible implementation manner in the first aspect, or executes operations corresponding to the method shown in the implementation manner in the second aspect, or executes operations corresponding to the method shown in the implementation manner in the third aspect.

[0042] In a ninth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements operations corresponding to the method shown in any possible implementation manner in the first aspect, or operations corresponding to the method shown in the implementation manner in the second aspect, or operations corresponding to the method shown in the implementation manner in the third aspect.

[0043] In summary, the text compression method provided by this application includes the following beneficial technical effects:

[0044] Obtain the target text to be compressed, input the target text to be compressed into the feature extraction module of the text compression model, use deep learning technology to capture the semantic and syntactic features of the text, extract the key information and express it in the form of an embedded vector sequence; use the q-former quantization processing module to quantize the embedded vector sequence, convert the continuous embedded vector sequence into a discrete token sequence, and achieve effective compression of text information. The combination of the feature extraction module and the q-former structure can significantly reduce the redundancy of text data while maintaining the high-quality restoration capability of the original text.

[0045] In addition, the present application also provides a text decompression method, model training method, device and equipment, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a schematic diagram of an application scenario of a text compression method provided in an embodiment of the present application;

[0048] Figure 2 This is a flowchart of a text compression method provided by an embodiment of the present application;

[0049] Figure 3 It is a flowchart of a text decompression method provided by an embodiment of the present application;

[0050] Figure 4 This is a flowchart of a text compression + decompression method provided by an embodiment of the present application;

[0051] Figure 5 It is a flow chart of a model training method provided in an embodiment of the present application;

[0052] Figure 6 is a structural schematic diagram of a text compression device provided in an embodiment of the present application;

[0053] Figure 7 It is a structural schematic diagram of a text decompression device provided in an embodiment of the present application;

[0054] Figure 8 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0055] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the present application, they are protected by the patent law.

[0057] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0060] At present, conventional text compression algorithms, such as Huffman coding and LZW coding, achieve compression by analyzing the frequency of occurrence of characters in the text, constructing the optimal prefix code or through string matching. For example, Huffman coding achieves compression by constructing a binary tree based on character frequency and assigning a unique variable-length code to each character. The LZW algorithm achieves compression by constructing a string dictionary and replacing repeated strings with indexes in the dictionary.

[0061] These algorithms work well when processing short texts and simple language structures, but their compression rate and decompression effects are often unsatisfactory when facing long texts and complex language structures. In addition, these algorithms have difficulty processing text content with rich context and high language dependence, resulting in information loss or compression distortion.

[0062] Therefore, the inventors found that the conventional technology has the following defects:

[0063] Limited compression rate: It is difficult to further improve the compression rate when dealing with long texts and complex language structures.

[0064] Poor decompression effect: During the compression process, conventional algorithms may lose some key contextual information, resulting in degraded quality of the decompressed text.

[0065] Difficulty in processing complex language structures: Conventional algorithms mainly rely on character frequency and local pattern matching, and it is difficult to effectively process text content with rich context and high language dependence.

[0066] In recent years, the development of artificial intelligence technology, especially large language models (LLM), has provided a new perspective for text processing. LLM can understand complex language structures and contextual information through deep learning of large amounts of text data, and has demonstrated outstanding capabilities in text generation, translation, summarization and other fields. However, although LLM has made significant progress in text generation and other aspects, its application in the field of text compression is still relatively rare, especially in achieving high compression rates and high-quality restoration, and there is still a lack of effective solutions.

[0067] Based on this, in order to solve the problem of low compression rate and poor decompression effect of existing text compression technology when facing long texts and complex language structures, the embodiment of the present application proposes a new text compression technology by utilizing the natural language processing capabilities of a large language model (LLM), so as to achieve a text compression technology with high compression rate and high-quality restoration. Specifically, the present invention extracts text features through a deep learning encoder, combines quantization processing with a q-former module, and generates predictions through an autoregressive language model (LLM). The present invention can effectively compress text data while maintaining a high restoration quality, especially in scenarios where long texts and complex language structures are processed.

[0068] The embodiment of the present application not only improves the efficiency of text compression, but also ensures the quality of the text after decompression, which has important practical application value for application scenarios that need to process a large amount of text data, such as cloud computing, big data analysis, mobile communications, and digital storage. Through the technical solution of the embodiment of the present application, while maintaining the integrity of text information, the resources required for data storage and transmission can be significantly reduced, providing users with a more efficient and economical text processing solution.

[0069] Please refer to Figure 1 , Figure 1 A scenario diagram of an application of a text compression method provided in an embodiment of the present application, the text-based compression method can be applied to a text compression system. In some embodiments, the text compression system includes an electronic device and a terminal device, and the terminal device can send the target text to be compressed. Of course, it can also be other servers that send target files to the electronic device, or read the target files of the electronic device itself, which is not limited in the embodiment of the present application. Among them, the terminal device includes but is not limited to: mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc. Electronic devices include but are not limited to servers or terminal devices, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The electronic device is used to implement a text compression method. The terminal device and the electronic device can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. It can be understood that the above is only an example, and this embodiment is not limited here.

[0070] The present application embodiment provides a text compression method, such as Figure 2 As shown, the method includes:

[0071] S101, obtaining a target text to be compressed;

[0072] In the embodiment of the present application, the target text to be compressed is the target text uploaded by the terminal device or the target text in the electronic device.

[0073] For example, in the field of cloud computing, the target file may be a large number of log files generated by cloud computing platform servers, applications, etc. The log files include but are not limited to operating status, error information, and user access records; the target file may also be user data stored in cloud storage services, such as documents, pictures, and videos. In the field of big data analysis, the target file may be data from various sources, such as social media, IoT devices, and enterprise databases. When data is transmitted or stored, the large amount of data requires file compression to save bandwidth and storage space. In the field of mobile communications, the target file may be text messages, multimedia content, or voice call records, etc., which may need to be compressed before transmission to save transmission bandwidth and costs; the target file may also be data transmitted by mobile users when accessing web pages.

[0074] It should be noted that the embodiments of the present application can effectively process texts in different languages ​​and formats, and do not limit the language and format of the target text.

[0075] S102, obtaining a text compression model, the text compression model comprising: a feature extraction module and a q-former quantization processing module;

[0076] S103, inputting the target text to be compressed into the text compression model, using the feature extraction module to perform feature extraction according to the target text to be compressed, and obtaining an embedded vector sequence; using the q-former quantization processing module to quantize the embedded vector sequence, and obtaining a token sequence, so as to realize text compression.

[0077] Among them, the feature extraction module is used to extract the features of the target file. In some embodiments, the feature extraction module can use a deep learning encoder to process the text; the encoder can be a model based on the Transformer architecture, such as BERT. The feature extraction module can capture the deep semantic and syntactic features of the text; convert the text into an embedded vector sequence in a high-dimensional space. It can be understood that the embedded vector sequence contains the key information of the text and can provide a basis for subsequent compression.

[0078] The q-former quantization processing module refers to a neural network module specially designed for quantizing embedded vector sequences, which can convert continuous vector values ​​into discrete token values, thereby realizing text compression. The q-former quantization processing module receives the embedded vector sequence output by the encoder and performs quantization processing on it. Among them, the q-former quantization processing module can use the embedded vector sequence output by the feature extraction module as the input data of this module by learning the distribution of the embedded vector sequence, discretize the continuous value, and map it to a limited token set to obtain a token sequence. The token sequence after quantization has a high compression rate while retaining the key information of the original text. The token sequence refers to the result after quantization processing, which is a series of discrete symbols or marks used to represent the approximate representation of the original text data under quantization, thereby realizing the compressed storage of the text. The size of the token sequence obtained in the embodiment of the present application is much smaller than the target text, but the embodiment of the present application adopts an innovative feature extraction module, namely the encoder-q-former structure, which can significantly reduce the redundancy of text data while maintaining the high-quality restoration capability of the original text. The encoder uses deep learning technology to capture the semantic and syntactic features of the text, while the q-former module efficiently compresses these features, so that even if the token sequence is small, it is still sufficient to support high-quality text restoration.

[0079] It can be seen that in the embodiment of the present application, the target text to be compressed is obtained, the target text to be compressed is input into the feature extraction module of the text compression model, the semantic and syntactic features of the text are captured by deep learning technology, and the extracted key information is expressed in the form of an embedded vector sequence; the embedded vector sequence is quantized by the q-former quantization processing module, and the continuous embedded vector sequence is converted into a discrete token sequence, thereby achieving effective compression of the text information. The combination of the feature extraction module and the q-former structure can significantly reduce the redundancy of the text data while maintaining the high-quality restoration capability of the original text.

[0080] In a possible implementation of the embodiment of the present application, the text compression model further includes: a preprocessing module;

[0081] Before extracting features using a feature extraction module according to the target text to be compressed to obtain an embedded vector sequence, the method further includes: preprocessing the target text to be compressed using a preprocessing module to obtain a processed text;

[0082] Correspondingly, the feature extraction module is used to extract features according to the target text to be compressed to obtain an embedded vector sequence, including: using the feature extraction module to extract features from the processed text to obtain an embedded vector sequence.

[0083] Among them, the preprocessing module is responsible for preliminarily processing the input text data (target text) to improve the efficiency and accuracy of subsequent processing. The preliminary processing includes at least one of the following:

[0084] Word segmentation processing: Decompose the continuous text string of the target text into meaningful lexical units.

[0085] Stop word removal processing: Delete common and meaningless words in the target text, such as "de" (of), "shi" (is), etc.

[0086] Punctuation processing: Identify and retain the punctuation marks in the target text because they have an important impact on the text structure.

[0087] It can be seen that in the embodiments of the present application, by preprocessing the target text, cleaning and processing the target text, redundant information can be removed, the data quality can be improved, and thus the compression quality can be improved.

[0088] A possible implementation manner of the embodiments of the present application is that the feature extraction module includes: GPT (Generative Pre-trained Transformer) or XLNet.

[0089] The extraction module of the Transformer architecture directly captures the dependencies in the text through the self-attention mechanism, greatly improving the efficiency of the model in processing text data. Among them, GPT is a language model based on the Transformer architecture. XLNet is another language model based on the Transformer architecture, which adopts the method of permutation language model.

[0090] It can be seen that in the embodiments of the present application, in the feature extraction stage, different pre-trained models, such as GPT or XLNet, can be used to adapt to different types of text data.

[0091] A possible implementation manner of the embodiments of the present application is to use the q-former quantization processing module to perform quantization processing on the embedded vector sequence to obtain a token sequence to achieve text compression, including:

[0092] Obtain the quantization granularity; based on the quantization granularity, use the q-former quantization processing module to perform quantization processing on the embedded vector sequence to obtain a token sequence to achieve text compression.

[0093] The quantization granularity refers to the degree of fineness or resolution used when quantizing the embedded vector during the text compression process, which determines the level of detail of the information that the quantized token can represent. In an embodiment of the present application, when a large amount of text data needs to be efficiently stored or transmitted, the text needs to be embedded first and converted into an embedded vector sequence; according to the preset quantization granularity, these vectors are quantized using the q-former quantization processing module, the continuous vector space is divided into a series of discrete intervals, and each vector is mapped to the token corresponding to the interval to which it belongs, thereby achieving text compression. In the quantization processing stage, the number of compressed tokens can be changed by adjusting the output dimension of the q-former to adjust the quantization granularity and achieve different compression rates and restoration quality. The quantization granularity can be adjusted according to actual needs, for example, the embedded vector can be quantized to a set of 256, 512 or 1024 tokens.

[0094] Specifically, the quantization granularity can be determined based on the characteristics of the text or the compression requirements or the default granularity. For example, for texts that need to highly retain semantic information (such as legal documents, medical reports, etc.), a finer quantization granularity is set; while for texts that do not require high semantic information (such as log data, social media content, etc.), a coarser quantization granularity can be set in exchange for a higher compression ratio.

[0095] Therefore, in some embodiments, the step of obtaining the quantization granularity can be implemented in a variety of ways:

[0096] The first way is to determine the quantization granularity through experience or expert knowledge, and a suitable quantization granularity can be set manually. The second way is to dynamically determine the quantization granularity through an automated algorithm. For example, analyze the vocabulary distribution, sentence length and other features in the text, and automatically adjust the quantization granularity to optimize the compression effect based on the correspondence between the features and the preset features and the granularity, so as to more flexibly adapt to various compression requirements. It is understandable that other methods can also be used to implement the step of obtaining the quantization granularity, such as an adjustment mechanism based on user feedback. This is not limited here, and the specific implementation method will vary depending on the application scenario and requirements.

[0097] Furthermore, an embodiment of the present application provides a text decompression method, which includes: obtaining a token sequence and a request language, wherein the token sequence is obtained through the above-mentioned text compression method; obtaining a decompression model; inputting the token sequence into the decompression model to use the decompression model to perform text restoration on the token sequence to obtain output text.

[0098] Furthermore, the decompression model is a large language model for decompression; see Figure 3 , Figure 3A text decompression method provided in an embodiment of the present application includes:

[0099] S201, obtaining a token sequence and a request language, wherein the token sequence is obtained by a text compression method;

[0100] The request language is used to instruct the large language model to perform a decoding operation, for example, "please decode the text according to the provided token sequence."

[0101] S202, obtaining a large language model for decompression;

[0102] S203: input the token sequence and the request language into the large language model, and perform text restoration on the token sequence using the large language model according to the request language to obtain an output text.

[0103] In the decoding stage, the large language model LLM uses its powerful language understanding ability to gradually predict and restore the original text to obtain the output text based on the compressed token sequence and the request language. The structure of the large language model is no longer limited in this embodiment of the application. The user can set it according to actual needs. The exemplary large language model is a QWEN2 structure.

[0104] The present application embodiment proposes a text compression technology based on a large language model (LLM), which aims to solve the challenges faced by existing text compression algorithms when processing long and complex language structure texts. Figure 4 This technical solution extracts the features of the input text through a deep learning encoder, converts it into an embedding vector sequence, and then uses the q-former module to quantize these vectors to generate a token sequence with a high compression rate. These tokens can retain the key information of the original text and are decoded through an autoregressive language model (LLM) to predict and restore the original text content.

[0105] It can be seen that the decompression method provided in the embodiment of the present application gradually expands the compressed token sequence through LLM, thereby restoring the original long text, which not only improves the compression rate but also ensures the integrity and accuracy of the text information.

[0106] Furthermore, an embodiment of the present application provides a model training method, including: obtaining multiple training original texts and token sequences corresponding to each of the multiple training original texts; obtaining a training model, the training model including a compression training model and a decompression training model, the compression training model including: a feature extraction training module, a q-former quantization processing training module; inputting the training original text into the compression training model, and performing feature extraction using the feature extraction training module according to the training original text to obtain a training embedded vector sequence; performing quantization processing on the training embedded vector sequence using the q-former quantization processing training module to obtain a training token sequence; inputting the training token sequence into the decompression training model, and performing text restoration on the training token sequence using the decompression training model to obtain a training output text; iteratively training the training model according to the training output text and the training original text to obtain a text processing model, the text processing model including: a text compression model and a decompression model, the text compression model is used to compress the text, and the decompression model is used to decompress the token sequence obtained from the compressed text.

[0107] Furthermore, the decompression training model is a large language training model; see Figure 5 , Figure 5 A model training method provided in an embodiment of the present application includes:

[0108] S301, obtaining multiple original training texts, a request language, and token sequences corresponding to each of the multiple original training texts;

[0109] Based on multiple on-site and off-site data from multiple fields, the training format is constructed as follows: request language (such as please restore the original text) + compressed token sequence + original text.

[0110] S302, obtaining a training model, the training model including a compression training model and a large language training model, the compression training model including: a feature extraction training module and a q-former quantization processing training module;

[0111] S303, inputting the original training text into the compression training model, performing feature extraction using the feature extraction training module according to the original training text, and obtaining a training embedded vector sequence; performing quantization processing on the training embedded vector sequence using the q-former quantization processing training module, and obtaining a training token sequence;

[0112] S304, inputting the training token sequence and the request language into the large language training model, so as to perform text restoration on the training token sequence according to the request language using the large language training model to obtain a training output text;

[0113] S305. Iteratively train the training model according to the training output text and the training original text to obtain a text processing model. The text processing model includes: a text compression model and a large language model. The text compression model is used to compress the text, and the large language model is used to decompress the token sequence obtained from the compressed text.

[0114] During the training phase, the large language training model learns how to predict and generate original text based on the compressed token sequence and the request language. For model parameters: The hidden layer size of LLM can be set between 128 and 512 to balance model complexity and performance. Use a large amount of text data to adjust model parameters; learn the generation rules of text in order to accurately restore the structure and semantic information of the original text from the compressed token sequence.

[0115] In one feasible method, only the original text is used for training, and the loss is calculated to achieve iterative training. Therefore, the difference between the training output text and the expected output is calculated, and the model parameters are adjusted according to the difference until the difference meets the requirements and the training stops.

[0116] The training process includes compression and decompression, where:

[0117] Compression process:

[0118] Step a101: input the preprocessed text.

[0119] Step a102: extract features through an encoder.

[0120] Step a103: Perform quantization processing through the q-former module.

[0121] Step a104: Generate a compressed token sequence.

[0122] Decompression process: The decompression process is the reverse process of the compression process and includes the following steps:

[0123] Step a201: Input a compressed token sequence.

[0124] Step a202: Use the autoregressive language model LLM to predict text based on the token sequence and context information.

[0125] Step a203: gradually expand the token sequence to restore the original text.

[0126] Furthermore, during the training phase of the autoregressive language model, different optimization algorithms, such as Adam or RMSprop, can be used to improve the training efficiency.

[0127] In summary, this embodiment provides an efficient and flexible text compression technology that can significantly reduce the resources required for data storage and transmission while maintaining high restoration quality.

[0128] Specifically, 1. High compression rate: Through quantization processing and LLM's predictive ability, a high compression rate is achieved. 2. High restoration quality: q-former's quantization processing has a high compression rate, while retaining the key information of the original text, and LLM's natural language processing capabilities ensure high-quality restoration of the text. 3. Wide applicability: It is suitable for texts in different languages ​​and formats and has broad application prospects.

[0129] It should be noted that the text compression method, text decompression method, and model training method provided in the embodiments of the present application can refer to each other and will not be elaborated in this embodiment.

[0130] A text compression device provided in an embodiment of the present application is introduced below. The text compression device described below and the text compression method described above can be referred to each other. The text compression device of this embodiment is set in an electronic device. Figure 6 , Figure 6 This is a structural block diagram of a text compression device according to one embodiment of the present application, comprising:

[0131] The first acquisition module 510 is used to acquire the target text to be compressed; acquire a text compression model, the text compression model includes: a feature extraction module, a q-former quantization processing module;

[0132] The compression module 520 is used to input the target text to be compressed into the text compression model, and use the feature extraction module to perform feature extraction according to the target text to be compressed to obtain an embedded vector sequence; and use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

[0133] In one achievable manner, the text compression model further includes: a preprocessing module;

[0134] Also includes:

[0135] A preprocessing module, used for preprocessing the target text to be compressed using the preprocessing module to obtain a processed text;

[0136] Accordingly, the compression module 520 is used to:

[0137] The feature extraction module is used to extract features from the processed text to obtain an embedded vector sequence.

[0138] In one achievable manner, the feature extraction module includes: GPT or XLNet.

[0139] In one achievable manner, the compression module 520 is configured to:

[0140] Obtain quantization granularity; based on the quantization granularity, use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

[0141] A text decompression device provided in an embodiment of the present application is introduced below. The text decompression device described below and the text decompression method described above can be referred to each other. The text decompression device of this embodiment is set in an electronic device. Figure 7 , Figure 7 : is a structural block diagram of a text decompression device according to one embodiment of the present application, comprising:

[0142] The second acquisition module 610 is used to acquire a token sequence and a request language, wherein the token sequence is obtained by a text compression method; and to acquire a large language model for decompression;

[0143] The decompression module 620 is used to input the token sequence and the request language into the large language model, so as to perform text restoration on the token sequence according to the request language using the large language model to obtain output text.

[0144] A model training device provided in an embodiment of the present application is introduced below. The model training device described below and the model training method described above can be referred to each other. The model training device of this embodiment is set in an electronic device. Figure 8 , Figure 8 This is a structural block diagram of a model training device according to one embodiment of the present application, comprising:

[0145] The third acquisition module 710 is used to acquire multiple training original texts, request languages, and token sequences corresponding to the multiple training original texts; acquire a training model, the training model includes a compression training model and a large language training model, and the compression training model includes: a feature extraction training module and a q-former quantization processing training module;

[0146] The training module 720 is used to input the original training text into the compression training model, so as to use the feature extraction training module to perform feature extraction based on the original training text to obtain a training embedded vector sequence; use the q-former quantization processing training module to quantize the training embedded vector sequence to obtain a training token sequence; input the training token sequence and the request language into the large language training model, so as to use the large language training model to perform text restoration on the training token sequence based on the request language to obtain a training output text; iteratively train the training model based on the training output text and the original training text to obtain a text processing model, wherein the text processing model includes: a text compression model and a large language model, wherein the text compression model is used to compress the text, and the large language model is used to decompress the token sequence obtained from the compressed text.

[0147] An electronic device is provided in an embodiment of the present application, such as Fig. 9 As shown, Fig. 9 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0148] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0149] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0150] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0151] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0152] Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0154] An embodiment of the present application provides a computer program product, including a computer program, which implements the corresponding contents of the aforementioned method embodiment when the computer program is executed by a processor.

[0155] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0156] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text compression method, characterized in that: include: Get the target text to be compressed; Acquire a text compression model, wherein the text compression model includes: a feature extraction module and a q-former quantization processing module; The target text to be compressed is input into the text compression model, and the feature extraction module is used to perform feature extraction according to the target text to be compressed to obtain an embedded vector sequence; the embedded vector sequence is quantized by the q-former quantization processing module to obtain a token sequence to achieve text compression.

2. The text compression method according to claim 1, characterized in that: The text compression model also includes: a preprocessing module; Before extracting features using the feature extraction module according to the target text to be compressed to obtain the embedded vector sequence, the method further includes: Using the preprocessing module to preprocess the target text to be compressed to obtain a processed text; Accordingly, the feature extraction module is used to extract features according to the target text to be compressed to obtain an embedded vector sequence, including: The feature extraction module is used to extract features from the processed text to obtain an embedded vector sequence.

3. The text compression method according to claim 1, characterized in that: The feature extraction module includes: GPT or XLNet.

4. The text compression method according to claim 1, characterized in that: The embedded vector sequence is quantized by the q-former quantization processing module to obtain a token sequence to achieve text compression, including: Acquire a quantization granularity; based on the quantization granularity, use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

5. A text decompression method, characterized in that: include: Obtaining a token sequence and a request language, wherein the token sequence is obtained by the text compression method according to any one of claims 1 to 4; Get the large language model for decompression; The token sequence and the request language are input into the large language model, so as to perform text restoration on the token sequence using the large language model according to the request language to obtain an output text.

6. A model training method, characterized in that: include: Get multiple training original texts, request language, and token sequences corresponding to each of the multiple training original texts; Acquire a training model, wherein the training model includes a compression training model and a large language training model, wherein the compression training model includes: a feature extraction training module and a q-former quantization processing training module; Input the original training text into the compression training model, perform feature extraction using the feature extraction training module according to the original training text, and obtain a training embedded vector sequence; perform quantization processing on the training embedded vector sequence using the q-former quantization processing training module, and obtain a training token sequence; Inputting the training token sequence and the request language into the large language training model, so as to perform text restoration on the training token sequence using the large language training model according to the request language to obtain a training output text; The training model is iteratively trained according to the training output text and the training original text to obtain a text processing model, wherein the text processing model includes: a text compression model and a large language model, wherein the text compression model is used to compress the text, and the large language model is used to decompress a token sequence obtained from the compressed text.

7. A text compression device, characterized in that: include: A first acquisition module, used for acquiring a target text to be compressed; Acquire a text compression model, wherein the text compression model includes: a feature extraction module and a q-former quantization processing module; The compression module is used to input the target text to be compressed into the text compression model, so as to use the feature extraction module to perform feature extraction according to the target text to be compressed to obtain an embedded vector sequence; and use the q-former quantization processing module to quantize the embedded vector sequence to obtain a token sequence to achieve text compression.

8. A text decompression device, characterized in that: include: A second acquisition module is used to acquire a token sequence and a request language, wherein the token sequence is obtained by the text compression method according to any one of claims 1 to 4; and to acquire a large language model for decompression; The decompression module is used to input the token sequence and the request language into the large language model, so as to perform text restoration on the token sequence using the large language model according to the request language to obtain an output text.

9. A model training device, characterized in that: include: The third acquisition module is used to obtain multiple training original texts, request languages, and token sequences corresponding to each of the multiple training original texts; Acquire a training model, wherein the training model includes a compression training model and a large language training model, wherein the compression training model includes: a feature extraction training module and a q-former quantization processing training module; The training module is used to input the original training text into the compression training model, so as to use the feature extraction training module to perform feature extraction according to the original training text to obtain a training embedded vector sequence; use the q-former quantization processing training module to quantize the training embedded vector sequence to obtain a training token sequence; input the training token sequence and the request language into the large language training model, so as to use the large language training model to perform text restoration on the training token sequence according to the request language to obtain a training output text; iteratively train the training model according to the training output text and the original training text to obtain a text processing model, wherein the text processing model includes: a text compression model and a large language model, wherein the text compression model is used to compress the text, and the large language model is used to decompress the token sequence obtained from the compressed text.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: perform the steps of the method according to any one of claims 1 to 6.

Images