Text compression method based on natural language features and self-information

Abstract

The invention belongs to the technical field of natural language information processing, and discloses a text compression method based on natural language features and self-information, and the specific technical scheme is as follows: step 1, carrying out data preprocessing, useless information deletion and multi-document retrieval simulation on a Google question and answer data set in an open source corpus to obtain a data set oriented to long context compression, obtaining a multi-document long text; 2, performing self-information evaluation on the multi-document long text to obtain phrase granularity and text self-information distribution, extracting natural language features of part-of-speech analysis, named entity analysis and dependency relationship analysis, and optimizing distribution based on the natural language features to obtain a compressed text after distribution optimization; and step 3, compressed text prompt setting, target question and answer model construction, question and answer performance index obtaining and application to downstream tasks, the method is very excellent in long text compression accuracy, and the text compression performance is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language information processing, and in particular relates to a text compression method based on natural language features and self-information. Background Art

[0002] The Transformer model was originally designed to solve the sequence-to-sequence (Seq2Seq) problem in machine translation. With its deep neural network structure, especially the innovative attention mechanism, it effectively captures the global dependencies of each element in the sequence, thereby demonstrating excellent performance in parallel processing of complex language structures. Despite this, the inherent time and space complexity of this mechanism grows quadratically with the input length, which forms a bottleneck for computational efficiency. In addition, the model's performance declines when processing sequences that exceed the length of the training data, indicating generalization limitations in long text understanding and generation tasks, which is particularly critical for applications such as question-answering systems that rely on long sequence understanding.

[0003] In order to reduce the cost of model training and reasoning for long text problems, long text preprocessing solutions are a relatively convenient and versatile research direction. However, most existing long text preprocessing solutions have significant information loss and require high computational overhead for high performance, making it difficult to balance versatility, information retention, and computing resource efficiency.

[0004] Prior art 1:

[0005] The text compression scheme based on self-information was mentioned by Li in his 2023 article (Y.Li.UnlockingContextContraintsofLLMS:EnhancingContextEfficiencyofLLMSwithSelf-Information-BasedContentFiltering[EB / OL].ttps: / / arxiv.org / abs / 2304.12102,2023-04-24.), and the proposed scheme is Selective Context. It uses a smaller model to evaluate the self-information of the text, and only uses the evaluated text self-information for compression. It proposes compression schemes at token granularity, phrase granularity, and sentence granularity. The advantage of the scheme based on self-information is that when the world knowledge of the small model and the world knowledge of the target model have a certain degree of overlap, it will have a good effect. At the same time, the compression speed, convenience and versatility of the scheme are all very good.

[0006] This solution cannot guarantee the alignment of the world knowledge of the small model and the world knowledge of the target model, and there is a certain compression loss. At the same time, the effect in specific downstream tasks will also be poor. For example, in the question-answering task, the text answer to a specific question is required, but the comprehensive compression will lose the information of the answer to the question.

[0007] LLMLingua was proposed by a research team at Microsoft. This technology is based on the self-information solution of the existing technology. The LLMLingua solution focuses more on the engineering part. It uses an iterative solution in the compression of token granularity to make the reasoning faster, and through the multi-segment processing of KV-cache, it can handle longer texts. At the same time, the solution also supports document-level compression solutions.

[0008] Prior art 2:

[0009] This technology adds token-scale compression and KV-cache solutions on the basis of the existing technology 1 (H. Jiang, Q. Wu, CY Lin, et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models [A] / / In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing [C]. Singapore: Association for Computational Linguistics, 2023: 13358-13376.), which makes the reasoning speed faster, but its performance in token-scale compression is low. At the same time, it also has the same disadvantages as technology 1. The effect in specific downstream tasks will also be poor.

[0010] The existing technology is still relatively rough in sorting and filtering the self-information of the text. As far as the concept of self-information is concerned, a very obvious defect is that the self-information fails to take into account the relationship between the contexts, and fails to extract and condense key information such as some proper nouns. Summary of the invention

[0011] In order to solve the technical problems existing in the prior art, the present invention provides a text compression method based on natural language features and self-information. The present invention further extracts text information so that the generated text can be better aligned with the knowledge of the target model, thereby enhancing the performance of compressed text in question-answering tasks.

[0012] To achieve the above purpose, the technical solution adopted by the present invention is: a text compression method based on natural language features and self-information, the specific steps are as follows:

[0013] Step 1: Dataset acquisition and processing: Preprocess the Google Q&A dataset in the open source corpus, delete useless information, simulate multi-document retrieval, obtain a dataset for long context compression, and then obtain multi-document long text;

[0014] Step 2: Self-information evaluation and distribution optimization based on natural language features: The phrase granularity and text self-information distribution are obtained by evaluating the self-information of the multi-document long text, and natural language features such as part-of-speech analysis, named entity analysis, and dependency analysis are extracted to optimize the distribution based on natural language features and obtain the compressed text after distribution optimization.

[0015] Step 3: Downstream task evaluation: Compress the text prompt setting, build the target question-answering model, obtain question-answering performance indicators and apply them to downstream tasks.

[0016] Optimize and filter text information, for text s = {s1, s2, ..., s n}, each text component has an importance expressed as w = {w1, w2, ..., w n};

[0017] Among them, w i and i are mutually corresponding, s i is the i-th word or phrase, w i It means s i The importance of w i is a real number;

[0018] The natural language features corresponding to each text are unified and expressed as θ = {θ1, θ2, ..., θ n};

[0019] Among them, θ i Indicates i The natural language characteristics of

[0020] The importance of each text unit will be adjusted and weighed according to the natural language features. i Further expressed as a mapping function based on natural language features, the importance of each new text unit is obtained, m = {θ1(w1), θ2(w2), ..., θ n (w n )};

[0021] Through black box optimization, each part-of-speech feature, each named entity, and each dependency relationship is optimized and defined to obtain a better parameter as θ i The scheme of the function;

[0022] After obtaining the reordered text units, the percentile-based filtering method is used to adaptively select the part with the largest amount of information, and the content with the largest amount of information is sorted in order to obtain the final compressed text, which is described in detail as follows:

[0023] First, assume that the obtained text self-information distribution is m=m1,m2,...,m n , retain the first p percentile texts, and calculate m p =np.percentile([m1,m2,...,m n ], 100-p) is the information content of the text of the pth percentile from large to small;

[0024] Among them, np.percentile(l, p) means to find the value of the pth percentile in the list l from small to large, and this value is obtained through the numpy library of Python;

[0025] Retain the vocabulary units whose information content is greater than or equal to the pth percentile and construct a compressed new text S′={s1, s2, ..., s t |m t ≥m p}.

[0026] The present invention is excellent in the accuracy of long text compression and improves the performance of text compression; the present invention can further improve the performance of the compression algorithm based on the self-information text compression algorithm, making it more accurate and effective in compression, and can also reduce the inference cost of the target model for long sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall flow chart of the present invention.

[0028] Figure 2 Optimizing flow charts for text information based on natural language features.

[0029] Figure 3 A bar chart showing the distribution ratio of common part-of-speech features for all uncompressed original texts.

[0030] Figure 4 This is the distribution diagram of the part-of-speech ratio before and after part-of-speech optimization.

[0031] Figure 5 This is a compression example analysis diagram. DETAILED DESCRIPTION

[0032] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] The text compression method based on natural language features and self-information firstly adopts the self-information evaluation technology of the existing scheme to obtain the text self-information distribution at the phrase granularity. Secondly, the distribution dynamic optimization scheme based on natural language features proposed by the present invention is used to tune the self-information distribution, which is then aligned with the target question-answering model, and finally a better compressed text is obtained under the same compression rate.

[0034] Figure 1 The technical route of the present invention is demonstrated. First, for various data sets, relevant multi-document long texts are collected, and the focus is on long-context tasks. Secondly, the self-information distribution of the text is obtained by evaluating the self-information of the multi-document long text. This technology adopts the above-mentioned self-information evaluation scheme. At the same time, natural language features such as part-of-speech analysis, named entity analysis, and dependency analysis are also extracted. The two are combined to perform distribution optimization based on natural language features to obtain compressed text after distribution optimization, which is finally used in downstream tasks.

[0035] The detailed flow chart of distribution optimization based on natural language features is as follows: Figure 2 As shown, first, text features are extracted based on the long text, including part-of-speech features, named entity features and dependency grammatical features; secondly, the long text self-information is adjusted through dynamic weight adjustment; finally, the top k percent of the text is selected for extraction through re-ranking to obtain the final compressed text.

[0036] Table 1 Part of speech table

[0037]

[0038]

[0039] Part-of-Speech Tagging is a classic natural language processing task. Part-of-speech is a special term in linguistics, which is obtained by classifying a word based on grammatical features. The current unified part-of-speech classification system mainly includes adjectives (ADJ), adverbs (ADV), nouns (NOUN), verbs (VERB), etc. At the same time, in the current global unified part-of-speech tags, part-of-speech tags are divided into three categories as a whole. The first category is open words, including adjectives, adverbs, nouns, etc. These words can be expanded, which means that the model may not learn newly created words; the second category is non-open words, including prepositions, conjunctions, etc. These words are fixed with the language and cannot be changed, but the position of these words in the article can express the structural relationship, so they are also relatively important; the third category is other words, including punctuation marks, symbols, etc.; all the parts of speech used are shown in Table 1.

[0040] Named Entity Recognition is also a classic natural language processing task. Named entities refer to entities with some specific meanings in unstructured texts, including proper nouns such as names of people and places. At present, there are two more important issues in the NER task, namely recognition and classification. This task is an important basic tool for many natural language processing tasks, so the present invention will also be designed and experimented based on this. At the same time, the commonly used data annotation methods for named entity recognition will also be introduced here. NER is a sequence annotation problem. Generally speaking, there are two types of implementation schemes, one is rule-based and the other is machine learning-based. The rule-based NER method is actually a scheme based on some specific instructions and rules, which is actually equivalent to having a "named entity dictionary" to correspond and mark the text. It can target some specific models, such as "Beijing, China", which can clearly identify "Beijing" as a place name and "China" as a country name.

[0041] Of course, the cost of rule-based solutions is relatively low and it is also relatively fast. It is a very convenient solution to obtain most of the named entities in the corpus. However, if some entities do not conform to the rules, then it is necessary to use the NER method based on machine learning. When the entity pattern of a corpus is more complex, it is necessary to use a machine learning solution. Even if it is an unfamiliar entity, the solution has the ability to generalize and identify it with the correct label.

[0042] Table 2 Common dependency table

[0043]

[0044] Dependency Parsing is a method of analyzing the dependency relationship of words in a sentence and constructing a syntax tree, which helps to understand the structure and semantic relationship of a sentence. The dependency relationship is a binary asymmetric relationship between a central word and its subordinates. The central word of a sentence is usually a verb, and all other words either depend on the central word or are associated with it through a dependency path. Therefore, the dependency structure is actually a labeled directed graph, with the direction from the central word to the subordinate word. Generally, what is obtained through existing natural language processing tools is head pointing to child. At the same time, since there are many relationships used in dependency relationships, the present invention only uses more commonly used and more important tags, as shown in Table 2.

[0045] The present invention mainly uses the above three natural language features. At present, there are also relatively convenient tool integrations that can simply realize the extraction of the above natural language features, such as spaCy, NLTK, THULAC and stanfordnlp, etc. The present invention adopts the integration of spaCy tools. spaCy has higher performance, so compared with NLTK, it is more focused on practical applications in production environments. The tool has relatively comprehensive functions, which supports multiple languages, including English, German, Chinese, etc. It can perform common NLP tasks such as word segmentation, part-of-speech tagging, named entity recognition, dependency grammar analysis, word vector representation, text classification and sentiment analysis, and also supports certain expansion and customization functions.

[0046] Secondly, the present invention will explain how to optimize and filter text information based on the characteristics of the text. First, the present invention expresses the problem in a mathematical formula. For the text s = {s1, s2, ..., s n}, the present invention has an importance representation for each text component unit as w={w1, w2, ..., W n}. Among them, w i and i are mutually corresponding, s i is the i-th word or phrase, and w i It means s i The importance of is a real number. The present invention uses self-information as its importance representation. At the same time, the present invention unifies the natural language features corresponding to each text and expresses them as θ = {θ1, θ2, ..., θ n}, where θ i Indicates iThe natural language features of the text unit include its part of speech features, position in the dependency relationship, whether it is a named entity, etc. Therefore, it is possible that these relationships of some words are the same or similar. However, since the natural language features are extracted more abundantly, it can be temporarily assumed that these natural language features are more or less different. Next, the present invention will adjust and weigh the importance of the natural language features of each text unit.

[0047] If θ i Further expressed as a mapping function based on natural language features, the importance of each new text unit can be obtained, m = {θ1(w1), θ2(w2), ..., θ n (w n )}, according to the optimized importance, the present invention can sort and conduct corresponding research and other work.

[0048] Of course, how to find the right θ i The mapping function is also a key point. The goal of this invention is to align the question-answering system to better improve the ability to compress text. The optimization objective function is as follows:

[0049]

[0050] in, is the optimized evaluation index (the present invention adopts the Exact Match index), For the real answer, is the target question answering model, s(θ1, θ2, ..., θ n ) is the compressed text adjusted by natural language features. The present invention only optimizes θ, sets multiple θ channels, and includes the following weight dictionaries: ADJ, ADV, NOUN, NUM, PRON, PROPN, VERB, NER, NSUBJ, CSUBJ, DOBJ, IOBJ, POBJ. Among them, ADJ to VERB are all part-of-speech categories, NER is a named entity category, and NSUBJ to POBJ are all dependency grammar categories. The above weights can be optimized by using a Bayesian optimization algorithm, and the OpenBox tool can be used to implement the relevant optimization.

[0051] Through the above scheme, the present invention can obtain the order of importance of texts. Among them, a relatively important part is the function θ i How to define it? Since the present invention does not adopt the method of training model, therefore, θ i The definition of θ is generally more heuristic. Each part-of-speech feature, each named entity, and each dependency relationship is optimized and defined through black box optimization. Finally, a better parameter is obtained as θi Function scheme.

[0052] Through the above scheme, the present invention obtains the reordered text units. Since the present invention has the information content of all text units, the present invention does not need to fix a threshold of information content for processing. The present invention uses a percentile-based filtering method to adaptively select the part with the largest amount of information, and sorts these contents with large amounts of information in order to obtain the final compressed text. The following is a description of the formula of this part.

[0053] First, assume that the text self-information distribution obtained after the above scheme is m=m1,m2,...,m n , we need to keep the text of the first p percentiles and calculate m p =np.percentile([m1,m2,...,m n ], 100-p) is the information content of the pth percentile text from large to small, where np.percentile(l, p) means finding the value of the pth percentile in list l from small to large. This method can be directly obtained through Python's numpy library.

[0054] Next, we selectively retain vocabulary units whose information content is greater than or equal to the pth percentile and construct a compressed new text S′={s1, s2, ..., s t |m t ≥m p}.

[0055] In short, percentage-based text extraction is a more flexible method that can retain the text content with the most information based on the distribution of information in a given context. In specific applications, the ratio of the maximum text token length supported by the target question-answering model to the current text token length can also be used as the text compression ratio for compression, which is a more convenient and ideal extraction solution.

[0056] Finally, the influence of the part-of-speech distribution of the compressed text on the performance of the target model will be further explored. First of all, the part-of-speech feature is a relatively simple feature for classifying each word unit, and it is also a very important feature processing method for understanding sentences in human language. For example, the sentence "Hobbits are a fictional race of people in the novels of Tolkien.", if it is extremely simplified to "Hobbits, fictional, race of people, Tolkien.". If the present invention only looks at the latter sentence, it can almost restore the original sentence that "Hobbits" is a race fictionalized by "Tolkien". Therefore, it can be clearly seen from this example that humans have a certain ability to understand compressed texts, and based on this assumption, the present invention can explore and find that the large model should also have a similar ability to understand compressed texts.

[0057] Secondly, the ability to understand such compressed text can usually be attributed to some specific categories, such as the classification of part-of-speech features. Looking at the above example again, "Hobbits" and "Tolkien" are both proper nouns, "fictional" is an adjective, and "race of people" is a noun phrase. Therefore, the present invention can further find that proper nouns, adjectives, noun phrases, etc. in compressed text can help text understanding.

[0058] According to the above analysis, the present invention will heuristically explore the most common parts of speech in the text, and the main parts of speech in a text are adjectives, adverbs, nouns, verbs, quantifiers, pronouns, and proper nouns. Of course, other less common parts of speech, such as interjections, will not be presented as charts for this exploration.

[0059] Figure 3 It represents the distribution ratio of common part-of-speech features of all uncompressed original texts. It can be seen from the figure that nouns are the component with the largest proportion. Obviously, the largest number in the text must also be nouns. Nouns are an important cornerstone of natural language texts. However, the second largest proportion is punctuation. This is obviously because punctuation is a key component to assist humans in reading, so it is normal to have a lot of punctuation, but there is actually no need for so many punctuation in compressed texts. Only some key punctuation needs to be retained. The next components with more parts are prepositions, proper nouns, and verbs. These three parts are also the parts that affect reading. The adjectives, adverbs, and several times analyzed previously occupy less parts, but they are also relatively important parts of speech in specific scenarios.

[0060] Figure 4The distribution diagram of the part-of-speech ratio without and after part-of-speech optimization is shown. No pos represents no part-of-speech optimization, and pos represents after part-of-speech optimization. The accuracy of the part without part-of-speech optimization is 0.51, and the accuracy after part-of-speech optimization is 0.63. The part-of-speech adjustment is an optimization method with higher accuracy obtained by the present invention after black box optimization. Compared with the distribution of the original text, it can be found that the proportion of punctuation marks is significantly reduced after compression. Since the information that the model has learned will be eliminated after the text compression based on self-information, the reduction of this part is in line with expectations. At the same time, the proportion of other parts remains almost unchanged. Therefore, the present invention adjusts the part-of-speech ratio of the compressed text, expands the nouns, proper nouns, verbs and other parts, thereby improving the accuracy of the text by 12 percentage points. It can be seen that the optimization of the text information is very effective.

[0061] Through the black-box optimization process of the research content, the present invention obtains the part-of-speech distribution of compressed texts under different configurations, and can analyze the performance gap of the target model under different part-of-speech distributions; it can be found that when the text acts on the target question-answering model, the text with a higher proportion of nouns, verbs, proper nouns, etc. has better performance, which confirms that the research idea of ​​the present invention is correct.

[0062] Figure 5 An example in the dataset is shown. Origin represents uncompressed text. LLMLingua and Selective Context are the results of compression using their methods. Ours is the solution of the present invention, and the compression rates of these compression methods are all 0.5. It can be seen that in the compression of the paragraph containing the text of the answer to the question, LLMLingua and Selective Context are discarded, while the method of the present invention retains important answer information, such as the number weight of May 18, 2018 is relatively large and is retained after optimization.

[0063] This technology has been extensively tested on long text question answering datasets. In this graduation project experiment, two datasets were mainly selected. Google's Natural Questions dataset is used for multi-document question answering tasks, while the TriviaQA dataset is suitable for single-document question answering tasks. The selection of these two datasets is intended to evaluate and compare the performance of different types of question answering systems. Table 3 shows the relevant information of the two datasets used. Among them, the average token length of the Natural Questions Multi-document QA dataset is longer. At the same time, the present invention uses the multi-document location solution of this dataset.

[0064] Table 3. Dataset brief information

[0065]

[0066] Selective-Context conducts Phrase granularity experiments, and LLMLingua conducts Content granularity experiments. The compression ratio is set to 0.5. The target model adopts the Qwen1.5-7b-chat model, and the evaluation indicator used is EM. Table 4 shows the experimental results.

[0067] Table 4 Experimental results

[0068]

[0069] From the experimental results, it can be seen that the method of the present invention is more outstanding in accuracy in long text compression schemes and improves the performance of text compression.

[0070] In summary, the present invention can further improve the performance of the compression algorithm based on the self-information-based text compression algorithm, making the compression more accurate and effective, and can also reduce the inference cost of the target model for long sequences.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A text compression method based on natural language features and self-information, characterized in that: The specific steps are as follows: Step 1: Dataset acquisition and processing: Preprocess the Google Q&A dataset in the open source corpus, delete useless information, simulate multi-document retrieval, obtain a dataset for long context compression, and then obtain multi-document long text; Step 2: Self-information evaluation and distribution optimization based on natural language features: The phrase granularity and text self-information distribution are obtained by evaluating the self-information of the multi-document long text, and natural language features such as part-of-speech analysis, named entity analysis, and dependency analysis are extracted to optimize the distribution based on natural language features and obtain the compressed text after distribution optimization. Step 3: Downstream task evaluation: Compress the text prompt setting, build the target question-answering model, obtain question-answering performance indicators and apply them to downstream tasks.

2. The text compression method based on natural language features and self-information according to claim 1 is characterized in that: Optimize and filter text information, for text s = {s1, s2, ..., s n }, each text component has an importance expressed as w = {w1, w2, ..., w n }; Among them, w i and i are mutually corresponding, s i is the i-th word or phrase, w i It means s i The importance of w i is a real number; The natural language features corresponding to each text are unified and expressed as θ = {θ1, θ2, ..., θ n }; Among them, θ i Indicates i The natural language characteristics of The importance of each text unit will be adjusted and weighed according to the natural language features. i Further expressed as a mapping function based on natural language features, the importance of each new text unit is obtained, m = {θ1(w1), θ2(w2), ..., θ n (w n )}; Through black box optimization, each part-of-speech feature, each named entity, and each dependency relationship is optimized and defined to obtain a better parameter as θ i The scheme of the function; After obtaining the reordered text units, the percentile-based filtering method is used to adaptively select the part with the largest amount of information, and the content with the largest amount of information is sorted in order to obtain the final compressed text, which is described in detail as follows: First, assume that the obtained text self-information distribution is m=m1,m2,...,m n , retain the first p percentile texts, and calculate m p =np.percentile([m1,m2,...,m n ], 100-p) is the information content of the text of the pth percentile from large to small; Among them, np.percentile(l, p) means to find the value of the pth percentile in the list l from small to large, and this value is obtained through the numpy library of Python; Retain the vocabulary units whose information content is greater than or equal to the pth percentile and construct a compressed new text S′={s1, s2, ..., s t |m t ≥m p }.

Images