Context redundancy optimization method oriented to large language model retrieval enhancement

Abstract

The invention relates to a context redundancy optimization method (CRORALLM) oriented to large language model retrieval enhancement. According to the method, the input context of the large language model is optimized by adopting a strategy of clustering first and then abstracting, so that the redundant influence is reduced, and the content comprehensiveness is ensured while the retrieval accuracy is improved. CRORALLM firstly classifies a candidate document set in order in a clustering mode, so that content themes can be recognized more effectively, and the subsequent abstract generation quality is improved; then, a fine-tuning abstract model is used for analyzing the correlation of the clustering document set according to the problem, and a weighted abstract is generated to serve as a new context: texts with high relevancy with the problem are reserved, texts with low relevancy with the problem are subjected to high-degree abstracting, and texts irrelevant to the problem are filtered. And finally, the contexts are used for large language model retrieval enhancement, so that the problems of information overload and redundant noise are solved, and the model answering accuracy is improved. According to the method, four base models at home and abroad such as Qwen are selected in an experiment, a common data set is adopted, different noise ratios are set for the experiment, and the result shows that the model accuracy of the CRRALLM is obviously improved under the different noise ratios, so that the method can effectively optimize the context in large language retrieval enhancement and improve the accuracy of large language model answering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a retrieval enhancement method for a large language model. BACKGROUND

[0002] Large language models (LLM) have made good progress in recent years, but still face many limitations, such as "hallucinations" when faced with real-time and unknown problems. To address this problem, retrieval-augmented large language models (RALLM) retrieve relevant documents from an external knowledge base based on semantic similarity calculations, thereby improving the accuracy of LLM answers. By introducing an external knowledge base, RALLM can effectively reduce the generation of content that does not conform to the facts, improving the applicability of LLMs in real life.

[0003] Although RALLM effectively improves the accuracy of LLM answers, when faced with knowledge-intensive tasks [1], it still needs to process longer context information, which can lead to the following problems: (1) The model can only effectively capture the information at the beginning and end when processing long texts, while ignoring the middle part of the text, which weakens the model's ability to perceive key information (2) Long texts may contain redundant and noisy information, which can interfere with model reasoning and thus reduce answer accuracy. To address these issues, Zhuang et al. and Ma et al. proposed a retrieval document re-ranking strategy; Jiang et al. proposed a strategy to filter irrelevant information within the document based on the question. These methods can only shorten the length of the retrieval text and cannot guarantee low redundancy while maintaining the comprehensiveness of the retrieval text content. To alleviate this problem, current research can be divided into pre-retrieval optimization and post-retrieval optimization according to the retrieval enhancement method.

[0004] The pre-retrieval optimization research method mainly optimizes the retrieval text:

[0005] Retrieval text optimization. Teja et al. proposed a document chunking strategy that splits the document into a fixed number of token blocks. Larger blocks can capture more context, but also bring more noise, while smaller blocks may not be able to express complete semantics. Langchian proposed a text splitting method that uses recursive splitting and sliding window optimization to merge global relevant information in multiple retrieval processes to achieve hierarchical retrieval. However, these methods still cannot balance between semantic completeness and text length.

[0006] The post-retrieval optimization research method can be divided into two ways: document re-ranking and document context compression:

[0007] Document reordering. Zhuang et al. use a large language model as a query likelihood model to reorder the retrieved documents and select the top-ranked documents. Ma et al. propose a filtering-reordering mode that fully combines the inference speed of a small language model and the inference accuracy of a large language model; a small language model is used as a filter to filter the retrieved information, and a large language model is used to reorder the filtered samples, and finally the top-k documents are selected. Hu et al. perform fine-grained splitting according to the paragraphs in the document, discard irrelevant paragraphs, select top-k paragraphs according to the cosine similarity with the question, and sort the paragraphs according to their positions in the document.

[0008] Document context compression. Xu et al. propose an extractive compressor that uses a double encoder to encode the question and the sentences of the document, selects relevant sentences by calculating the similarity between the two, and converts the unimportant information in the retrieved document into a form that is difficult for humans to understand but can be understood by a large language model. Yang et al. propose a two-stage training method to compress the retrieved document. First, a pre-trained model is used to minimize the difference between the output of the compressed document and the true context through supervised training. Then, reinforcement learning is used to adjust the model parameters of the previous stage according to the reward value of the final answer.

[0009] Although the above methods have achieved good results in knowledge-intensive tasks, their role is only to shorten the length of the retrieved text, and they cannot guarantee the comprehensiveness and diversity of the retrieved content.

[0010] Therefore, an improved retrieval method is proposed to enhance the large language model, which can solve the problem of computational burden and redundancy caused by long text, and the problem of fabrication that may exist in the summary generation process of the large language model, etc., thereby ensuring the comprehensiveness of the retrieved text while reducing its noise and redundancy, effectively improving the accuracy of the large language model in answering questions, and having important practical significance. SUMMARY

[0011] The application provides a context redundancy optimization method CRORALLM for large language model retrieval enhancement, adopts a strategy of clustering first and then summarizing to optimize the input context of the large language model, reduces the influence of redundancy, improves the retrieval accuracy, and ensures the comprehensiveness of the content. CRORALLM first classifies the candidate document set in order through clustering, so as to more effectively identify the content theme and improve the quality of subsequent summary generation; then a fine-tuned summary model is used to analyze the relevance of the clustered document set according to the question, and a weighted summary is generated as a new context: the text with high relevance to the question is retained, the text with low relevance to the question is highly summarized, and the text irrelevant to the question is filtered. Finally, these contexts are used for large language model retrieval enhancement to solve the problems of information overload and redundant noise, and improve the accuracy of model answers.

[0012] In order to achieve the above-mentioned purpose, the application adopts the following technical solutions.

[0013] The retrieval enhancement method for the large language model improves, comprising:

[0014] Step one: according to the question, as many relevant documents as possible are retrieved in the knowledge base, and the candidate documents are represented by vectors for subsequent clustering;

[0015] Step two: use a clustering algorithm to cluster the candidate documents, which can divide longer text information into several shorter and approximate text sets, improve the accuracy of the subsequent summary model, and at the same time, the model can retrieve more information to ensure the comprehensiveness of the retrieval content;

[0016] Step three: use the LoRA parameter fine-tuning method to fine-tune the large language model, train it to generate weighted summaries for candidate documents of different importance, retain or summarize the documents closely related to the question, and filter the documents irrelevant to the question;

[0017] Step four: the generated summary and the question are input into the large language model to generate answers, improving the accuracy of the large language model in answering questions. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a model diagram of the CRORALLM model described in the application;

[0019] Figure 2 is a prompt word fine-tuning example of the method described in the application;

[0020] Figure 3 is a fine-tuning model parameter setting of the method described in the application;

[0021] Figure 4 is the experimental results of the method described in the application on the noise robustness data set;

[0022] Figure 5 are the experimental results of the method of the present application on the information integration dataset;

[0023] Figure 6 are the experimental results of the method of the present application on different numbers of candidate documents;

[0024] Figure 7 is a case analysis of filtering noise information by the method of the present application. DETAILED DESCRIPTION

[0025] The method of the present application will be further described in detail based on the implementation principle and specific data processing flow.

[0026] The first step is to perform vectorization representation on the candidate documents, i.e., document embedding, for subsequent clustering.

[0027] CRORALLM first retrieves as many relevant documents as possible in the knowledge base according to the problem. Then, the candidate documents are represented by embedding as shown in formula 1, wherein the embedding model can be Word2Vec, BERT, etc.

[0028] index doc =Embedding(document) (1)

[0029] wherein index doc represents the embedding vector of the document, Embedding represents the embedding model, and document represents the candidate document set.

[0030] After completing the vectorization representation of the candidate documents, a clustering algorithm is used to cluster the candidate documents.

[0031] The second step is to use a clustering algorithm to cluster the candidate documents, and divide the longer text information into several shorter and approximate text sets.

[0032] After clustering the candidate documents using the k-means clustering algorithm, n clusters can be obtained as shown in formula 2:

[0033] C1,C2,…,C n =Kmeans(index1,index2,…,index k ) (2)

[0034] wherein C represents the clustered document cluster, and index is the document embedding vector in the first step of document embedding.

[0035] The third step is to optimize the summary model.

[0036] To reduce the noise generated by large language models during the summary generation process, the GLM-4-Flash model is fine-tuned using LoRA parameter fine-tuning as shown in equation 3, ensuring that the summary information is concise and accurate.

[0037] M' = LoRA(M) (3)

[0038] where M represents the GLM-4-Flash base model, and M' represents the model after LoRA fine-tuning. LoRA aims to improve the efficiency of fine-tuning large language models by freezing the original parameter matrix and injecting a trainable low-rank decomposition matrix into the original parameter matrix that needs to be adjusted, as shown in equation 4:

[0039] W frozen = W frozen + αBA (4)

[0040] where W frozen represents the frozen original parameter matrix, α represents the weight factor of lora, A is a reduced rank matrix usually initialized with Gaussian, and B is an increased rank matrix usually initialized with all 0. The purpose of LoRA fine-tuning is to enable large language models to generate weighted summaries for candidate documents of different importance.

[0041] A prompt word case for fine-tuning and summarization is shown in the following figure. After fine-tuning with this prompt, the summary model can generate weighted summaries for candidate documents of different importance, retaining or summarizing documents closely related to the question, and filtering out documents unrelated to the question.

[0042] Step 4: Generate summaries using the fine-tuned summary model.

[0044] To improve the accuracy of generated summaries, a set of prompt words for the summary model is designed as shown in equation 5. For the n clusters of candidate documents after clustering in the second step, the model trained in the third step is used to generate text summaries as shown in equation 6.

[0045] input = prompt + C i (5)

[0046] A i = M'(input) (6)

[0047] where prompt is the designed prompt word, C i represents the documents contained in the i-th cluster, and A i represents the summary information generated by the i-th cluster. If the documents in C i are not related to the question, then A i is empty.

[0048] Step 5: Answer generation.

[0049] Finally, all the generated summary information is spliced together with the question as input to the large language model to generate an answer, as shown in formulas 7 and 8:

[0050]

[0051] answer = LLM(context) (8)

[0052] The generated summary is input to the large language model together with the question to generate an answer, which can improve the accuracy of the large language model in answering questions.

[0053] Step 6: Use two datasets to measure the effectiveness of the method from different angles.

[0054] There are references in the field of retrieval augmented generation benchmark for large language models, which include four capabilities of the model: noise robustness, negative rejection, information integration, and counterfactual robustness. The focus of the invention is mainly on the redundancy and noise of the candidate documents, so the noise robustness and information integration datasets proposed by it are used to verify the method proposed in the invention.

[0055] Noise robustness dataset. Noise robustness is to detect the robustness of the large language model in noisy documents. Since the retrieved documents are sometimes not completely accurate, the external knowledge obtained often has high redundancy and noise. This dataset simulates a noisy and redundant environment by adding noisy documents to the candidate documents, and users can adjust the proportion of noisy documents according to the noise ratio.

[0056] Information integration dataset. Information integration is to test the ability of the model to integrate information from multiple documents to organize answers. This dataset contains many complex questions, i.e. the answer to a question may be contained in multiple documents. At the same time, this dataset also contains some noisy documents, and users can adjust the proportion of noisy documents according to the noise ratio.

[0057] The invention selects large language models Chatglm-6B, Qwen-7B-chat, Vicuna-7B-v1.3, and Llama-7B as experimental models for comparison. The above four models have similar parameters and can represent the current large language models at home and abroad. At the same time, the invention uses accuracy to measure the performance of the model on the noise robustness dataset and the information integration dataset. The dataset provides multiple candidate correct answers, and we use the exact matching method. If any of the candidate correct answers appears in the final model's answer, the model is considered correct.

[0058] The parameter settings of the GLM-4-Flash and the parameter settings of the answer using the large language model are fine-tuned in the manner of LoRA parameter fine-tuning, as shown in the table:

[0059] Table 1 model parameter settings

[0060]

[0061] In the noise robustness data set, different noise ratios in the candidate documents are set to evaluate the accuracy, and the results are shown in Table 2. The experimental results show that: (1) The improved retrieval method proposed in the application can improve the accuracy of the large language model under different noise. The accuracy improvement effect is particularly significant when the noise ratio is 0.6 and 0.8, which shows that our method can effectively compress and save relevant documents for noise documents and redundant information. (2) The accuracy improvement effect of Chatglm and Vicuna is better than that of Qwen and Llama, which shows that our method is more effective when the initial accuracy of the model is low. (3) When the noise ratio is 1, the accuracy decreases significantly, and accurate external knowledge documents play a crucial role in enhancing the large language model.

[0062] Table 2 experimental results on noise robustness data set

[0063]

[0064] In the information integration data set, different noise ratios in the candidate documents are also set to evaluate the accuracy, and the results are shown in Table 3. Compared with the performance of the model in the noise robustness data set, it can be seen that the large language model has weak information integration ability. The experimental results show that: (1) The method proposed in the application can improve the accuracy of the model under different noise ratios, and the improvement effect has no obvious rule with the noise ratio. We speculate that this is because the problems in the information integration data set require the combination of multiple document information, and these information are mutually exclusive and can be regarded as each other's noise, making it more difficult for the model to extract effective information. (2) The accuracy improvement effect of Chatglm and Llama is better than that of Qwen and Vicuna, and the experimental results are significant when the initial accuracy of the model is low.

[0065] Table 3 experimental results on information integration data set

[0066]

[0067] To verify the effect of the method of the application in processing redundant information, we select Chatglm with better experimental promotion effect to perform experiments on different numbers of candidate documents, and set the noise ratio to 0, and the experimental results are shown in Table 4. It can be seen that when the number of documents is 6, it reaches the highest, and thereafter, with the increase of the number of documents, the accuracy of the Chatglm model presents a downward trend, which shows that the redundancy of the document is not conducive to the model to extract useful information; and the accuracy of our method remains stable and presents a growth trend when the number of documents increases, which shows that our method can reduce the redundancy of the candidate documents and retain relevant information.

[0068] Table 4 Experimental results on different numbers of candidate documents

[0069]

[0070] In some cases, when the noise ratio is high, most of the information in the candidate documents is noise information, and the model is difficult to extract useful information from these documents; and our method can filter noise information to reduce the interference of noise on the performance of the model. For example, the case shown in Table 5, our method can accurately filter irrelevant information and retain relevant information to the question, so the method can make the model generate accurate answers.

[0071] Table 5 Case analysis

[0072]

[0073]

[0074] The above is only the preferred specific embodiment of the application, but the protection scope is not limited thereto, and any changes or replacements easily thought of by those skilled in the art within the technical range disclosed by the application should be covered in the protection scope of the application. Therefore, the protection scope should be subject to the content of the claims.

Claims

1. A context redundancy optimization method for large language model retrieval enhancement, characterized in that, Comprise: (1) As much as possible to retrieve relevant documents in the knowledge base according to the problem, and vectorize the candidate documents; (2) Then use clustering algorithm to cluster the candidate documents, which can divide the longer text information into several shorter and approximate text sets, improve the accuracy of the subsequent summary model, and the model can retrieve more information to ensure the comprehensiveness of the retrieved content; (3) Optimize the summary model: fine-tune the large language model using LoRA parameter fine-tuning method, train it to generate weighted summary for candidate documents with different importance, retain or summarize the documents closely related to the problem, and filter the documents irrelevant to the problem; (4) Finally, input the generated summary and the question into the large language model to generate answers, improve the accuracy of the large language model in answering questions; (5) Adopt the data sets in terms of noise robustness and information integration, and set different noise ratios for experiments to measure the effectiveness of the method in solving the redundancy and noise of the retrieved text.

2. The method of claim 1, wherein, Vectorize the candidate documents, first retrieve as many relevant documents as possible in the knowledge base according to the problem by CRORALLM. Then embed the candidate documents as shown in formula 1, where the embedding model can be Word2Vec, BERT, etc.: index doc = Embedding(document) # (1) where index doc denotes the embedding vector of the document, Embedding denotes the embedding model, and document denotes the set of candidate documents.

3. The method of claim 1, wherein, The method of clustering the candidate documents using clustering algorithm is as follows: after clustering the candidate documents using k-means clustering algorithm, n clusters can be obtained as shown in formula 2: C1, C2,..., C n = Kmeans(index1, index2,..., index k ) # (2) Where C represents the clustered document cluster, and index is the document embedding vector in the first step of document embedding.

4. The method of claim 1, wherein, To reduce the noise that may be generated in the process of generating summary by large language model, optimize the summary model: (1) Set the fine-tuning data set, refer to the experimental configuration of LoRA parameter fine-tuning by Malladi et al. and Xia et al., and randomly sample 1000 from the three public data sets of Hotpotqa, TriviaQA and Natural Question (NQ) in the ratio of 10:1:

1. Extract the question, paragraph and summary as the fine-tuning data set; (2) The purpose of LoRA fine-tuning is to enable the large language model to generate weighted summary for candidate documents with different importance.

5. The method according to claim 1 or 4, characterized in that, Fine-tune the GLM-4-Flash model using LoRA parameter fine-tuning method as shown in formula 3: M' = LoRA(M) #(3) Where M represents the GLM-4-Flash base model, M' represents the model after LoRA fine-tuning, and LoRA aims to improve the efficiency of fine-tuning large language model. Its method is to freeze the original parameter matrix and inject a trainable low-rank decomposition matrix into the original parameter matrix that needs to be adjusted as shown in formula 4: W frozen = W frozen + αBA#(4) where W frozen represents the frozen original parameter matrix, a represents the weight factor of lora, A is a descending rank matrix, usually initialized by Gaussian, B is an ascending rank matrix, usually initialized by all 0, and the purpose of LoRA fine-tuning is to enable large language models to generate weighted summaries for candidate documents of different importance.

6. The method of claim 1, wherein, In order to improve the accuracy of the generated summary, a set of prompt words for the summary model is designed as shown in formula 5 when generating summary after clustering; the trained summary model is used to generate text summary for the n clusters of candidate documents after clustering as shown in formula 6: input = prompt + C i #(5) A i = M'(input) # (6) where prompt is a designed prompt word, C i represents the document contained in the i-th cluster, A i represents the summary information generated by the i-th cluster, if the document in C i is irrelevant to the question, then A i is empty.

7. The method of claim 1, wherein, Finally, all the generated summary information is spliced and input into the large language model together with the question to generate answers as shown in formulas 7 and 8: answer = LLM(context) # (8) The accuracy of large language models answering questions can be improved.

8. The method of claim 1, wherein, Experiments were conducted using datasets in terms of noise robustness and information integration, with different noise ratios to measure the effectiveness of the proposed method in addressing retrieval text redundancy and noise: (1) This paper selects large language models Chatglm-6B, Qwen-7B-chat, Vicuna-7B-v1.3, and Llama-7B for comparison. The four models have similar parameter quantities and can represent the current large language models at home and abroad; (2) This paper uses accuracy to measure the performance of the model on the noise robustness dataset and the information integration dataset. The dataset provides multiple candidate correct answers. We use the exact matching method. If any of the candidate correct answers appears in the final model's answer, the model's answer is considered correct; (3) Conclusion: Generating summaries after clustering can ensure the comprehensiveness of the retrieved document information while improving the accuracy of the summary model. Weighted summaries filter noise in documents while retaining relevant information, improving the accuracy of model answers.

Images