Evaluation method for generating large model through long document retrieval enhancement

Abstract

The invention provides an evaluation method of a long document retrieval enhancement generation large model, and relates to the technical field of data processing, the method comprises the following steps: constructing an automatic question answering strategy based on a focus segment; constructing an evaluation data set, and processing the evaluation data set based on an automatic problem strategy to obtain an optimal evaluation data set for generating a large model by aiming at long document retrieval enhancement; and performing evaluation on the basis of an optimal evaluation data set for generating the large model by aiming at long document retrieval enhancement. According to the method, the problems that a long document question and answer data set is difficult to construct and an evaluation method aiming at a long document retrieval enhancement generation large model system is lacked are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to an evaluation method for enhancing the generation of a large model for long document retrieval. Background Art

[0002] Large language models (LLMs), such as Llama, Claude2, and GPT4, have performed well in numerous NLP tasks. However, certain shortcomings still exist. The phenomenon of hallucination in LLM refers to situations where the content generated by the model is unrealistic, inaccurate, or meaningless. Researchers have made efforts to address this issue. Retrieval-augmented generation (RAG) technology integrates external knowledge bases to retrieve relevant context, which helps large language models (LLMs) to generate more reliable and accurate responses. The application of retrieval-augmented generation (RAG) effectively alleviates the hallucination problem in large language models (LLMs).

[0003] Comprehensive and objective evaluation of retrieval-enhanced generation large model systems can promote their improvement. At present, researchers have proposed several evaluation methods. For example, RGB analyzed the performance of different large language models in the four basic capabilities required for retrieval-enhanced generation RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. MultiHop-RAG focuses specifically on multi-hop queries, retrieval from multiple documents, and reasoning evidence. However, the external knowledge bases involved in these evaluation methods consist of relatively short documents.

[0004] In the real world, external knowledge bases are usually in the form of long documents. When the external knowledge base consists of long documents, the retrieval-enhanced generation of RAG task becomes more challenging. However, existing retrieval-enhanced generation of RAG evaluation methods mainly target short documents and do not give enough consideration to long documents. One of the important reasons is that it is difficult to construct a dedicated long document retrieval-enhanced generation of RAG evaluation dataset. The traditional question-and-answer dataset construction strategy requires manual annotation of questions and their corresponding answers, which has a high annotation cost. An alternative strategy for constructing a question-and-answer dataset based on LLM is to use LLM to automatically generate questions and their corresponding answers, replacing manual annotators. However, when the external knowledge base consists of long documents, it is difficult for LLM to accurately generate questions and answers. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an evaluation method for generating a large model for long document retrieval enhancement, which solves the problem of lack of long document evaluation methods in the current retrieval enhancement field.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is: an evaluation method for enhancing the generation of a large model for long document retrieval, comprising the following steps:

[0007] S1. Build an automatic question-answering strategy based on focus segments;

[0008] S2. Construct an evaluation dataset and process the evaluation dataset based on the automatic question strategy to obtain the optimal evaluation dataset for enhancing the generation of a large model for long document retrieval;

[0009] S3. Evaluation is performed based on the optimal evaluation dataset for enhancing the generation of large models for long document retrieval.

[0010] The beneficial effects of the present invention are as follows: the present invention uses the bilingual dataset LoRAG evaluation method to evaluate the existing large-scale language models LLMs, and analyzes the main challenges faced by retrieval-enhanced generation of RAGs on long documents. Experimental results show that the main challenge for retrieval-enhanced generation of RAGs on long documents in the future is to balance the relevance and refinement of model responses. The LoRAG evaluation method of the present invention evaluates the existing large-scale language models LLMs, and analyzes the main challenges faced by retrieval-enhanced generation of RAGs on long documents in the future: namely, balancing the relevance and refinement of model responses.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S101, obtaining each long document in the knowledge base, and randomly extracting n segments, wherein each segment is a focus segment;

[0013] S102, setting prompts to guide large language models (LLMs) to generate qualified question-answer pairs;

[0014] S103, combining the focus segment and the prompt, and inputting them into the large language model LLMs to generate context-related question-answer pairs:

[0015] (Q^,A)=ChatGPT(d,p)

[0016] d=D[start:start+N d ]

[0017] Among them, Q^ represents the generated question, A represents the corresponding answer to the generated question, ChatGPT() large model generation process, d represents the focus segment, p represents the set prompt, D represents the long document, N d Indicates the length of the segment, and start indicates the starting position of the segment;

[0018] S104. Use the following formula to filter the generated context-related question and answer pairs to obtain a result data set that meets the requirements, where the result data set includes question and answer instances (Qi, Ai, di, D):

[0019] (Q,A)=Human(Q,A^)

[0020] Among them, (Q,A) represents the question-answering instances that meet the requirements after screening, and Human() represents the manual screening process;

[0021] S105, based on the result data set, use the document to evaluate the large language model LLMs and generate the answer A'i;

[0022] S106: Compare the answer A'i with the reference answer Ai to obtain a comparison result, thereby completing the construction of the automatic question-answering strategy.

[0023] The beneficial effect of the above further solution is that the present invention proposes an automatic question-answering construction strategy based on focus segments, which effectively solves the challenges of high annotation cost and inaccuracy faced by building question-answering datasets on long documents, and can efficiently and accurately build question-answering datasets for long documents. .

[0024] Furthermore, the expression of the comparison result is as follows:

[0025] Score=Evaluation(A'i,Ai,Qi,di)

[0026] A'i=LLMs(D,Qi)

[0027] Dataset = {(Qi, Ai, di, D)}

[0028] Among them, Score represents the evaluation score, Evaluation() represents the evaluation process, LLMs() represents the process of generating the large language model to be tested, Ai represents the reference answer i, Qi represents the question i, di represents the focus segment corresponding to Ai and Qi, and Dataset() represents the final generated data style.

[0029] The beneficial effect of the above further solution is that the present invention supplements the evaluation process on the basis of the automatic question-answering construction strategy. This process makes a comprehensive and objective evaluation of the performance of the baseline model on the long document question-answering task through a variety of evaluation methods, which enables researchers to better understand the shortcomings of the current long document retrieval enhancement generation task and optimize it on this basis.

[0030] Furthermore, step S2 includes the following steps:

[0031] S201, in data selection, collect Chinese and English datasets from long documents, and merge the collected datasets into the bilingual dataset LoRA;

[0032] S202, extracting focus segments from the long document, and inputting the extracted focus segments and the generated qualified question-answer pair prompts into the large language model LLMs, extracting text by generating qualified questions and their corresponding answers;

[0033] S203: Screen the extracted text to obtain the optimal evaluation data set for enhancing the generation of a large model for long document retrieval.

[0034] The beneficial effect of the above further scheme is: Based on the automatic question-answering construction strategy of the focus fragment, the present invention proposes a bilingual dataset LoRAG, an evaluation method focusing on the large model system of long document retrieval enhancement generation. The present invention is the first to tailor a retrieval enhancement generation RAG evaluation method specifically for long documents. The evaluation method of the present invention solves the contradiction between the growing demand for long document question-answering in real life and the lack of corresponding long document retrieval enhancement generation RAG evaluation method, and can effectively promote the development of retrieval enhancement generation RAG.

[0035] Furthermore, step S3 includes the following steps:

[0036] S301, obtaining the baseline model and evaluation indicators to be evaluated;

[0037] S302, based on the optimal evaluation data set for the enhanced generation of a large model for long document retrieval, using the baseline model to generate corresponding responses in the evaluation data;

[0038] S303. Using the obtained evaluation indicators, the corresponding responses in the generated evaluation data are compared with the reference answers to obtain evaluation scores, thereby completing the evaluation of the long document retrieval enhanced generation large model.

[0039] The beneficial effect of the above further scheme is that the present invention adds an evaluation link based on the automatic question-answering construction strategy. Through a variety of evaluation methods, the performance of the baseline model in the long document question-answering task is comprehensively and objectively evaluated, which helps researchers to deeply understand the shortcomings of the current long document retrieval enhancement generation task, and provides a basis for subsequent optimization, promoting the development of the retrieval enhancement generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flow chart of the method.

[0041] Figure 2 Schematic diagram of the evaluation results based on BLEU.

[0042] Figure 3 A schematic diagram of the evaluation results of different models using BERTScore.

[0043] Figure 4 Schematic diagram of the evaluation results of large language models (LLMs).

[0044] Figure 5 Schematic diagram of human evaluation results.

[0045] Figure 6 Answer length diagrams for different models.

[0046] Figure 7 Schematic diagram of examples of responses given by different baseline models in the financial field. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0048] Example

[0049] Before describing the present invention, the following abbreviations and key terms are defined:

[0050] LLMs: Large Language Models

[0051] NLP: Natural Language Processing

[0052] RAG: Retrieval Enhanced Generation

[0053] Token: In natural language processing, a token refers to the basic unit in a text, which can be a word, symbol, or character.

[0054] RGB: A benchmark dataset for evaluating the performance of large language models in terms of the four basic capabilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness.

[0055] MultiHop-RAG: The RAG benchmark dataset specifically focuses on multi-hop queries, retrieval and reasoning about evidence from multiple documents.

[0056] LoRAG: This paper proposes a RAG benchmark dataset for long documents, which is a bilingual dataset covering literature, law, finance and other aspects.

[0057] ChatGPT: A powerful large language model from OpenAI, also known as GPT-3.5-turbo.

[0058] GPT4: A powerful large model launched by OpenAI, which is an iterative update of ChatGPT.

[0059] Llama: An open source large language model launched by MetaAI.

[0060] Qwen: The open source large language model launched by Alibaba.

[0061] Vicuna: An open-source large language model developed by fine-tuning LLaMA using conversations collected from ShareGPT.

[0062] Claude2: Anthropic AI has launched a powerful commercial large model, which is comparable to ChatGPT in performance.

[0063] API: Application programming interface, which enables developers to take advantage of existing functionality or services.

[0064] Mediastack: is a real-time news API that provides news articles from major news sources around the world.

[0065] BLEU: BLEU (Bilingual Evaluation Understudy) is an automated metric for evaluating the quality of machine translation. It measures the accuracy of a translation by comparing the similarity between the generated translation and one or more reference translations.

[0066] BERT model: BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that uses a bidirectional encoder to improve the performance of natural language processing tasks by simultaneously considering the left and right information of the context.

[0067] BERTScore: BERTScore is a metric for evaluating the quality of text generation. It calculates the similarity between generated text and reference text based on the pre-trained BERT model. By comparing the contextual representation of vocabulary, BERTScore can better capture semantic similarity and provide more accurate evaluation than traditional methods.

[0068] Based on the background technology, this paper proposes an innovative automatic question-answering construction strategy, which randomly extracts a segment from the original long document, called the focus segment, and then uses GPT3.5-turbo (a powerful large language model) to generate questions and reference answers based on the focus segment. Testing the LLM (large language model) requires providing an answer to the question based on the entire document. This strategy effectively overcomes the high annotation cost of traditional strategies and is more accurate than LLM-based strategies.

[0069] According to the question-answer construction strategy based on focus fragments, the present invention proposes an evaluation method specifically for long document retrieval enhancement generation of large model systems, called LoRAG. The data in this evaluation method covers multiple fields such as law, finance, and literature, with an average length of 57k tokens (English) and 76k tokens (Chinese). Specifically, LoRAG collects long documents in the above-mentioned fields as an external knowledge base. Then, using the question-answer construction strategy based on focus fragments, the present invention automatically constructs six sub-datasets for each language (English, Chinese) in three fields (literature, law, and finance). In addition, the present invention manually selects high-quality question-answer pairs to form two core sub-datasets in English and Chinese. Finally, the evaluation data set in LoRAG contains eight sub-datasets. Finally, a comprehensive and objective evaluation of different retrieval enhancement generation of large model systems is carried out through a variety of evaluation indicators such as traditional evaluation indicators, large model evaluation, and expert evaluation.

[0070] like Figure 1 As shown, the present invention provides an evaluation method for enhancing the generation of a large model for long document retrieval, and the implementation method is as follows:

[0071] S1. Build an automatic question-answering strategy based on focus segments. The implementation method is as follows:

[0072] S101, obtaining each long document in the knowledge base, and randomly extracting n segments, wherein each segment is a focus segment;

[0073] S102, setting prompts to guide large language models (LLMs) to generate qualified question-answer pairs;

[0074] S103, combining the focus segment and the prompt, and inputting them into the large language model LLMs to generate context-related question-answer pairs;

[0075] S104, screening the generated context-related question-answer pairs to obtain a result data set that meets the requirements, wherein the result data set includes question-answer instances (Qi, Ai, di, D);

[0076] S105, based on the result data set, use the document to evaluate the large language model LLMs and generate the answer A'i;

[0077] S106: Compare the answer A'i with the reference answer Ai to obtain a comparison result, thereby completing the construction of the automatic question-answering strategy.

[0078] In this embodiment, in order to improve the efficiency and accuracy of constructing evaluation data, the present invention proposes an automatic question-answering construction strategy based on focus segments. The strategy includes three main steps. First, for each long document in the knowledge base, n segments are randomly extracted, and each segment is called a focus segment. Next, a personalized prompt is designed to guide the large language model LLMs to generate qualified question-answer pairs. Finally, the focus segments and prompts are combined and input into the large language model LLMs (GPT-3.5-turbo is used in our experiments) to generate more accurate and context-related question-answer pairs.

[0079] Specifically, we first extract a set of N d There are d marked focus segments. The starting point of the focus segment is in the range 0 to len(D)-N d Internal random sampling:

[0080] d=D[start:start+N d ]

[0081] Among them, start~Sample(0,len(D)-N d ).

[0082] When N d When is large enough (here it is set to 2000), the focus segment d contains enough contextual information to understand the pronoun. In addition, the starting point is only selected from the beginning of the sentence, ensuring that the focus segment d is semantically coherent.

[0083] Next, ChatGPT is used to generate a series of question-answer pairs based on the focus segment d and the designed prompt p:

[0084] (Q^,A)=ChatGPT(d,p)

[0085] Among them, Q^ represents the generated question, A represents the corresponding answer to the generated question, ChatGPT() represents the large model generation process, d represents the focus segment, p represents the set prompt, D represents the long document, and N d Indicates the length of the segment, and start indicates the starting position of the segment.

[0086] In order to obtain high-quality and highly relevant question-answering examples, we conduct a manual screening process, where human evaluators select examples that meet the requirements:

[0087] (Q,A)=Human(Q,A^)

[0088] The resulting dataset contains question-answer instances (Qi, Ai, di, D):

[0089] Dataset = {(Qi, Ai, di, D)}

[0090] Based on this dataset, the large language model LLMs is then evaluated with the entire document D. For each question Qi in the dataset, the large language model LLMs generates the answer A'i:

[0091] A'i=LLMs(D,Qi)

[0092] The generated answer A'i is compared with the reference answer Ai using the evaluation metric:

[0093] Score=Evaluation(A'i,Ai,Qi,di)

[0094] Among them, Score represents the evaluation score, Evaluation() represents the evaluation process, LLMs() represents the process of generating the large language model to be tested, Ai represents the reference answer i, Qi represents the question i, di represents the focus segment corresponding to Ai and Qi, and Dataset() represents the final generated data style.

[0095] Evaluation can involve various methods, including automated metrics such as BLEU and BERTScore, LLMs judgment, or human review.

[0096] In this example, the strategy overcomes the inefficiency and high cost of traditional manual question-answering construction strategies. In addition, by using focus segments, it effectively addresses the contextual challenges in long document processing, ensuring high accuracy and relevance of large language models (LLMs) when processing such documents. This approach is particularly suitable for building large-scale, high-quality question-answering datasets.

[0097] S2. Construct an evaluation dataset and process the evaluation dataset based on the automatic question strategy to obtain the optimal evaluation dataset for enhancing the generation of a large model for long document retrieval. The implementation method is as follows:

[0098] S201, in data selection, collect Chinese and English datasets from long documents, and merge the collected datasets into the bilingual dataset LoRA;

[0099] S202, extracting focus segments from the long document, and inputting the extracted focus segments and the generated qualified question-answer pair prompts into the large language model LLMs, extracting text by generating qualified questions and their corresponding answers;

[0100] S203: Screen the extracted text to obtain the optimal evaluation data set for enhancing the generation of a large model for long document retrieval.

[0101] In this embodiment, data selection: In the process of selecting data, the present invention mainly considers the following factors:

[0102] (1) Language: In order to obtain a more comprehensive dataset, English and Chinese are selected as representatives for data collection. The integration of multilingual data will enhance the universality and authority of the benchmark. (2) Document length: Focus on collecting long documents to better reflect external knowledge in real-world scenarios. (3) Domain: By integrating documents in the fields of law, finance, and literature into LoRAG, we can provide the dataset with richer and more vivid language data and improve its usability.

[0103] In this embodiment, data collection: judicial documents in English and Chinese are collected from the International Court of Justice and the China Judgments Online, financial documents in English and Chinese are collected from StockLight (a public English financial report website) and the Shanghai Stock Exchange, and literary documents in English and Chinese are collected from Zlibrary (an online e-book website) as the knowledge source of the benchmark data set. Throughout the data collection process, the reliability and authority of the data source are analyzed and evaluated, the accuracy and completeness of the data are verified, and any potential errors are identified and corrected. Special attention is paid to ensuring access to data from publicly available data sources, and the present invention filters out data that is too short and of poor quality. In the end, a total of 579 documents were obtained. The average length of the document source ranges from 19k tokens to 197k tokens.

[0104] In this embodiment, data processing: the collected documents are processed according to the automatic question-answering construction strategy based on focus fragments. First, focus fragments of a certain length are extracted from the document. These focus fragments are input into GPT-3.5-turbo together with the pre-designed prompt template. Then, qualified questions and their corresponding answers are generated using GPT-3.5-turbo. Finally, the generated text is extracted to obtain a QA instance with the structure of (Qi, Ai, di, D). The LoRAG evaluation data has a total of 9.5k unique QA instances.

[0105] In this embodiment, core data selection: In order to obtain more fine-grained and high-quality evaluation data, a manual screening process was performed. From the generated data instances, QA pairs with appropriate questions and accurate answers were selected to form a refined core QA dataset. Specifically, high-quality QA pairs were selected from 3 sub-datasets in each language to create additional monolingual core datasets. The Core-en dataset (English core dataset) contains 200 QA pairs, and the Core-cn dataset (Chinese core dataset) contains 263 QA pairs. Finally, a RAG evaluation dataset consisting of 8 sub-datasets was obtained.

[0106] S3. Based on the optimal evaluation data set for long document retrieval enhancement generation of large models, the evaluation is carried out. The implementation method is as follows:

[0107] S301, obtaining the baseline model and evaluation indicators to be evaluated;

[0108] S302, based on the optimal evaluation data set for the enhanced generation of a large model for long document retrieval, using the baseline model to generate corresponding responses in the evaluation data;

[0109] S303. Using the obtained evaluation indicators, the corresponding responses in the generated evaluation data are compared with the reference answers to obtain evaluation scores, thereby completing the evaluation of the long document retrieval enhanced generation large model.

[0110] In this example, several latest LLMs are selected as baseline models, and their performance on the evaluation dataset is evaluated using a variety of evaluation metrics.

[0111] ①Baseline model:

[0112] Vicuna: An open-source large language model developed by fine-tuning LLaMA using user shared conversations collected from ShareGPT.

[0113] ChatGLM2: An open source large language model proposed by Tsinghua University Zhipu Qingyan.

[0114] LLlama2: An open source large language model proposed by MetaAI.

[0115] Claude2: A powerful commercial large language model proposed by Anthropic AI.

[0116] ②Evaluation indicators:

[0117] The baseline models are comprehensively evaluated, including traditional evaluation, LLMs evaluation, and human evaluation.

[0118] Traditional evaluation: The traditional method is adopted to evaluate by evaluating the BLEU indicator. In addition, in order to avoid the limitation of focusing only on character matching, BERTScore is also combined. BERTScore calculates the similarity between the reference text and the generated text by considering the context embedding, thereby capturing not only the lexical overlap but also the semantic similarity of the text.

[0119] LLMs evaluation: The advantages of the above evaluation are its speed and objectivity, but there are also some disadvantages, such as imprecise semantic understanding. Therefore, considering the powerful ability of LLMs in natural language understanding, LLMs are introduced to evaluate the responses generated by different baseline models. Specifically, GPT-3.5-turbo is used as the evaluation model. For a given question, the task of the evaluation model is to comprehensively consider factors such as fluency, grammatical correctness, and relevance, and assign appropriate scores to answers generated by different models. The scores range from 1 to 5, representing very low, low, neutral, high, and very high, respectively.

[0120] Manual evaluation: Experts in related fields are invited to evaluate the responses of different models. Specifically, indicators are selected from three perspectives: fluency, conciseness, and relevance. Fluency refers to the ability to serve as a language model, conciseness refers to the lack of redundant information in the response, and relevance refers to the degree of consistency between the response and the reference answer. As with the LLMs evaluation, the score for each aspect also ranges from 1 to 5, representing very low, low, neutral, high, and extremely high, respectively. It is worth noting that the evaluation is conducted in a blind review, and the experts are unaware of the identity of the model.

[0121] This embodiment is, the evaluation results and analysis are as follows:

[0122] ①Evaluation results:

[0123] We first evaluate multiple baseline models using traditional metrics (i.e., BLEU and BERTScore). Figure 2 and Figure 3 As shown in Figure 2, the scores between the models under traditional indicators do not show any significant patterns. Therefore, both are not ideal evaluation indicators.

[0124] The evaluation results of LLMs are as follows: Figure 4 As shown in the figure, Claude2 outperforms other baseline models on all 8 sub-datasets. Llama2 ranks second overall. However, Llama2 performs slightly worse on the Chinese dataset, which may be related to the relatively limited Chinese corpus used in the pre-training process of Llama2. Vicuna13B and ChatGLM2-6B perform poorly, with the highest scores of only 2.95 and 2.58.

[0125] For human evaluation, two annotators were hired to score the responses of different baseline models. Figure 5 All baseline models achieved high levels of fluency, further demonstrating the robustness of LLMs. In terms of relevance, Claude2 performed best, scoring 4.36 and 4.35 on the English and Chinese core datasets, respectively, while Vicuna performed the worst. This may be related to the length of the model response.

[0126] ②Result analysis:

[0127] The number of tokens answered by different baseline models on the Chinese and English evaluation datasets was calculated, and the average value was obtained. Figure 6 As shown in Figure 2, Claude2 has the largest average number of tokens. Correspondingly, Vicuna has the least number of response tokens. In addition, the length of the reference answers in each sub-dataset and the length of the responses of different baseline models were measured and the average value was calculated.

[0128] In summary, it can be inferred that longer responses tend to contain more information, resulting in higher relevance scores for the corresponding models. However, this phenomenon may lead to an increase in redundant text in the responses, which in turn introduces irrelevant information. Therefore, a model with a higher relevance score may also exhibit a lower refinement score. In the RAG task of long documents, balancing the relevance and refinement of model responses may be a key focus of future research.

[0129] In this embodiment, the case is shown as follows:

[0130] For a question-answer pair in the dataset, the RAG system is used to test different baseline models to observe whether the response given by the model is correct. Figure 7 An example from the finance domain is shown. The question contains two sub-questions: "What is the total number of common shares repurchased by the company in 2021?" and "What is the cost?" ChatGLM2, Llama2, and Claude2 can answer both sub-questions, showing good comprehensiveness. In contrast, Vicuna's answer is "It is impossible to answer this question based on the known information", which is significantly inferior to other benchmark models. In terms of the content of the answer, Claude2 accurately provides two correct answers, "4.4 million shares" and "US$1 billion", and performs best in terms of relevance. Llama2 only correctly answers "4.4 million shares", while ChatGLM2 does not answer correctly at all. The answer generated by Claude2 is closer to the reference answer in terms of comprehensiveness and accuracy. Claude2 cites the original context, while Llama2 makes inferences and summaries based on specific details and data. It is worth noting that financial documents usually contain a large number of numbers, which poses a challenge for LLM to generate accurate target numbers.

[0131] The problems to be solved by the present invention are as follows:

[0132] 1. Lack of evaluation methods for long document retrieval enhancement and large model generation systems

[0133] The hallucination phenomenon in large language models (LLMs) refers to situations where the model generates untrue, inaccurate or meaningless content. Retrieval-enhanced generation (RAG) technology helps large language models (LLMs) generate more reliable and accurate responses by integrating external knowledge bases to retrieve relevant contexts. The application of retrieval-enhanced generation (RAG) effectively alleviates the hallucination problem in large language models (LLMs). A comprehensive and objective evaluation of retrieval-enhanced generation of large model systems can promote their improvement. To this end, multiple evaluation methods have been proposed. However, the external knowledge bases involved in these evaluation methods mainly consist of relatively short documents, while in actual scenarios, the external knowledge bases are usually long documents. When the external knowledge base is a long document, the retrieval-enhanced generation of RAG task becomes significantly more complicated, and the current evaluation methods cannot meet this requirement. In order to bridge this gap, the present invention proposes a focus-based automatic data construction strategy, and based on this strategy, proposes an evaluation method for long document retrieval-enhanced generation of large model systems.

[0134] 2. Long document question answering datasets are difficult to build

[0135] Existing evaluation methods for retrieval enhancement generation of large model systems mainly target short documents and lack sufficient consideration for long documents. The main reason for this problem is that it is very difficult to construct specialized evaluation data for long documents. The traditional question-answering evaluation data construction strategy requires manual annotation of questions and their corresponding answers, which leads to considerable annotation costs. An alternative strategy is to use large language models (LLMs) to automatically generate questions and their answers to replace manual annotators. However, when the external knowledge base consists of long documents, large language models (LLMs) are difficult to generate accurate question-answer pairs. To this end, the present invention proposes an innovative automatic question-answering construction strategy based on focus fragments. Specifically, the present invention randomly extracts a fragment from the original long document, called the focus fragment. Then, a question and reference answer are generated using the large model, and the large language model (LLMs) to be tested need to provide answers to the questions based on the entire document. This strategy effectively overcomes the high annotation cost of traditional strategies and is more accurate than strategies based on large language models (LLMs).

[0136] In summary, the LoRAG of the present invention can cover multiple fields, including literature, law, finance, etc. This multi-field evaluation method can provide broader and deeper knowledge, support the verification of the performance of the retrieval enhancement generation large model system in various professional fields, and promote the progress and innovation of large models in RAG applications. The external documents selected by the LoRAG of the present invention are all long documents. The average length of Chinese documents is 76,000 characters, and the average length of English documents is 57,000 characters. The ability to perform retrieval enhancement and generation of long documents is indispensable in real life. LoRAG meets the needs of understanding long documents in reality, and reasonably evaluates the performance of large models on long document RAG tasks to promote its development.

Claims

1. A method for evaluating a large model for long document retrieval enhancement, characterized in that: The following steps are involved: S1. Build an automatic question-answering strategy based on focus segments; S2. Construct an evaluation dataset and process the evaluation dataset based on the automatic question strategy to obtain the optimal evaluation dataset for enhancing the generation of a large model for long document retrieval; S3. Evaluation is performed based on the optimal evaluation dataset for enhancing the generation of large models for long document retrieval.

2. The evaluation method for enhancing the generation of large models for long document retrieval according to claim 1 is characterized in that: The step S1 comprises the following steps: S101, obtaining each long document in the knowledge base, and randomly extracting n segments, wherein each segment is a focus segment; S102, setting prompts to guide large language models (LLMs) to generate qualified question-answer pairs; S103, combining the focus segment and the prompt, and inputting them into the large language model LLMs to generate context-related question-answer pairs: (Q^,A)=ChatGPT(d,p) d=D[start:start+N d ] Among them, Q^ represents the generated question, A represents the corresponding answer to the generated question, ChatGPT() large model generation process, d represents the focus segment, p represents the set prompt, D represents the long document, N d Indicates the length of the segment, and start indicates the starting position of the segment; S104. Use the following formula to filter the generated context-related question and answer pairs to obtain a result data set that meets the requirements, where the result data set includes question and answer instances (Qi, Ai, di, D): (Q,A)=Human(Q,A^) Among them, (Q,A) represents the question-answering instances that meet the requirements after screening, and Human() represents the manual screening process; S105, based on the result data set, use the document to evaluate the large language model LLMs and generate the answer A'i; S106: Compare the answer A'i with the reference answer Ai to obtain a comparison result, thereby completing the construction of the automatic question-answering strategy.

3. The evaluation method for enhancing the generation of large models for long document retrieval according to claim 2 is characterized in that: The expression of the comparison result is as follows: Score=Evaluation(A'i,Ai,Qi,di) A'i=LLMs(D,Qi) Dataset = {(Qi, Ai, di, D)} Among them, Score represents the evaluation score, Evaluation() represents the evaluation process, LLMs() represents the process of generating the large language model to be tested, Ai represents the reference answer i, Qi represents the question i, di represents the focus segment corresponding to Ai and Qi, and Dataset() represents the final generated data style.

4. The evaluation method for enhancing the generation of large models for long document retrieval according to claim 2 is characterized in that: The step S2 comprises the following steps: S201, in data selection, collect Chinese and English datasets from long documents, and merge the collected datasets into the bilingual dataset LoRA; S202, extracting focus segments from the long document, and inputting the extracted focus segments and the generated qualified question-answer pair prompts into the large language model LLMs, extracting text by generating qualified questions and their corresponding answers; S203: Screen the extracted text to obtain the optimal evaluation data set for enhancing the generation of a large model for long document retrieval.

5. The evaluation method for enhancing the generation of large models for long document retrieval according to claim 1 is characterized in that: The step S3 comprises the following steps: S301, obtaining the baseline model and evaluation indicators to be evaluated; S302, based on the optimal evaluation data set for the enhanced generation of a large model for long document retrieval, using the baseline model to generate corresponding responses in the evaluation data; S303. Using the obtained evaluation indicators, the corresponding responses in the generated evaluation data are compared with the reference answers to obtain evaluation scores, thereby completing the evaluation of the long document retrieval enhanced generation large model.

Images