Hierarchical retrieval enhancement generation method and system

Abstract

The invention discloses a hierarchical retrieval enhancement generation method and system. The method comprises the following steps: determining question information to be retrieved according to a hierarchical retrieval enhancement generation task; performing document-level retrieval on an external knowledge base based on the problem information to obtain a most relevant document set; slicing the most relevant document set according to paragraphs and performing paragraph-level retrieval to obtain a most relevant paragraph set; slicing the most relevant paragraph set according to sentences and performing sentence-level retrieval to obtain a most relevant sentence set; performing fragment generation operation on each sentence in the most relevant sentence set by utilizing a sliding window, and then performing fragment-level retrieval to obtain a most relevant fragment set; and carrying out information integration on the most relevant fragment set and knowledge in the model by utilizing a large language model, and carrying out context information supplement by utilizing factual knowledge obtained based on an external knowledge graph to obtain a context answer text of question information. According to the scheme, the accuracy and the reliability of a retrieval result can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval enhancement generation, and in particular to a hierarchical retrieval enhancement generation method and system. Background Art

[0002] Currently, the amount of information data is growing explosively. Most of this data exists in semi-structured, unstructured and other forms that are difficult to use directly. Among them, text data is a common and important data. In the current highly informationized environment, it is crucial to quickly and accurately extract key information from massive unstructured documents.

[0003] Traditional data processing methods cannot meet the modern needs of rapid understanding and decision-making when faced with these complex and ever-changing text data. Large Language Models (LLMs) provide new possibilities for processing massive amounts of text data due to their powerful language understanding and generation capabilities. However, these models are often limited by their static knowledge base and limited accuracy when faced with tasks that require real-time information or domain-specific knowledge. To address the above issues, a common solution is to use retrieval-augmented generation (RAG) technology, which integrates a rich external knowledge base and combines external knowledge with the model's internal knowledge, giving LLM the ability to acquire knowledge and information beyond its original training data. This type of technical integration enables LLM to generate more detailed and precise content when providing answers, while overcoming the limitations of static pre-trained models by providing the ability to dynamically access a wider knowledge base, thereby producing more accurate and relevant contextual answers.

[0004] In current models based on retrieval-enhanced generation, text processing usually relies on a simple chunking strategy. However, this strategy has significant limitations: when the chunks are too large, unnecessary noise information will be introduced, thus affecting the generation results of the model; and when the chunks are too small, the model may not be able to fully grasp the structure of the entire document, and can only rely on local information for reasoning, resulting in semantic understanding deviation. In this case, the model may misunderstand or make inaccurate judgments. Summary of the invention

[0005] In order to solve the technical problem that the current slicing strategy may cause misunderstanding or inaccurate judgment of the model generated based on retrieval enhancement, the embodiment of the present invention provides a hierarchical retrieval enhancement generation method and system.

[0006] The technical solution of the embodiment of the present invention is achieved as follows: An embodiment of the present invention provides a hierarchical retrieval enhancement generation method, the method comprising: Determine the question information to be retrieved according to the hierarchical retrieval enhancement generation task; Perform document-level retrieval on an external knowledge base based on the question information to obtain a document set most relevant to the question information; The most relevant document set is segmented into sections according to paragraphs, and a paragraph-level search is performed on the segmented paragraphs based on the question information to obtain a section of paragraphs that is most relevant to the question information; The most relevant paragraph set is segmented into sentences, and sentence-level retrieval is performed on the segmented sentences based on the question information to obtain a sentence set that is most relevant to the question information; Locate the position of each sentence in the most relevant sentence set in the paragraph, and according to the position, use a sliding window to splice each sentence in the most relevant sentence set with a plurality of surrounding continuous sentences into a segment, complete the segment generation operation, and perform segment-level retrieval on the spliced ​​segments based on the question information to obtain the segment set most relevant to the question information; A large language model is used to integrate the most relevant set of fragments and the internal knowledge of the large language model, and the integrated information is supplemented with contextual information using factual knowledge obtained based on an external knowledge graph to obtain a contextual answer text for the question information.

[0007] In one embodiment, a document-level search is performed on an external knowledge base based on the question information to obtain a document set most relevant to the question information, including: The following calculation formula is used to perform document-level retrieval on the external knowledge base based on the question information to obtain the document set most relevant to the question information: in, is the set of documents most relevant to the problem information, Indicates problem information. Represents a document in an external knowledge base, Represents document-level retrieval.

[0008] In one embodiment, paragraph-level retrieval is performed on the segmented paragraphs based on the question information to obtain a set of paragraphs most relevant to the question information, including: The following calculation formula is used to perform paragraph-level retrieval on the segmented paragraphs based on the question information to obtain the paragraph set most relevant to the question information: in, is the set of paragraphs most relevant to the question information, Indicates problem information. Indicates the segment after segmentation. Indicates paragraph-level retrieval.

[0009] In one embodiment, a sentence-level search is performed on the segmented sentences based on the question information to obtain a set of sentences most relevant to the question information, including: in, is the set of sentences most relevant to the question information, Indicates problem information. Indicates the sentence after segmentation. Represents sentence-level retrieval.

[0010] In one embodiment, performing segment-level retrieval on the spliced ​​segments based on the question information to obtain a segment set most relevant to the question information includes: in, is the set of fragments most relevant to the question information, Indicates problem information. Represents the spliced ​​fragments, Represents fragment-level retrieval.

[0011] In one embodiment, the performing document-level search on an external knowledge base based on the question information includes: Based on the question information, use any one of sparse retrieval, dense retrieval and integrated retrieval to perform document-level retrieval on the external knowledge base; The performing paragraph-level retrieval on the segmented paragraphs based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform paragraph-level retrieval on the segmented paragraphs; The performing sentence-level retrieval on the segmented sentences based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform sentence-level retrieval on the segmented sentences; The performing segment-level retrieval on the spliced ​​segments based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform segment-level retrieval on the spliced ​​segments.

[0012] In one embodiment, the sparse search includes: The relevance score between the text and question information is calculated using the following formula based on the BM25 algorithm; in, Is the text under sparse retrieval Information about the problem The correlation score between , is the number of words in the question information, Yes word In text The frequency in Yes word The inverse text frequency of Represents text The character length of represents the average character length of all texts, and are adjustable parameters, representing the term frequency saturation parameter and length normalization parameter respectively; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the sparse retrieval output set.

[0013] In one embodiment, the intensive search includes: The query information and text are encoded into a high-dimensional vector representation using a pre-trained language model, the cosine similarity between the question information and the text is calculated using the following formula, and the cosine similarity is used as the correlation score between the text and the question information; in, To search the text in dense Information about the problem The correlation score between and The problem information Vectors and text The vector of represents the magnitude of a vector; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the intensive retrieval output set.

[0014] In one embodiment, the integrated search includes: Obtaining a sorted set for sparse retrieval sorted by relevance score; and obtaining a sorted set for dense retrieval sorted by relevance score; Based on the above two sorting sets, the relevance score between the text and the question information is calculated according to the following calculation formula; in, Is the integrated search text Information about the problem The correlation score between When sparse or dense retrieval is indicated, the text In the text set Ranking in , For text set The number of Chinese characters, , is the sorting ranking of the sparse search BM25 algorithm, is the ranking of intensive retrieval, It is an adjustable parameter; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the integrated retrieval output set.

[0015] An embodiment of the present invention further provides a hierarchical retrieval enhancement generation system, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to execute the steps of the above-mentioned method when running the computer program.

[0016] This embodiment has the following beneficial effects: This embodiment proposes a hierarchical retrieval enhancement generation method for unstructured documents. The method can filter out redundant information and retain rich context information, thereby reducing the occurrence of hallucination phenomena. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a process flow of a hierarchical retrieval enhancement generation method according to an embodiment of the present invention; Figure 2 A schematic diagram of a hierarchical retrieval enhancement generation framework according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hierarchical retrieval process according to an embodiment of the present invention; Figure 4 A schematic diagram of a detailed process of hierarchical retrieval according to an embodiment of the present invention; Figure 5 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0019] The embodiment of the present invention provides a hierarchical retrieval enhancement generation method, such as Figure 1 As shown, the method includes: Step 101: determining the question information to be retrieved according to the hierarchical retrieval enhancement generation task; Step 102: Perform document-level retrieval on an external knowledge base based on the question information to obtain a document set most relevant to the question information; Step 103: Segment the most relevant document set into blocks according to paragraphs, and perform paragraph-level retrieval on the segmented paragraphs based on the question information to obtain a set of paragraphs most relevant to the question information; Step 104: segmenting the most relevant paragraph set into blocks according to sentences, and performing sentence-level retrieval on the segmented sentences based on the question information to obtain a set of sentences most relevant to the question information; Step 105: locate the position of each sentence in the most relevant sentence set in the paragraph, and splice each sentence in the most relevant sentence set and a plurality of surrounding continuous sentences into a segment using a sliding window according to the position, completing the segment generation operation, and performing segment-level retrieval on the spliced ​​segments based on the question information to obtain the segment set most relevant to the question information; Step 106: Use a large language model to integrate the most relevant set of fragments and the internal knowledge of the large language model, and use factual knowledge obtained based on an external knowledge graph to supplement the integrated information with context information to obtain a context answer text for the question information.

[0020] Unstructured documents often contain a lot of noise information, and existing block strategies are difficult to effectively distinguish, resulting in a lot of irrelevant content mixed into the blocks, affecting the quality of model generation. To address this problem, this embodiment proposes a hierarchical retrieval enhanced generation method, which retrieves the block information layer by layer to obtain high-quality, low-redundancy information blocks, ensuring the accuracy and reliability of the model output.

[0021] like Figure 2 As shown, the method of this embodiment adopts the strategy of layered slicing and layer-by-layer retrieval in the indexing and retrieval stages to filter out redundant information and retain relevant content. Then, large language models (LLMs) are used to summarize the retrieved information and refine it into a coherent summary, thereby forming information with complete context and avoiding the problem of hallucination caused by semantic understanding bias. At the same time, in order to avoid the loss of key information and affect the final answer, this embodiment also introduces an external knowledge graph to obtain factual knowledge and supplement context information to alleviate the problem of hallucination.

[0022] The core of this embodiment is to effectively integrate external knowledge with the internal knowledge of the model. This integration not only enriches the depth of the context, but also ensures the accuracy of the facts. and a set of documents retrieved from an external source , to generate the optimal output , the target of this embodiment is defined as: in, Represents the set of documents retrieved. These document blocks are designed to maximize the probability, namely: The model of this embodiment firstly inputs Retrieve the most relevant set of documents , and then generate by integrating this external knowledge with the internal knowledge of LLM This framework can dynamically acquire external knowledge and effectively integrate relevant factual knowledge with the model's internal knowledge, thereby enhancing the model's adaptability and contextual understanding level, and enabling it to maintain a high degree of relevance and reliability in a constantly changing information environment, thereby improving the overall generation quality.

[0023] Specifically, for text data, this embodiment draws on the cognitive mechanism of human reading and proposes a hierarchical retrieval method based on sentence window retrieval technology, such as Figure 3 and Figure 4 As shown, the method of this embodiment consists of two modules: hierarchical retrieval and information aggregation.

[0024] Hierarchical retrieval module. The first step is document-level retrieval, the purpose is to filter out a small number of most relevant documents from a large number of documents. , through document-level retrieval selection and query Highly relevant front Documents form a collection : Since LLM has a context length limit, the second step is to perform paragraph-level retrieval to reduce a large amount of redundant information. After segmenting by paragraph, perform paragraph-level search to obtain a paragraph set In order to ensure the coherence of the context between paragraphs, overlapping information can be retained between consecutive paragraphs: in, The document set The paragraph set obtained by segmentation processing, The most relevant A set of paragraphs, whose information content relationship is . Filtering by paragraph can quickly get the location of key information, but it also contains a lot of noise information. In the third step, sentence-level retrieval, the information is further refined. During the fine-grained block processing, ensure that the information between each block is independent of each other to reduce noise interference, and perform sentence-level retrieval to obtain the sentence set : in, Represents a set of paragraphs The information content of the sentence set after segmentation is: Since the correlation between the sentences selected in the third step to reduce redundant information is low, in the fourth step of fragment retrieval, a sliding window is used to splice multiple consecutive sentences into one fragment to ensure that the fragment can retain some context information of the sentence, and a fragment-level retrieval is performed to obtain the fragment set. : in, Represents a set of sentences The spliced ​​fragment collection is retrieved Only part of The information is supplemented by other sentence information, and the information volume relationship is and .

[0025] According to the above-mentioned layer-by-layer retrieval method, key information is repeatedly refined while the amount of information is continuously reduced. In order to avoid semantic understanding deviation caused by missing context, fragments are used to supplement some contextual information of key sentences.

[0026] After the hierarchical retrieval module has retrieved the information, there may still be fragments of missing information, such as reference resolution problems. Therefore, the knowledge injection module is used to obtain factual knowledge related to the user's query entity from the knowledge graph to supplement the context. Finally, through the carefully crafted information summary prompt words, the LLM language understanding ability is used to identify and complete the summary of the key content of the retrieved information around the question. This step aims to further eliminate redundant information and generate information with a more complete context to alleviate the illusion caused by semantic understanding bias. The refined information is finally handed over to the LLM in the generation stage to obtain high-quality answers.

[0027] In this embodiment, when performing document-level retrieval, paragraph-level retrieval, sentence-level retrieval, and fragment-level retrieval, one of sparse retrieval, dense retrieval, and integrated retrieval can be selected as a retrieval algorithm. Among them, sparse retrieval uses the BM25 algorithm, dense retrieval uses a pre-trained language model, and integrated retrieval integrates the BM25 algorithm and dense retrieval to complete the retrieval task.

[0028] The core of the BM25 algorithm is based on word frequency and inverse document frequency, and introduces document length information to calculate a document and query The correlation between them is calculated as follows: in, It is a sparse search document With query The relevance score of Word composition , Yes Query A word in . Yes word In the documentation The frequency in Yes word The inverse document frequency of . Representation Document The character length of Represents the average character length of all documents. and are adjustable parameters, representing the term frequency saturation parameter and length normalization parameter respectively.

[0029] Dense retrieval is to use pre-trained language models to convert queries and Documentation Encoded as a high-dimensional vector representation, the cosine similarity between the two is used as a measure of the relevance between the query and the document. The mathematical formula is as follows: in, and Query Vectors and documentation The vector of represents the magnitude of a vector, For intensive search and The relevance score.

[0030] The core of integrated retrieval is to use multiple retrieval methods to search for target information at the same time. The Reciprocal Rank Fusion (RRF) algorithm is used to integrate sparse retrieval and dense retrieval to build an integrated retriever. A collection of documents , according to the above two retrieval methods, calculate each document With query The scores are sorted by scores to obtain two sorted sets of search results. , is the sorting ranking of the sparse search BM25 algorithm, is the ranking of intensive retrieval. Finally, RRF is used to evaluate the final score of each information block: in, Representation Retriever Search Query When the document In the information set Ranking in . It is an adjustable parameter. It is a query under integrated search With Documentation The final score of relevance.

[0031] That is, when performing document-level retrieval, paragraph-level retrieval, sentence-level retrieval, and fragment-level retrieval, for a query and a set of texts , select one of sparse retrieval, dense retrieval and integrated retrieval as the retrieval algorithm. Through this algorithm, obtain the previous The most relevant texts form a set , the formula is defined as follows: Indicates that the search algorithm calculates and The relevance score of each text in The most relevant text collection.

[0032] The method of this embodiment can effectively reduce noise information while retaining context information through layer-by-layer slicing and layered retrieval, provide high-quality text blocks to alleviate model hallucinations, and thus ensure the accuracy of the answer.

[0033] In order to verify the effectiveness of the method of this embodiment, this embodiment also conducts a comprehensive evaluation and detailed analysis on the Chinese domain question answering dataset and the CRAG question answering evaluation benchmark.

[0034] 1 Experimental Configuration In this experiment, llama3-8b-instruct is used as the base model, and the model is deployed on the server using the vllm architecture. The model parameter configuration used is: , and limit the search content to 6000 characters. For the retriever algorithm, the BM25 algorithm is used to implement sparse search, bge-large-zh-v1.5 is used as the embedding model for dense search on the Chinese domain question answering dataset, and all-MiniLM-L6-v2 is used as the embedding model for dense search on the CRAG base dataset.

[0035] In this experiment, unless otherwise specified, all hierarchical retrievers use the following parameter configuration: ,in and are the number of candidate documents, paragraphs, sentences, and fragments, respectively. In addition, the chunk sizes of paragraph chunks and sentence chunks are 500 and 50, respectively, and each fragment consists of 3 sentences.

[0036] 2 Evaluation Set Introduction Multi-document question answering is an information extraction technology that aims to answer users' questions by integrating the contents of multiple documents. Due to the lack of multi-document question answering datasets in the Chinese field, we used GLM-4-0520 and Wenxin Yiyanda model to generate question-answer pairs based on existing reading comprehension datasets, and manually reviewed and adjusted them to construct a multi-document question answering dataset containing six types of questions. This dataset divides questions into six types of questions: simple questions based on information summaries, collection questions that require comprehensive answers from multiple pieces of information, computational questions that require combining multiple pieces of information and calculating, multi-hop questions that undergo multi-step reasoning, unfounded questions without information sources, and invalid questions with incorrect premises. Each question contains a standard answer and 100 documents to imitate web search results, and the evaluation set consists of 200 pieces of data.

[0037] The CRAG dataset contains multi-document question-answer pairs from five fields, namely finance, sports, music, movies, and other professional fields. 200 questions were randomly selected from them as a private dataset for performance evaluation.

[0038] 3 Evaluation methods and indicators The scoring indicators are set to quantify the quality performance of the model when answering questions. Specifically, the quality of the answer is divided into four levels: perfect, correct, missing, and incorrect. When the model's answer is completely consistent with the standard answer, it is rated as a perfect answer and receives 1 point. If the model cannot provide an answer, that is, the answer is "I don't know", it is considered a missing answer and such an answer receives 0 points. For other types of answers, the semantic understanding ability of the large language model GLM-4-0520 is used to determine their correctness. A correct answer receives 1 point; an incorrect answer receives -1 point. Based on these scores, the accuracy rate, hallucination rate, and missing rate are calculated as evaluation indicators of the answer quality. The final score of the model, which is a quantitative indicator of the model's question-answering ability, is determined by the difference between the accuracy rate and the hallucination rate. The specific evaluation indicator formula is as follows: in, Represents the total number of questions, and Represents the number of questions the model answered correctly and could not answer, respectively. , and Represent the accuracy, missing rate and hallucination rate of the model's answers, respectively, and It represents the final score of the model’s question-answering ability.

[0039] 4 Experimental Methods 4.1 CRAG Baseline Comparison Experiment (1) Official LLM baseline: the standard RAG framework officially provided by Meta KDD Cup 2024; (2) Official LLM+KG baseline: Based on the official LLM baseline, knowledge graph calls are added; (3) db3: The first-place solution in the Meta KDD Cup 2024 competition; (4) ElectricSheep: an outstanding solution in the Meta KDD Cup 2024 competition; (5) HRAG: This embodiment proposes a hierarchical retrieval enhancement generation method and calls the knowledge graph provided by the CRAG dataset to implement the knowledge injection module.

[0040] 4.2 Retrieval Algorithm Comparison Experiment (1) HRAG (BM25): The hierarchical retrieval enhancement generation method proposed in this embodiment uses the BM25 algorithm to build a retriever; (2) HRAG (Embedding): The hierarchical retrieval enhancement generation method proposed in this embodiment uses a pre-trained embedding model to build a retriever; (3) HRAG (Ensemble): The hierarchical retrieval enhancement generation method proposed in this embodiment uses an integrated approach to construct an integrated retriever.

[0041] 4.3 Cutting size comparison experiment (1) HRAG(P_x): During hierarchical retrieval, the maximum length of each paragraph block is limited to x characters, each sentence is limited to 50 characters, and each segment consists of 3 sentences; (2) HRAG(S_y): During the hierarchical retrieval process, the maximum length of each sentence is limited to y characters, the maximum length of each paragraph block is limited to 500 characters, and each paragraph consists of 3 sentences; (3) HRAG(F_z): In the hierarchical retrieval process, each segment is composed of z consecutive sentences. The maximum length of each paragraph block is limited to 500 characters, and each sentence is limited to 50 characters.

[0042] 4.4 Ablation Experiment (1) HRAG (All): The complete hierarchical retrieval method proposed in this embodiment. For the CRAG benchmark, this method also includes a knowledge injection module; (2) w / o Fragment: The hierarchical retrieval stage removes the fragment-level retrieval and outputs the candidate sentences as the retrieval results; (3) w / o Fragment & Sentence: In the hierarchical retrieval stage, sentence-level retrieval and fragment-level retrieval are removed, and the candidate paragraphs are output as retrieval results; (4) w / o Paragraph & Sentence: In the hierarchical retrieval stage, paragraph-level retrieval and sentence-level retrieval are removed, and the candidate segments are output as retrieval results to simulate ordinary RAG retrieval results; (5) w / o KG: removing the knowledge injection module in the hierarchical retrieval enhancement generation method; (6) w / o KG&Summary: Remove the knowledge injection module and information summary module in the hierarchical retrieval enhancement generation method.

[0043] 5. Experimental results analysis 5.1 Overall Experimental Results In order to comprehensively evaluate the impact of different retrievers on the method of this embodiment, detailed comparative experiments were conducted on the CRAG evaluation set and the domain multi-document question-answering evaluation set. Specifically, three different retrieval algorithms were tested: BM25 sparse retrieval, dense retrieval, and integrated retrieval. According to the experimental results, it can be seen that for the hierarchical retrieval enhancement generation method, the use of a dense retrieval algorithm can more effectively capture information related to the query and can better reduce the introduction of irrelevant information. This helps to reduce the risk of the model generating erroneous or irrelevant information. Based on this finding, in order to improve the experimental results, a dense retrieval algorithm was used to construct the retriever in subsequent experiments: The analysis results show that the method proposed in this embodiment has achieved significant improvement in the key indicator of accuracy. On the CRAG evaluation set, the accuracy is improved by 11% compared with the best baseline model. This result shows that the layered retrieval enhancement generation method allows LLM to make fuller use of external knowledge sources to improve the accuracy of its answers.

[0044] Compared with the traditional block cutting strategy, the method of this embodiment has significantly improved the accuracy and hallucination rate indicators of the Chinese domain dataset and the CRAG dataset. Experiments have proved that this method has achieved an effective balance between the efficiency and accuracy of information retrieval, thereby optimizing the overall retrieval performance.

[0045] In summary, the hierarchical retrieval enhancement generation method proposed in this embodiment can effectively extract fine-grained information through hierarchical retrieval and information summary modules, while retaining relatively rich contextual information, and greatly reducing redundant and noisy information through the information summary module. This comprehensive strategy significantly improves the accuracy and reliability of retrieval results.

[0046] The main contributions of this embodiment can be summarized as follows: (1) A domain-specific multi-document question answering dataset was constructed. Based on the existing reading comprehension dataset, LLM and manually constructed question pairs were used, and a set of 100 documents, including relevant documents and noisy documents, was assigned to each question to evaluate the performance of the model on the question answering task.

[0047] (2) A hierarchical retrieval enhancement generation method for unstructured documents is proposed. This method can filter out redundant information and retain rich contextual information, thereby reducing the occurrence of hallucination phenomena.

[0048] (3) A comprehensive evaluation and detailed analysis were conducted on the domain question answering dataset and the CRAG benchmark. The experimental results show that this method has obvious advantages in improving the accuracy of LLM.

[0049] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a hierarchical retrieval enhancement generation system, including: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the steps of the above-mentioned method.

[0050] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0051] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0052] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0053] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0054] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0055] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0056] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0060] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0061] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0062] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0063] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0064] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A hierarchical retrieval enhancement generation method, characterized in that: The method comprises: Determine the question information to be retrieved according to the hierarchical retrieval enhancement generation task; Perform document-level retrieval on an external knowledge base based on the question information to obtain a document set most relevant to the question information; The most relevant document set is segmented into sections according to paragraphs, and a paragraph-level search is performed on the segmented paragraphs based on the question information to obtain a section of paragraphs that is most relevant to the question information; The most relevant paragraph set is segmented into sentences, and sentence-level retrieval is performed on the segmented sentences based on the question information to obtain a sentence set that is most relevant to the question information; Locate the position of each sentence in the most relevant sentence set in the paragraph, and according to the position, use a sliding window to splice each sentence in the most relevant sentence set with a plurality of surrounding continuous sentences into a segment, complete the segment generation operation, and perform segment-level retrieval on the spliced ​​segments based on the question information to obtain the segment set most relevant to the question information; A large language model is used to integrate the most relevant set of fragments and the internal knowledge of the large language model, and the integrated information is supplemented with contextual information using factual knowledge obtained based on an external knowledge graph to obtain a contextual answer text for the question information.

2. The hierarchical retrieval enhancement generation method according to claim 1, characterized in that: Based on the question information, a document-level search is performed on an external knowledge base to obtain a document set most relevant to the question information, including: The following calculation formula is used to perform document-level retrieval on the external knowledge base based on the question information to obtain the document set most relevant to the question information: in, is the set of documents most relevant to the problem information, Indicates problem information. Represents a document in an external knowledge base, Represents document-level retrieval.

3. The hierarchical retrieval enhancement generation method according to claim 1, characterized in that: Based on the question information, paragraph-level retrieval is performed on the segmented paragraphs to obtain a set of paragraphs most relevant to the question information, including: The following calculation formula is used to perform paragraph-level retrieval on the segmented paragraphs based on the question information to obtain the paragraph set most relevant to the question information: in, is the set of paragraphs most relevant to the question information, Indicates problem information. Indicates the segment after segmentation. Indicates paragraph-level retrieval.

4. The hierarchical retrieval enhancement generation method according to claim 1, characterized in that: Based on the question information, sentence-level retrieval is performed on the segmented sentences to obtain a set of sentences most relevant to the question information, including: in, is the set of sentences most relevant to the question information, Indicates problem information. Indicates the sentence after segmentation. Represents sentence-level retrieval.

5. The hierarchical retrieval enhancement generation method according to claim 1, characterized in that: Performing segment-level retrieval on the spliced ​​segments based on the question information to obtain a segment set most relevant to the question information includes: in, is the set of fragments most relevant to the question information, Indicates problem information. Represents the spliced ​​fragments, Represents fragment-level retrieval.

6. The hierarchical retrieval enhancement generation method according to claim 1, characterized in that: The performing document-level retrieval on the external knowledge base based on the question information includes: Based on the question information, use any one of sparse retrieval, dense retrieval and integrated retrieval to perform document-level retrieval on the external knowledge base; The performing paragraph-level retrieval on the segmented paragraphs based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform paragraph-level retrieval on the segmented paragraphs; The performing sentence-level retrieval on the segmented sentences based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform sentence-level retrieval on the segmented sentences; The performing segment-level retrieval on the spliced ​​segments based on the question information includes: Based on the question information, any one of sparse retrieval, dense retrieval and integrated retrieval is used to perform segment-level retrieval on the spliced ​​segments.

7. The hierarchical retrieval enhancement generation method according to claim 6, characterized in that: The sparse search comprises: The relevance score between the text and question information is calculated using the following formula based on the BM25 algorithm; in, Is the text under sparse retrieval Information about the problem The correlation score between , is the number of words in the question information, Yes word In text The frequency in Yes word The inverse text frequency of Represents text The character length of represents the average character length of all texts, and are adjustable parameters, representing the term frequency saturation parameter and length normalization parameter respectively; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the sparse retrieval output set.

8. The hierarchical retrieval enhancement generation method according to claim 6, characterized in that: The intensive search includes: The query information and text are encoded into a high-dimensional vector representation using a pre-trained language model, the cosine similarity between the question information and the text is calculated using the following formula, and the cosine similarity is used as the correlation score between the text and the question information; in, To search the text in dense Information about the problem The correlation score between and The problem information Vectors and text The vector of represents the magnitude of a vector; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the intensive retrieval output set.

9. The hierarchical retrieval enhancement generation method according to claim 6, characterized in that: The integrated search comprises: Obtaining a sorted set for sparse retrieval sorted by relevance score; and obtaining a sorted set for dense retrieval sorted by relevance score; Based on the above two sorting sets, the relevance score between the text and the question information is calculated according to the following calculation formula; in, Is the integrated search text Information about the problem The correlation score between When sparse or dense retrieval is indicated, the text In the text set Ranking in , m For text set The number of Chinese characters, , is the sorting ranking of the sparse search BM25 algorithm, is the ranking of intensive retrieval, It is an adjustable parameter; Sort the relevance scores from high to low, and use the first preset number of texts in the sequence as the integrated retrieval output set.

10. A hierarchical retrieval enhancement generation system, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 9 are executed.

Images