Retrieval optimization method based on dynamic mixed retrieval and adjacent paragraph introduction

Abstract

The invention discloses a retrieval optimization method based on dynamic mixed retrieval and adjacent paragraph introduction. The method comprises the following steps: taking related documents as a corpus of a retriever; performing retrieval by using a BM25 algorithm and a HyDE algorithm to generate corresponding preliminary retrieval document paragraphs; calculating a query specificity value for user query by using TF-IDF; determining weights of a BM25 algorithm and a HyDE algorithm, and respectively performing weighted fusion on a document paragraph correlation score based on word frequency statistical calculation in the BM25 algorithm and a document paragraph correlation score based on semantic embedding similarity calculation in the HyDE algorithm to generate candidate document paragraphs and comprehensive scores corresponding to the candidate document paragraphs; the document paragraphs ranked in the top are selected from the preliminary retrieval document paragraphs, adjacent document paragraphs are obtained, weighted scores of the adjacent document paragraphs are calculated, the adjacent document paragraphs are added into a candidate document paragraph list, and the document paragraphs with the top scores are selected from the list to serve as retrieval results.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction. Background Art

[0002] In recent years, with the rapid development of natural language processing (NLP) technology, large language models (LLMs) have demonstrated excellent performance in language understanding and generation tasks and have been widely used in academic research and various practical fields.

[0003] However, LLMs are prone to hallucinations during the generation process—generating factual errors or inconsistent outputs that are inconsistent with the context. This severely limits their application in high-reliability scenarios. To address this issue, Retrieval-Augmented Generation (RAG) technology has emerged. By retrieving relevant documents from external knowledge bases and feeding them into large language models as context, RAG reduces hallucinations and improves response accuracy. The retrieval capability of a RAG system directly determines the quality of its generation. Its accuracy and relevance are crucial for reducing hallucinations and improving the reliability of LLMs, and therefore have become a core focus of research and optimization.

[0004] Research has shown that if the retrieval component of the RAG system converts the user-entered query into different levels of abstraction before conducting a document search, the relevance between the retrieved data and the user query can be significantly improved. Methods include expanding a single query into multi-query retrieval, chain of verification, pseudo-context search, etc. Hypothetical Document Embeddings (HyDE) proposed by Gao L et al. in 2023 is a retrieval enhancement technology for pseudo-context search. HyDE generates hypothetical documents related to the query, converts the generated hypothetical documents into embedding vectors, and uses these embedding vectors to perform similarity comparisons with document vectors in an external knowledge base. HyDE is very suitable for retrieval that needs to capture a wide range of semantic content. However, when faced with some query situations that require exact keyword matching, HyDE is slightly inferior to sparse retrievers.

[0005] Existing RAG systems typically use two main retrieval methods: sparse retrieval (such as keyword matching based on BM25) and dense retrieval (such as semantic search based on Sentence Transformers). Sparse retrieval relies on vocabulary matching and is suitable for processing specific queries, but it is not good at processing synonyms or semantic relationships; dense retrieval captures semantic information through embedding vectors and is suitable for fuzzy or broad queries, but it has high computational complexity and depends on model quality. In order to improve the overall accuracy of retrieval, hybrid retrieval methods are gradually being applied to RAG. Hybrid retrieval methods achieve the effect of balancing precision and recall by fusing the advantages of sparse and dense searchers. However, the fusion method of existing hybrid searchers is usually fixed, and the fusion processing process is not dynamically adjusted according to the type of query, resulting in insufficient adaptability of the searcher to different queries (such as specific or fuzzy queries) and limitations. Currently, Chandana Sree Mala et al. proposed a dynamically weighted hybrid retriever method in 2025 (Mala CS, Gezici G, GiannottiF, "Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysis"), which adjusts the contribution of sparse and dense retrievers according to query specificity and optimizes retrieval performance. However, in actual application, its effect may be limited by various factors and fails to fully achieve the expected performance.

[0006] In real-world applications of RAG systems, corpora are typically segmented into multiple document segments for retrieval. However, existing segmentation methods struggle to ensure that the complete information corresponding to a query is accurately segmented into a single document segment. As a result, the most relevant paragraph may contain only partial information, while other hidden information is distributed among its neighboring document segments. Therefore, by focusing on the most similar document segments and their neighboring document segments, this hidden information can be effectively mined, thereby improving the contextual completeness and accuracy of retrieval. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a retrieval optimization method based on dynamic hybrid retrieval and introduction of neighboring paragraphs, dynamically adjusting the retrieval strategy according to the query specificity, and at the same time introducing neighboring document paragraphs of the retrieval document paragraph with the highest matching score to mine hidden related information that may be overlooked, thereby improving the accuracy and contextual relevance of the retrieved document paragraphs.

[0008] The present invention is achieved through at least one of the following technical solutions.

[0009] A retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction, the document paragraph retrieval method comprises the following steps:

[0010] The text blocks obtained after preprocessing the relevant documents are used as the corpus of the retriever;

[0011] Obtain user queries and use large language models to generate pseudo documents related to the queries;

[0012] Use the BM25 algorithm and HyDE algorithm to search the corpus and generate the corresponding preliminary search document paragraphs;

[0013] Calculate the query specificity value using TF-IDF for user queries;

[0014] Based on the query specificity value, the weights of the BM25 algorithm and the HyDE algorithm are determined. The document paragraph relevance scores calculated based on word frequency statistics in the BM25 algorithm and the document paragraph relevance scores calculated based on semantic embedding similarity in the HyDE algorithm are weighted and fused to generate candidate document paragraphs and their corresponding comprehensive scores.

[0015] From the preliminary retrieval document paragraph rankings generated by the BM25 algorithm and the HyDE algorithm, select the top-ranked document paragraphs and obtain their adjacent document paragraphs. Combine the retriever weights and the set neighborhood weights to calculate the weighted scores of the adjacent document paragraphs, and add the adjacent document paragraphs to the list of candidate document paragraphs. Among the candidate document paragraphs, select the document paragraphs with the highest scores as the retrieval results.

[0016] Furthermore, the text blocks obtained after preprocessing the relevant documents are used as the corpus of the retriever, specifically including:

[0017] Use optical character recognition tools and document parsing tools to convert PDF and Word documents into Markdown format.

[0018] Use a text segmenter to segment the text into blocks of N characters each, with an overlap of h characters between adjacent blocks.

[0019] The text segmenter is used to segment the converted Markdown format document into paragraphs, generate multiple text blocks and store them.

[0020] Furthermore, we obtain user queries and use large language models to generate pseudo documents related to the queries, specifically including:

[0021] Constructing a structured pseudo-document prompt template, wherein the pseudo-document prompt template includes a user query, a generation instruction, and an example query-pseudo-document pair;

[0022] The structured pseudo-document prompt template is input into the large language model to guide the model to generate m pseudo-documents semantically relevant to the user query.

[0023] Furthermore, the BM25 algorithm is used to search the corpus and generate the corresponding preliminary search document paragraphs, including:

[0024] Split the query content input by the user based on spaces to generate a word segmentation query sequence, wherein the word segmentation query sequence is a word sequence composed of words in the query;

[0025] Calculate the retrieval score S of the word segmentation query and each document paragraph in the corpus based on the BM25 algorithm BM25_base (d i );

[0026] According to the retrieval score S BM25_base (d i ) Sort the document paragraphs in descending order, select the top k document paragraphs with the highest scores, and generate the BM25 preliminary retrieval document paragraphs.

[0027] Furthermore, the HyDE algorithm is used to search the corpus and generate the corresponding preliminary search document paragraphs, including:

[0028] Use the pre-trained embedding model all-MiniLM-L6-v2 to generate user query embedding vectors and hypothetical document embedding vectors for the user query and m pseudo documents generated by the pre-trained large language model;

[0029] Calculate the arithmetic mean of the hypothetical document embedding vectors and perform a proportional weighted average of the vectors with the user query embedding vector to generate a final query embedding vector;

[0030] Based on the final query embedding vector, similarity search is performed in the corpus using the FAISS index to generate the HyDE similarity score S for each document paragraph. HyDE_base (d i );

[0031] According to the HyDE similarity score S HyDE_base (d i ) Sort the document paragraphs in descending order, select the top k document paragraphs with the highest scores, and generate the HyDE preliminary retrieval document paragraphs.

[0032] Furthermore, the user query is subjected to TF-IDF to calculate the query specificity value, including:

[0033] Using all the document paragraphs in the corpus, generate a vocabulary V = {t1, t2, ..., t |V|}, where |V| represents the size of the vocabulary; t |V|represents the |V|th word;

[0034] Calculate the inverse document frequency of each word, where the calculation formula for the inverse document frequency is:

[0035]

[0036] Where N is the total number of document paragraphs in the corpus, t represents a word in the corpus vocabulary, and df(t) is the number of document paragraphs containing word t. The addition of 1 is to avoid the denominator being 0.

[0037] Perform word segmentation on the user query and calculate the word frequency of each word t in the query statement q. The word frequency tf(t,q) represents the word frequency of word t in the query statement q and is used to measure the relative importance of word t in the query. The calculation formula is as follows:

[0038]

[0039] Where t represents a word in the corpus vocabulary, count(t) represents the number of times word t appears in the query statement q, and |q| represents the total number of words in the query statement q;

[0040] Calculate the specificity value S(q) of the user query:

[0041]

[0042] q TF-IDF =[tf_idf(t1,q),tf_idf(t2,q),…,tf_idf(t |V| ,q)], t∈V;

[0043] Among them, nnz(q TF-IDF ) represents the number of non-zero elements in the user query TF-IDF vector, |q TF-IDF | is the total length of the TF-IDF vector; tf_idf(t,q) is the TF-IDF weight of term t in query q, which is obtained by multiplying the term frequency tf(t,q) by the inverse document frequency idf(t). Its calculation formula is as follows:

[0044] tf_if(t,q)=tf(t,q)×idf(t), t∈V.

[0045] Furthermore, based on the query specificity value, the weights of the BM25 algorithm and the HyDE algorithm are determined. The document paragraph relevance scores calculated based on word frequency statistics in the BM25 algorithm and the document paragraph relevance scores calculated based on semantic embedding similarity in the HyDE algorithm are weighted and fused to generate candidate document paragraphs and their corresponding comprehensive scores, including:

[0046] If the specificity value S(q) of the user query is greater than the preset threshold, the score of the document paragraph retrieved by the BM25 algorithm is assigned a larger weighted value w BM25 ;

[0047] On the contrary, if the specificity value S(q) of the user query is equal to or less than the preset threshold, the score of the document paragraph retrieved by the HyDE algorithm is given a larger weight value w HyDE ;

[0048] Calculate the weighted score S of the initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm BM25 (d i ) and S HyDE (d i ), S BM25 (d i ) and S HyDE (d i ) is calculated as follows:

[0049] S BM25 (d i )=S NM25_base (d i )×w BM25 ,

[0050] S HyDE (d i )=S HyDE_base (d i )×w HyDE ,

[0051] Among them, d i represents the i-th document paragraph, S BM25_base (d i ) represents the score of the document paragraph retrieved by the BM25 algorithm, S HyDE_base (d i ) represents the score of the document paragraph retrieved by the HyDE algorithm;

[0052] The initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm are put into the candidate document paragraph list.

[0053] Furthermore, from the document paragraphs obtained through the preliminary retrieval using the BM25 algorithm and the HyDE algorithm, the top k document paragraphs are selected, their adjacent document paragraphs are obtained, and the weighted scores of the adjacent document paragraphs are calculated by combining the retriever weight and the preset neighborhood weight. The adjacent document paragraphs are then added to the list of candidate document paragraphs, specifically including:

[0054] From the document paragraphs obtained by preliminary retrieval using the BM25 algorithm and the HyDE algorithm, select the top k document paragraphs DBM25_topK and D HyDE_topK , according to the index of the document paragraph, get its adjacent document paragraph D BM25_neighbor and D HyDE_neighbor ;

[0055] The scores of adjacent document paragraphs are fused with the neighborhood weight w neighhor Recalculate the weighted score S BM25_neighbor (d i ) and S HyDE_neighbor (d i ), the calculation process is as follows:

[0056] S BM25_neighbor (d i )=S BM25 (d i )×w neighbor

[0057] S HyDE_neighbor (d i )=S HyDE (d i )×w neighbor ;

[0058] Among them, S BM25_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the BM25 algorithm, S hyDE_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the HyDE algorithm, w neighbor Indicates the preset neighborhood weight value;

[0059] The adjacent document paragraph D BM25_neighbor and D HyDE_neighbor Put it into the candidate document paragraph list, and select the top n documents with the highest scores among the candidate document paragraphs as the retrieval results.

[0060] A system for implementing the retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction includes:

[0061] The corpus preprocessing unit is used to convert PDF and Word documents into Markdown format using optical character recognition and document parsing tools; a text segmenter is constructed, with the number of characters in a text block set to N and h characters of overlap between adjacent blocks; the segmenter is used to segment the Markdown document into blocks, generating and storing multiple text blocks;

[0062] The pseudo-document generation unit is used to construct a pseudo-document prompt template to guide the model to generate m pseudo-documents semantically related to the user query;

[0063] The retrieval unit is used to search the corpus using the BM25 algorithm and the HyDE algorithm to generate the corresponding preliminary retrieval document paragraphs;

[0064] The neighboring paragraph introduction unit is used to select the top-ranked document paragraphs from the preliminary retrieved document paragraphs and obtain their neighboring document paragraphs; combining the retriever weight and the preset neighborhood weight, the weighted score of the neighboring document paragraph is calculated and added to the list of candidate document paragraphs;

[0065] The screening output unit is used to select the top n document paragraphs with the highest scores from the candidate document paragraphs as the retrieval results.

[0066] A computer device of the present invention includes: a memory, a processor, and a computer program stored in the memory, and when the computer program is executed on the processor, the method described above is implemented.

[0067] Compared with the prior art, the beneficial effects of this invention are:

[0068] 1. This invention dynamically adjusts the weight assignments of the BM25 algorithm and the HyDE algorithm based on query specificity through TF-IDF calculation, rather than adopting the static fusion strategy commonly used in the prior art. This can more accurately adapt to different types of queries, thereby improving the relevance of search results.

[0069] 2. By introducing the neighboring document paragraphs of the retrieval document paragraph with the highest matching score and integrating dynamic weights and neighborhood weights to calculate the weighted score, the present invention effectively mines the hidden relevant information that is ignored by the existing technology due to incomplete corpus chunk cutting, makes up for the defect of missing context in the traditional RAG system, and improves the context completeness of the retrieval document paragraph. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart illustrating a retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction is provided;

[0071] Figure 2 A schematic diagram of a structure of a retrieval optimization system based on dynamic hybrid retrieval and adjacent paragraph introduction is provided as an embodiment;

[0072] Figure 3 A schematic structural diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0074] This embodiment discloses a retrieval optimization method based on dynamic hybrid retrieval and neighboring paragraph introduction, the purpose of which is to dynamically adjust the retrieval strategy according to the query specificity, introduce the neighboring paragraphs of the retrieval paragraph with the highest matching score to mine hidden related information that may be overlooked, thereby improving the accuracy and context relevance of the retrieval paragraph. Figure 1 As shown, the specific steps include:

[0075] S1. Using the text blocks obtained after preprocessing the relevant documents as the corpus of the retriever, including the following steps:

[0076] S101, using an optical character recognition tool (OCR) and a document parsing tool (Pandoc) to convert documents in PDF and Word formats into Markdown format;

[0077] S102. Use a text splitter to split the text into text blocks of N characters each, with a preset overlap of h characters between adjacent text blocks. As an embodiment, the text splitter is the RecursiveCharacterTextSplitter text splitter of LangChain.

[0078] S103: Use the text segmenter to segment the converted Markdown format document into paragraphs, generate multiple text blocks, and store them.

[0079] S2. Obtain user queries and use a large language model to generate pseudo documents related to the queries, including the following steps:

[0080] S201: Construct a structured pseudo-document prompt template, wherein the template includes a user query, a generation instruction, and an example query-pseudo-document pair;

[0081] S202: Input the structured pseudo-document prompt template into the large language model to guide the model to generate m pseudo-documents semantically relevant to the user query. As an embodiment, the large language model uses the pre-trained generative model Qwen2-7B-Instruct.

[0082] S3. Obtain the user query, use the BM25 algorithm and the HyDE algorithm to search the corpus, and generate the corresponding preliminary search document ranking based on the similarity score, including the following steps:

[0083] S301, obtaining the query content input by the user;

[0084] S302: Split the user query based on spaces to generate a word segmentation query sequence, where the word segmentation query sequence is a word sequence consisting of words in the query;

[0085] S303, calculating the retrieval score S of the word segmentation query sequence and each document paragraph in the corpus based on the BM25 algorithm BM25_base (d i );

[0086] S304, sorting the document paragraphs in descending order according to the BM25 algorithm retrieval scores, selecting the top k document paragraphs with the highest scores, and generating a BM25 preliminary retrieval document paragraph ranking;

[0087] S305, using the pre-trained generative model Qwen2-7B-Instruct to generate m hypothetical documents according to the user query;

[0088] S306, using the pre-trained embedding model all-MiniLM-L6-v2 to generate a user query embedding vector and a hypothetical document embedding vector for the user query and the hypothetical document respectively;

[0089] S307: Calculate the arithmetic mean of the hypothetical document embedding vector, and perform a weighted average of the hypothetical document embedding vector and the user query embedding vector in a 1:1 ratio to generate a final query embedding vector.

[0090] S308: Based on the final query embedding vector, similarity search is performed in the corpus using the FAISS index to generate a HyDE similarity score S for each document paragraph. HyDE_base (d i )

[0091] S309 , sorting the document paragraphs in descending order according to the similarity scores, selecting the top k document paragraphs with the highest scores, and generating HyDE preliminary retrieval document paragraphs.

[0092] S4. Calculate the query specificity value of the user query using TF-IDF. The calculated specificity value is used as the basis for the subsequent dynamic allocation of retriever weights. Specifically, the following steps are included:

[0093] S401, using all the document paragraphs in the corpus, generate a vocabulary V = {t1, t2, ..., t |V|}, where |V| represents the size of the vocabulary, t|V| Represents the |V|th word.

[0094] S402: Calculate the inverse document frequency (IDF) of each word, where the calculation formula for the inverse document frequency is:

[0095]

[0096] Where N is the total number of document paragraphs in the corpus, t represents a word in the corpus vocabulary, and df(t) is the number of document paragraphs containing word t. The addition of 1 is to avoid the denominator being 0.

[0097] S403: Perform word segmentation on the user query and calculate the term frequency (TF) of each term t in the query statement q. The term frequency tf(t,q) represents the frequency of term t in the query statement q and is used to measure the relative importance of term t in the query. The calculation formula is as follows:

[0098]

[0099] Where count(t) represents the number of times t appears in the query statement q, and |q| represents the total number of words in the query statement q.

[0100] S404. Calculate the user query specificity value S(q):

[0101]

[0102] q TF-IDF =[tf_idf(t1,q),tf_idf(t2,q),…,tf_idf(t |V| ,q)], t∈V;

[0103] Among them, nnz(q TF-IDF ) represents the number of non-zero elements in the user query TF-IDF vector, |q TF-IDF | is the total length of the TF-IDF vector; tf_idf(t,q) is the TF-IDF weight of term t in query q, which is obtained by multiplying the term frequency tf(t,q) by the inverse document frequency idf(t). Its calculation formula is as follows:

[0104] tf_idf(t,q)=tf(t,q)×idf(t), t∈V.

[0105] The calculated specificity value S(q) is used as the basis for the subsequent dynamic allocation of retriever weights.

[0106] S5. Score S for the document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm respectively according to the user query specificity value BM25_base (d i) and S HyDE_base (d i ) and put it into the candidate document paragraph list. The dynamic weighted assignment process includes:

[0107] S501. Determine the weighted assignments of the BM25 algorithm and the HyDE algorithm based on the specificity value S(q) of the user query:

[0108] If the specificity value S(q) of the user query is greater than the preset threshold, it means that the words contained in the query are more informative, usually rare words or terms in a specific field, the query intent is clearer, and more suitable for precise keyword matching. The score of the document paragraph retrieved by the BM25 algorithm will be given a weighted value w BM25 ;

[0109] On the contrary, if the specificity value S(q) of the user query is equal to or less than the preset threshold, it means that the query is relatively vague and it is more suitable to retrieve relevant document paragraphs by semantic similarity. The score of the document paragraph retrieved by the HyDE algorithm will be given a weighted value w HyDE ;

[0110] S502, calculate the weighted score S of the initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm BM25 (d i ) and S HyDE (d i ), S BM25 (d i ) and S HyDE (d i ) is calculated as follows:

[0111] S BM25 (d i )=S BM25_base (d i )×w BM25 ,

[0112] S HyDE (d i )=S HyDE_base (d i )×w HyDE ,

[0113] Among them, d i represents the i-th document paragraph, S BM25_base (d l ) represents the score of the document paragraph retrieved by the BM25 algorithm, S HyDE_base (d i ) represents the score of the document paragraph retrieved by the HyDE algorithm;

[0114] S503: Put the initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm into a candidate document paragraph list.

[0115] S6. Select the top k document paragraphs from the preliminary retrieved document paragraph rankings corresponding to the BM25 algorithm and the HyDE algorithm, obtain their adjacent document paragraphs, calculate the weighted scores of the adjacent document paragraphs by combining the retriever weight and the preset neighborhood weight, and add the adjacent document paragraphs to the list of candidate document paragraphs. The specific steps include:

[0116] S601: Select the top k document paragraphs D from the preliminary search document paragraphs generated by the BM25 algorithm and the HyDE algorithm. BM25_topK and D HyDE_topK , according to the index of the document paragraph, get its adjacent document paragraph D BM25_neighbor and D HyDE_neighbor ;

[0117] S602: The scores of adjacent document paragraphs S BM25 (d i ) and S HyDe (d i ), the neighborhood weight w of the fusion setting neighbor Recalculate the weighted score S BM25_neighbor (d i ) and S HyDE_neighbor (d i ), the calculation process is as follows:

[0118] S BM25_neighbor (d i )=S BM25 (d i )×w neighbor

[0119] S HyDE_neighbor (d i )=S HyDE (d o )×w neighbor ;

[0120] Among them, S BM25_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the BM25 algorithm, S HyDE_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the HyDE algorithm, w neighbor Indicates the preset neighborhood weight value;

[0121] S603, the adjacent document paragraph D BM25_neighbor and D HyDE_neighborPut the candidate document paragraphs into the list, and select the top n document paragraphs with the highest scores among the candidate document paragraphs as the retrieval results.

[0122] like Figure 2 As shown, a system for implementing the retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction includes:

[0123] The corpus preprocessing unit is used to convert PDF and Word documents into Markdown format using optical character recognition and document parsing tools; a text segmenter is constructed, with the number of characters in a text block set to N and h characters of overlap between adjacent blocks; the segmenter is used to segment the Markdown document into blocks, generating and storing multiple text blocks;

[0124] The pseudo-document generation unit is used to construct a pseudo-document prompt template to guide the model to generate m pseudo-documents semantically related to the user query;

[0125] The retrieval unit is used to search the corpus using the BM25 algorithm and the HyDE algorithm to generate the corresponding preliminary retrieval document paragraphs;

[0126] The neighboring paragraph introduction unit is used to select the top-ranked document paragraphs from the preliminary retrieved document paragraphs and obtain their neighboring document paragraphs; combining the retriever weight and the preset neighborhood weight, the weighted score of the neighboring document paragraph is calculated and added to the list of candidate document paragraphs;

[0127] The screening output unit is used to select the top n document paragraphs with the highest scores from the candidate document paragraphs as the retrieval results.

[0128] like Figure 3 As shown, this embodiment also provides a computer device, including a memory and a processor, wherein the memory is electrically connected to the processor, and the memory stores a computer program, and when the computer program is executed by the processor, the processor implements the described method.

[0129] This embodiment further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the method described above.

[0130] This paper conducted a validation experiment to verify the effectiveness of the proposed method. The document paragraph retrieval task was performed on the SciFix, NFCorpus, and FIQA datasets. This experiment used MRR@10, nDCG@10, and R@50 as evaluation metrics on the SciFix, NFCorpus, and FIQA datasets.

[0131] This experiment did not use the traditional method of evaluating relevance labels based on qrels (relevance judgment), but instead used the ROUGE-L score to dynamically judge the relevance of the retrieved documents. The reason is that in the actual Retrieval Augmented Generation (RAG) system, documents in the external database usually need to be cut into multiple small segments (chunks) to adapt to the retrieval model, but the chunks after cutting may be inconsistent with the paragraph range annotated by qrels, and contain some relevant information or irrelevant content, resulting in the binary label being unable to accurately reflect the semantic relevance. ROUGE-L (Longest Common Subsequence) flexibly evaluates the semantic overlap of chunks by setting a threshold, adapts to cutting deviations, and provides a more reliable performance evaluation. At the same time, in order to better evaluate the invention of introducing adjacent paragraphs, this experiment first splices all qrels-annotated paragraphs into a whole, and then divides them into chunks for retrieval.

[0132] Among them, MRR@10 is the reciprocal ranking of the first document paragraph whose ROUGE-L score exceeds the threshold among the first 10 retrieved document paragraphs, nDCG@10 represents the result of taking the first 10 relevant document paragraphs when calculating the normalized discounted cumulative return, and R@50 represents the recall rate of the first 50 relevant document paragraphs. The results of the document paragraph retrieval task on the Scifact, NFCorpus, and FIQA datasets are shown in Table 1:

[0133] Table 1. Document paragraph retrieval task results on the SciFix, NFCorpus, and FIQA datasets

[0134]

[0135] Dense, Sparse, and HyDE are all retrieval methods. Dense uses the pre-trained vector embedding model all-MiniLM-L6-v2 to generate query and document embedding vectors, respectively, and then performs similarity retrieval on these embedding vectors. Sparse, based on the BM25 algorithm, uses term frequency and inverse document frequency to calculate the relevance score between the query and the document for text retrieval. The HyDE retrieval method uses the pre-trained vector embedding model all-MiniLM-L6-v2 to generate hypothetical document embedding vectors corresponding to the query and then performs similarity retrieval on these vectors with the document embedding vectors. Bold values ​​indicate the highest value in that column.

[0136] Experimental results show that the proposed method achieves optimal performance in most metrics for paragraph retrieval tasks on the Scifact, NFCorpus, and FIQA datasets, outperforming the benchmark methods Dense, Sparse, and HyDE. This demonstrates that the proposed method effectively improves retrieval accuracy and contextual relevance through dynamic hybrid retrieval and the introduction of neighboring paragraphs.

[0137] In summary, the present invention provides a retrieval optimization method based on dynamic hybrid retrieval and neighboring paragraph introduction. It obtains user queries and generates preliminary rankings using the BM25 algorithm and the HyDE algorithm respectively, uses TF-IDF to calculate query specificity for dynamic weighted assignment, extracts adjacent document paragraphs of the highest-scoring document paragraph, fuses the dynamic weights and neighborhood weights, and adds them to a candidate list. Finally, the top n document paragraphs with the highest weighted scores are selected from the candidate list as retrieval results.

[0138] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein, but is intended to be embodied in the widest possible scope consistent with the principles and novel features disclosed herein.

Claims

1. A retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction, characterized in that: The document paragraph retrieval method comprises the following steps: The text blocks obtained after preprocessing the relevant documents are used as the corpus of the retriever; Obtain user queries and use large language models to generate pseudo documents related to the queries; Use the BM25 algorithm and HyDE algorithm to search the corpus and generate the corresponding preliminary search document paragraphs; Calculate the query specificity value using TF-IDF for user queries; Based on the query specificity value, the weights of the BM25 algorithm and the HyDE algorithm are determined. The document paragraph relevance scores calculated based on word frequency statistics in the BM25 algorithm and the document paragraph relevance scores calculated based on semantic embedding similarity in the HyDE algorithm are weighted and fused to generate candidate document paragraphs and their corresponding comprehensive scores. From the preliminary retrieval document paragraph rankings generated by the BM25 algorithm and the HyDE algorithm, select the top-ranked document paragraphs and obtain their adjacent document paragraphs. Combine the retriever weights and the set neighborhood weights to calculate the weighted scores of the adjacent document paragraphs, and add the adjacent document paragraphs to the list of candidate document paragraphs. Among the candidate document paragraphs, select the document paragraphs with the highest scores as the retrieval results.

2. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: The text blocks obtained after preprocessing the relevant documents are used as the corpus of the retriever, specifically including: Use optical character recognition tools and document parsing tools to convert PDF and Word documents into Markdown format. Use a text segmenter to segment the text into blocks of N characters each, with an overlap of h characters between adjacent blocks. The text segmenter is used to segment the converted Markdown format document into paragraphs, generate multiple text blocks and store them.

3. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: Obtain user queries and use large language models to generate pseudo documents related to the queries, including: Constructing a structured pseudo-document prompt template, wherein the pseudo-document prompt template includes a user query, a generation instruction, and an example query-pseudo-document pair; The structured pseudo-document prompt template is input into the large language model to guide the model to generate m pseudo-documents semantically relevant to the user query.

4. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: Use the BM25 algorithm to search the corpus and generate the corresponding preliminary search document paragraphs, including: Split the query content input by the user based on spaces to generate a word segmentation query sequence, wherein the word segmentation query sequence is a word sequence composed of words in the query; Calculate the retrieval score S of the word segmentation query and each document paragraph in the corpus based on the BM25 algorithm BM25_base (d i ); According to the retrieval score S BM25_base (d i ) Sort the document paragraphs in descending order, select the top k document paragraphs with the highest scores, and generate the BM25 preliminary retrieval document paragraphs.

5. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: Use the HyDE algorithm to search the corpus and generate the corresponding preliminary search document paragraphs, including: Use the pre-trained embedding model all-MiniLM-L6-v2 to generate user query embedding vectors and hypothetical document embedding vectors for the user query and m pseudo documents generated by the pre-trained large language model; Calculate the arithmetic mean of the hypothetical document embedding vectors and perform a proportional weighted average of the vectors with the user query embedding vector to generate a final query embedding vector; Based on the final query embedding vector, similarity search is performed in the corpus using the FAISS index to generate the HyDE similarity score S for each document paragraph. HyDE_base (d i ); According to the HyDE similarity score S HyDE_base (d i ) Sort the document paragraphs in descending order, select the top k document paragraphs with the highest scores, and generate the HyDE preliminary retrieval document paragraphs.

6. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: The user query is calculated using TF-IDF to calculate the query specificity value, including: Using all the document paragraphs in the corpus, generate a vocabulary V = {t1, t2, ..., t |V| }, where |V| represents the size of the vocabulary; t |V| represents the |V|th word; Calculate the inverse document frequency of each word, where the calculation formula for the inverse document frequency is: Where N is the total number of document paragraphs in the corpus, t represents a word in the corpus vocabulary, and df(t) is the number of document paragraphs containing word t. The addition of 1 is to avoid the denominator being 0. Perform word segmentation on the user query and calculate the word frequency of each word t in the query statement q. The word frequency tf(t,q) represents the word frequency of word t in the query statement q and is used to measure the relative importance of word t in the query. The calculation formula is as follows: Where t represents a word in the corpus vocabulary, count(t) represents the number of times word t appears in the query statement q, and |q| represents the total number of words in the query statement q; Calculate the specificity value S(q) of the user query: q TF-IDF =[tf_idf(t1,q),tf_idf(t2,q),…,tf_idf(t |V| ,q)],t∈V; Among them, nnz(q TF-IDF ) represents the number of non-zero elements in the user query TF-IDF vector, |q TF-IDF | is the total length of the TF-IDF vector; tf_idf(t,q) is the TF-IDF weight of term t in query q, which is obtained by multiplying the term frequency tf(t,q) by the inverse document frequency idf(t). Its calculation formula is as follows: tf_idf(t,q)=tf(t,q)×idf(t), t∈V.

7. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: Based on the query specificity value, the weights of the BM25 algorithm and the HyDE algorithm are determined. The document paragraph relevance scores calculated based on word frequency statistics in the BM25 algorithm and the document paragraph relevance scores calculated based on semantic embedding similarity in the HyDE algorithm are weighted and fused to generate candidate document paragraphs and their corresponding comprehensive scores. Specifically, the following are performed: If the specificity value S(q) of the user query is greater than the preset threshold, the score of the document paragraph retrieved by the BM25 algorithm is assigned a larger weighted value w BM25 ; On the contrary, if the specificity value S(q) of the user query is equal to or less than the preset threshold, the score of the document paragraph retrieved by the HyDE algorithm is given a larger weight value w HyDE ; Calculate the weighted score S of the initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm BM25 (d i ) and S HyDE (d i ), S BM25 (d i ) and S HyDE (d i ) is calculated as follows: S BM15 (d i )=S BM25_base (d i )×w BM25 , S HyDE (d i )=S HyDE_base (d i )×w HyDE , Among them, d i represents the i-th document paragraph, S bM25_base (d i ) represents the score of the document paragraph retrieved by the BM25 algorithm, S HyDE_base (d i ) represents the score of the document paragraph retrieved by the HyDE algorithm; The initial document paragraphs retrieved by the BM25 algorithm and the HyDE algorithm are put into the candidate document paragraph list.

8. The retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction according to claim 1 is characterized in that: From the document paragraphs obtained through the preliminary retrieval using the BM25 algorithm and the HyDE algorithm, select the top k document paragraphs, obtain their adjacent document paragraphs, and calculate the weighted scores of the adjacent document paragraphs by combining the retriever weights and the preset neighborhood weights. These adjacent document paragraphs are then added to the list of candidate document paragraphs. Specifically, the following steps are performed: From the document paragraphs obtained by preliminary retrieval using the BM25 algorithm and the HyDE algorithm, select the top k document paragraphs D BM25_topK and D HyDE_topK , according to the index of the document paragraph, get its adjacent document paragraph D BM25_neighbor and D HyDE_neighbor ; The scores of adjacent document paragraphs are fused with the neighborhood weight w neighbor Recalculate the weighted score S BM25_neighbor (d i ) and S HyDE_neighbor (d i ), the calculation process is as follows: S BM25_neighbor (d i )=S BM25 (d i )×w neighbor S HyDE_neighbor (d i )=S HyDE (d i )×w neighbor ; Among them, S BM25_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the BM25 algorithm, S HyDE_neighbor (d i ) represents the weighted score of the adjacent document paragraphs retrieved using the HyDE algorithm, w neighbor Indicates the preset neighborhood weight value; The adjacent document paragraph D BM25_neighbor and D HyDE_neihhnor Put it into the candidate document paragraph list, and select the top n documents with the highest scores among the candidate document paragraphs as the retrieval results.

9. A system for implementing the retrieval optimization method based on dynamic hybrid retrieval and adjacent paragraph introduction as claimed in claim 1, characterized in that: include: The corpus preprocessing unit is used to convert PDF and Word documents into Markdown format using optical character recognition and document parsing tools; a text segmenter is constructed, with the number of characters in a text block set to N and h characters of overlap between adjacent blocks; the segmenter is used to segment the Markdown document into blocks, generating and storing multiple text blocks; The pseudo-document generation unit is used to construct a pseudo-document prompt template to guide the model to generate m pseudo-documents semantically related to the user query; The retrieval unit is used to search the corpus using the BM25 algorithm and the HyDE algorithm to generate the corresponding preliminary retrieval document paragraphs; The neighboring paragraph introduction unit is used to select the top-ranked document paragraphs from the preliminary retrieved document paragraphs and obtain their neighboring document paragraphs; combining the retriever weight and the preset neighborhood weight, the weighted score of the neighboring document paragraph is calculated and added to the list of candidate document paragraphs; The screening output unit is used to select the top n document paragraphs with the highest scores from the candidate document paragraphs as the retrieval results.

10. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, which implements the method according to any one of claims 1 to 8 when the computer program is executed on the processor.

Images