A retrieval augmented generation optimization method, system, device, product, and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a retrieval enhancement generation optimization method, system, device, product and medium, which comprises the following steps: obtaining an original document, arranging and indexing the original document to obtain a retrieval document; constructing a hybrid retriever, obtaining a first candidate document through the hybrid retriever; generating a hypothetical document through the first candidate document, performing secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain a second candidate document; calculating the information correlation degree of the second candidate document; screening the second candidate document to obtain a coarsely screened document; recombining and segmenting the coarsely screened document to obtain a segmented document, calculating the key information density score of a word in the segmented document to obtain the lexical key information density score of the word; and screening the segmented document according to a fine screening threshold and the lexical key information density score to obtain a target document. The application effectively improves the reliability and retrieval quality of retrieval enhancement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation optimization method, system, device, product and medium. Background Art

[0002] Large-scale language models excel in reasoning and demonstrate potential for widespread application in real-world scenarios. However, when users require specialized responses tailored to specific domains, general-purpose large-scale language models often face challenges. Due to the breadth and limitations of their training data, the models may lack specialized domain knowledge, leading to "hallucinations" in their output. While the responses generated by the models appear linguistically coherent and fluent, they frequently contain factual errors, making the reliability of the responses unreliable.

[0003] Retrieval-augmented generation is a method that combines information retrieval techniques from external knowledge bases to inject external knowledge into model prompts to enhance the performance of large language models. It addresses the limitations of large language models in knowledge-intensive tasks, such as answering complex questions and generating detailed explanations, increases the transparency of the model's reasoning process, and mitigates the phenomenon of hallucinations in model generation. The introduction of retrieved documents significantly increases the number of prompt words fed into large language models, but this also presents the following difficulties: Large language models have a fixed context window, so prompts must be processed to fit within the context window. Simple processing methods can result in the loss of critical information; long context prompts increase the cost of calling the model and the time required to respond; long context prompts contain less critical content, making the model prone to overlooking information located in the middle of the prompt, thus affecting the reliability of the generated results. Summary of the Invention

[0004] The present invention aims to address at least one of the technical problems existing in the related art. To this end, the present invention provides a retrieval enhancement generation optimization method, system, device, product, and medium. These methods achieve improved performance in zero-shot document retrieval tasks through hybrid retrieval, and increase the proportion of key information in the prompt input of large language models, thereby effectively improving the performance of large language models.

[0005] The present invention provides a retrieval enhancement generation optimization method, comprising:

[0006] S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0007] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0008] S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0009] S4: Obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document using the local language model. Calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0010] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words;

[0011] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0012] According to a search enhancement generation optimization method provided by the present invention, step S1 further includes:

[0013] S11: Acquire the original document, determine the target format, remove useless characters in the original document, and divide and convert the original document into a format according to the target format to obtain an intermediate document;

[0014] S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, and fuse the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

[0015] According to a search enhancement generation optimization method provided by the present invention, step S2 further includes:

[0016] S21: Acquire a sparse searcher and a dense searcher, and construct the hybrid searcher by using the sparse searcher and the dense searcher;

[0017] S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

[0018] According to a search enhancement generation optimization method provided by the present invention, step S3 further includes:

[0019] S31: Acquire a task instruction and a large language model, input the task instruction, the query statement, and the first candidate document into the large language model to generate the hypothetical document;

[0020] S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

[0021] According to a search enhancement generation optimization method provided by the present invention, step S4 further includes:

[0022] S41: Decomposing the second candidate document to obtain a document sequence, a statement sequence, and an instruction sequence;

[0023] S42: Obtain a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence;

[0024] S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence;

[0025] S44: Determine a coarse screening compression rate, calculate a coarse screening word quantity limit based on the coarse screening compression rate, calculate a document key information density score based on the information relevance, the first cross entropy loss, and the second cross entropy loss, screen the second candidate document based on the document key information density score and the coarse screening word quantity limit to obtain the coarse screening document.

[0026] According to a retrieval enhancement generation optimization method provided by the present invention, in step S5, the coarsely screened documents are reorganized to obtain reorganized documents, paragraph lengths are determined, and the reorganized documents are segmented according to the paragraph lengths to obtain the segmented documents.

[0027] The present invention also provides a search enhancement generation optimization system, comprising:

[0028] Retrieval document module: used to obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0029] First candidate document module: used to construct a hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0030] A second candidate document module is configured to generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0031] A coarse document screening module is configured to obtain a first regularized statement, a second regularized statement, and a local language model, split the second candidate document, calculate the information relevance of the second candidate document using the local language model, and sequentially calculate a first cross entropy loss and a second cross entropy loss using the first regularized statement and the second regularized statement; screen the second candidate document using the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document;

[0032] A vocabulary key information density score module is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words;

[0033] Target document module: used to determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-described retrieval enhancement generation optimization methods are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-described retrieval enhancement generation optimization methods.

[0036] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of any one of the retrieval enhancement generation optimization methods described above.

[0037] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0038] The present invention provides a method, system, device, product, and medium for optimizing search-enhanced generation. By utilizing hybrid search, the advantages of dense and sparse search are combined, extracting information from documents from both semantic and keyword perspectives. This effectively improves search quality and reduces the computational resources consumed by the search. Furthermore, by using a local language model, the present invention automatically filters documents with low key information density scores, avoiding the degradation of search-enhanced generation quality caused by screening. This allows search-enhanced generation to produce more efficient and accurate prompts, improving its reliability.

[0039] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flow chart of a retrieval enhancement generation optimization method provided by the present invention.

[0042] Figure 2 It is a structural diagram of a retrieval enhancement generation optimization system provided by the present invention.

[0043] Figure 3 It is a structural diagram of a retrieval enhancement generation optimization device provided by the present invention.

[0044] Reference numerals:

[0045] 100, document retrieval module; 200, first candidate document module; 300, second candidate document module; 400, coarse document screening module; 500, vocabulary key information density score module; 600, target document module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0047] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0048] In the description of the embodiments of the present invention, it should be noted that, in the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are mutually inconsistent.

[0049] The following combination Figures 1 to 3 Describing embodiments of the present invention:

[0050] Figure 1 A flow chart of a retrieval enhancement generation optimization method provided by the present invention first requires obtaining a retrieval document, then performing a first hybrid retrieval and a second hybrid retrieval, which further includes: constructing a hybrid retriever, performing a first hybrid retrieval to obtain a first candidate document, generating a hypothesis document, and performing a second hybrid retrieval to obtain a second candidate document; subsequently screening to obtain a coarsely screened document, which further includes: splitting the second candidate document, calculating the information relevance of the second candidate document, calculating a first cross entropy loss, calculating a second cross entropy loss, screening the second candidate document to obtain a coarsely screened document; and finally, finely screening to obtain a target document.

[0051] The present invention provides a retrieval enhancement generation optimization method, comprising:

[0052] S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0053] Furthermore, the purpose of this stage is to organize the original documents, convert them into a unified format and remove unrecognizable characters in the original documents. In addition, the original documents are indexed to facilitate subsequent mixed retrieval of the documents. Step S1 further includes:

[0054] S11: Acquire the original document, determine the target format, remove useless characters in the original document, and divide and convert the original document into a format according to the target format to obtain an intermediate document;

[0055] S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, and fuse the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

[0056] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0057] First, the original document is obtained and the target format is determined. In this embodiment, the target format selected is jsonl format. jsonl format means that each line in the file is a complete json string, which has a relatively clear structure and is easy for the program to read and process. Next, the document is filtered and cleared of characters that cannot be correctly recognized. At the same time, the original document is divided and divided into blocks. The resulting document blocks will serve as the basic units of retrieval. In addition, because some document blocks contain a lot of content, it is not appropriate to search the entire content. Therefore, they need to be further divided into fields to distinguish content with different meanings or functions. At this point, the content of a certain field can be selected for retrieval through manual or other means. Then, the format of the cleaned and divided original document is converted to the target format to obtain the intermediate document.

[0058] Next, a bag-of-words model is obtained. This model can be used to place all words in the intermediate document into a set and obtain the word frequency vector. Here, the word frequency vector can represent the frequency of the word in the document block of the intermediate document. Using the word frequency vector, an inverted index with a word-document structure is constructed to query the distribution of words in the intermediate document, thus completing the construction of the sparse search index. Next, a dense search encoder is obtained. The dense search encoder can encode a portion of the content in the intermediate document into a vector. Then, a variety of index calculation methods can be selected to construct the dense search index, such as simple indexing and product quantization indexing. Finally, the sparse search index and dense search index are merged with the intermediate document to obtain the search document.

[0059] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0060] Furthermore, the purpose of this stage is to reduce the number of documents by using a hybrid search engine, so as to perform a first hybrid search on the search documents according to the query statement, and obtain the first candidate document that is more suitable for the query statement. Among them, step S2 further includes:

[0061] S21: Acquire a sparse searcher and a dense searcher, and construct the hybrid searcher by using the sparse searcher and the dense searcher;

[0062] S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

[0063] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0064] First, we obtain a sparse retriever and a dense retriever, and then construct a hybrid retriever by using the sparse retriever and the dense retriever. Indicates that the search is performed through the hybrid search engine. Then, the query statement q is obtained. Here, the query statement is the query statement entered by the user when performing a search, and its common form is a keyword. The query statement q is input into the hybrid search engine, and the hybrid search engine performs the first hybrid search on the search document D, and retains the previous search results. The first candidate document can be obtained by :

[0065]

[0066] S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0067] Furthermore, the purpose of this stage is to generate hypothetical documents through the large language model, expand the query statement through the hypothetical documents to enhance the query effect of the query statement, and perform a secondary hybrid search on the retrieved documents through the hybrid search engine to further obtain a second candidate document. Among them, step S3 further includes:

[0068] S31: Acquire a task instruction and a large language model, input the task instruction, the query statement, and the first candidate document into the large language model to generate the hypothetical document;

[0069] S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

[0070] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0071] First, obtain the task instruction I and the large language model, () indicates retrieval through a large language model. The task instruction is an instruction indicating the type of hypothetical document generated by the large language model. For example, if the query is "Qinghai-Tibet Railway" and the task instruction is "generate a popular science article", the large language model can generate a popular science article about the Qinghai-Tibet Railway. , Task instruction I and query statement q are combined into prompt statement, and the previous After getting the result, input the prompt sentence into the large language model to get the hypothesis document :

[0072]

[0073] Then, the query statement is expanded by assuming documents to obtain the sparse retrieval query statement :

[0074]

[0075] in, Indicates string concatenation, Indicates that the query statement copy This is to ensure that the query statement maintains a certain proportion in the sparse search query statement to ensure that it does not deviate from the query requirements.

[0076] At the same time, the query statement is expanded by the hypothetical document to obtain the dense retrieval query statement: the query statement and the hypothetical document are embedded in the text, and the query statement vectors can be obtained respectively. and assume document vector , then the query sentence vector is combined with Treat them as one and calculate the average to get the dense search query statement :

[0077]

[0078] Then, the sparse search query statement and intensive search queries Enter the hybrid retriever, To use the hybrid retriever for retrieval, the hybrid retriever performs a secondary hybrid retrieval on the retrieved documents and retains the previous results, you can get the initial second candidate document :

[0079]

[0080] Finally, the initial second candidate document is concatenated with the query statement and task instruction to obtain the second candidate document. :

[0081] S4: Obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document using the local language model. Calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0082] Furthermore, the purpose of this stage is to calculate the information relevance of the second candidate document using the local language model, and to calculate the first cross entropy loss and the second cross entropy loss respectively using the first regularization statement and the second regularization statement, and finally to screen the second candidate document using the first cross entropy loss and the second cross entropy loss to obtain a coarsely screened document. Step S4 further includes:

[0083] S41: Decomposing the second candidate document to obtain a document sequence, a statement sequence, and an instruction sequence;

[0084] S42: Obtain a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence;

[0085] S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence;

[0086] S44: Determine a coarse screening compression rate, calculate a coarse screening word quantity limit based on the coarse screening compression rate, calculate a document key information density score based on the information relevance, the first cross entropy loss, and the second cross entropy loss, screen the second candidate document based on the document key information density score and the coarse screening word quantity limit to obtain the coarse screening document.

[0087] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0088] First, the content of the second candidate document is disassembled to obtain multiple document sequences, one of which is used as Indicates that you can also get a sequence of statements and instruction sequence Then, the local language model is obtained, the document sequences are input into the local language model, and the corresponding predicted sentence sequence is generated for each document sequence. The cross entropy loss between the predicted sentence sequence and the sentence sequence is calculated. The information relevance of a single document sequence can be obtained:

[0089]

[0090] in, The first statement in the sequence words, The local language model predicts the first Word predictions are The probability of N is the number of words in the sentence sequence, and The larger the value, the higher the correlation between the document sequence and the sentence sequence. In a local language model, when the input is determined, the local language model outputs words according to probability. For example, when the input is A, the probability of outputting B is 90%, but there is also a 1% probability of outputting C, a 2% probability of outputting D, and so on.

[0091] Then obtain the first regularized statement and the second regularized statement. In this embodiment, the first regularized statement The second regularization statement is "We can get the answer to this question in the given documents." "We cannot get the answer to this question in the given documents." Input the single document sequence and sentence sequence into the local language model, generate the regularized prediction sentence, and calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence :

[0092]

[0093] in, is the jth word in the first regularized sentence, For the local language model, the jth word of the regularized prediction sentence is predicted based on a single document sequence and sentence sequence as O is the number of words in the regularized prediction sentence, and the number of words in the regularized prediction sentence is the same as the number of words in the first regularized sentence and the second regularized sentence. The larger the first cross entropy loss, the higher the possibility that the document sequence solves the problem. Then calculate the second cross entropy loss of the regularized prediction sentence and the second regularized sentence :

[0094]

[0095] in, is the jth word in the second regularized sentence, For the local language model, the jth word of the regularized prediction sentence is predicted based on a single document sequence and sentence sequence as The higher the second cross-entropy loss, the lower the likelihood that the document sequence solves the problem. The cross-entropy loss of the regularized statement is calculated because document sequences highly relevant to the question do not always solve the user's question. In addition to considering the correlation between document and statement sequences, the capabilities of the local language model can be more fully generalized, using both statement and document sequences as inputs to the local language model. The local language model's output reflects the local language model's judgment on whether the document sequence solves the problem. To quantify the local language model's positive or negative tendency toward a document sequence solving a problem, the cross-entropy loss is calculated between the document and question sequences and two predefined regularized statement sequences. When the local language model determines that the document sequence solves the problem, it is more likely to output the first regularized statement; otherwise, it is more likely to output the second regularized statement. Subtracting the first and second cross-entropy losses can reduce noise.

[0096] The coarse filter compression ratio is then determined based on experience and the expected document size of the coarse filter documents. , calculate the limit of the number of coarse filter words by the coarse filter compression ratio :

[0097]

[0098] Among them, L is the total number of words in multiple document sequences, statement sequences, and instruction sequences. is the number of words in the instruction sequence, is the number of words in the statement sequence; here, the number of words in the statement sequence and instruction sequence is almost negligible compared to the number of words in multiple document sequences. Therefore, although the coarse screening is performed on multiple document sequences, L can be directly used with the coarse screening compression rate. Multiply and subtract the number of words in the statement sequence and instruction sequence.

[0099] Then, the document key information density score of the document sequence is calculated by information relevance, first cross entropy loss and second cross entropy loss :

[0100]

[0101] Arrange all document sequences in descending order according to the document key information density score and calculate the word length of each document sequence, so that the word length of the document sequence can be accumulated from the front to the back, and the document sequence is marked at the same time until the accumulated value reaches the limit of the number of coarse screening words. The unmarked document sequences at this time are eliminated to complete the screening of the second candidate document, and the retained document sequences are used as coarse screening documents.

[0102] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words;

[0103] Furthermore, the purpose of this stage is to obtain segmented documents and calculate the key information density scores of the segmented documents, thereby obtaining paragraph key information density scores. First, the coarse-screened documents are spliced ​​with the instruction sequence and the statement sequence to complete the reorganization of the coarse-screened documents and obtain reorganized documents. Then, the number of words contained in each paragraph, that is, the paragraph length, is determined, and the repeated documents are segmented according to the paragraph length to obtain multiple segmented documents. Finally, each word in each segmented document is input into the local language model respectively, and the document key information density score is calculated according to the method for calculating the document key information density score of the document sequence in steps S42 to S45 described above, so as to obtain the lexical key information density score of each word in each segmented document.

[0104] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0105] Furthermore, the purpose of this stage is to filter the segmented documents by fine screening threshold and paragraph key information density score to obtain the target document. Specifically, first, it is necessary to calculate the first Fine-filter compression ratio of a document sequence ;

[0106]

[0107] in, () means taking the maximum value of the two values ​​in the brackets. () means taking the minimum value of the two values ​​in the brackets. Indicates the number of words after fine filtering, Indicates the first The word length of the document sequence, is the total number of document sequences in the coarse filter documents.

[0108] Then the compression ratio of each document sequence is fine-tuned The original The fine-filtering threshold is determined based on empirically assigned values ​​for the segmented documents corresponding to the content of each document sequence. If the content of a segmented document includes content from two adjacent document sequences, the fine-filtering threshold is determined based on the average fine-filtering compression ratio of the two adjacent document sequences. Finally, words in each segmented document with a lexical key information density score below the fine-filtering threshold are eliminated, and the remaining content is used as the target document.

[0109] The present invention also verifies the effectiveness of a retrieval enhancement generation optimization method. First, a document retrieval task is performed on the TREC DL19 and TREC DL20 datasets. This task requires retrieving up to 1,000 relevant documents from a complete document collection and sorting them according to the degree of relevance, thereby fully simulating the end-to-end retrieval process in actual applications. This experiment uses Map, nDCG@10, and R@1k as evaluation indicators on TREC DL19 and TREC DL20. Among them, Map is the category average precision, Ndcg@10 represents the result of taking the top 10 relevant documents when calculating the normalized discounted cumulative return, and R@1k represents the recall rate of the top 1,000 relevant documents; the results of the document retrieval task on the TREC DL19 and TREC DL20 datasets are shown in Table 1:

[0110] Table 1 Document retrieval task results on TREC DL19 and TREC DL20 datasets

[0111]

[0112] Among them, BM25, Contriever, HyDE, LameR, DPR, ANCE, SimLM, Q2D, etc. are all retrieval methods. The bold value indicates that the value is the highest value in the column.

[0113] It can be seen from Table 1 that, when the hybrid weights are not adjusted for the dataset and the parameters are fixed, compared with the baseline model of zero samples, the method of the present invention achieves the best performance in most indicators when using RRF, i.e., reciprocal ranking fusion, and also achieves excellent performance on TREC DL20 when using WSF, i.e., weighted score fusion. Here, reciprocal ranking fusion and weighted score fusion are methods for hybrid retrievers to perform retrieval. Compared with few samples and fine-tuning models, the method of the present invention also achieves competitive performance. This reflects the effectiveness of the approach of the present invention. In addition, it can be seen from the results that different hybrid algorithms have different performance on different datasets, and choosing the appropriate algorithm and parameter settings has a great impact on the retrieval performance.

[0114] In addition, the present invention has been validated in a RAG (Retrieval Augmented Generation) scenario: an open-domain question answering task is performed on a natural question dataset. This benchmark organizes the dataset into an open-domain question answering task similar to the RAG scenario: each example contains a question, one document containing the answer to the question as the correct answer document, and 19 documents that do not contain the answer to the question but are related to the question. The document containing the answer to the question can be fixed at the 1st, 5th, 10th, 15th, and 20th positions in the prompt to examine the large language model's ability to generate key information in different positions.

[0115] Here, exact substring matching is used to determine the generated results for each example, with accuracy used as the evaluation metric. The correct answer document is placed at different positions, i.e., the 1st, 5th, 10th, 15th, and 20th document in the document, to form five independent test data samples, namely the 1st, 5th, 10th, 15th, and 20th. The accuracy of the generated text is then calculated. The reordering strategy is used to determine the probability that the correct answer document will be ranked first after reordering the documents. Furthermore, the output accuracy of the large language model without any reference documents is used as the lower bound of the model's performance (i.e., no search documents). The output accuracy when only the correct answer document is provided, without any noise documents, is used as the theoretical upper bound of the compression method's performance (i.e., the optimal document). Each compression method is tested with compression ratios of 2x and 4x. The compression ratio is the multiple by which the hint length (i.e., the total number of words in the document) is compressed. For example, if the total number of words in a document is 1000, a compression ratio of 2 results in a total of 500 words in the compressed document. Table 2 shows the experimental results of our method and other research methods on a natural problem benchmark. R@5 represents the recall rate of the correct document among the top five relevant documents after reranking, and R@10 represents the recall rate of the correct document among the top ten relevant documents after reranking. Bold values ​​indicate the highest value in that column.

[0116] Table 2 Experimental results of the present invention and other research methods on natural problem benchmarks

[0117]

[0118] Experiments show that the accuracy of the present invention exceeds that of existing methods at compression rates of 2x and 4x, and when the correct documents are in different positions. The present invention also has a higher probability of placing the correct answer document first during rearrangement.

[0119] The following describes a retrieval enhancement generation optimization system provided by the present invention. The retrieval enhancement generation optimization system described below and the retrieval enhancement generation optimization method described above can refer to each other.

[0120] Figure 2 The following is a structural diagram of a retrieval enhancement generation optimization system: Figure 2 As shown, the method for performing the above-mentioned retrieval enhancement generation optimization method includes:

[0121] The document retrieval module 100 is used to obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0122] The first candidate document module 200 is used to construct a hybrid search engine, and to perform a first hybrid search on the search document through the hybrid search engine to obtain a first candidate document;

[0123] The second candidate document module 300 is configured to generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document.

[0124] A coarse document screening module 400 is configured to obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, calculate the information relevance of the second candidate document using the local language model, and sequentially calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; and screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0125] The vocabulary key information density score module 500 is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words;

[0126] The target document module 600 is used to determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0127] on the other hand, Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a search enhancement generation optimization method, which includes:

[0128] S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0129] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0130] S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0131] S4: Obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document using the local language model. Calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0132] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words;

[0133] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0134] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0135] In another aspect, the present invention further provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the computer is capable of performing a search enhancement generation optimization method provided by the above methods, the method comprising:

[0136] S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0137] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0138] S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0139] S4: Obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document using the local language model. Calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0140] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words;

[0141] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0142] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a search enhancement generation optimization method provided by the above methods, the method comprising:

[0143] S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents;

[0144] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document;

[0145] S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document;

[0146] S4: Obtain a first regularization statement, a second regularization statement, and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document using the local language model. Calculate a first cross-entropy loss and a second cross-entropy loss using the first regularization statement and the second regularization statement, respectively; screen the second candidate document using the information relevance, the first cross-entropy loss, and the second cross-entropy loss to obtain a coarsely screened document.

[0147] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words;

[0148] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0150] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A search enhancement generation optimization method, characterized in that: include: S1: Obtain original documents, organize and index the original documents, and obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularization statement, a second regularization statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document using the local language model, and calculating a first cross entropy loss and a second cross entropy loss using the first regularization statement and the second regularization statement, respectively; screening the second candidate document using the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; step S4 further includes: S41: Decomposing the second candidate document to obtain a document sequence, a statement sequence, and an instruction sequence; S42: Obtain a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence; S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence; S44: determining a coarse screening compression rate, calculating a coarse screening word quantity limit based on the coarse screening compression rate, calculating a document key information density score based on the information relevance, the first cross entropy loss, and the second cross entropy loss, screening the second candidate documents based on the document key information density score and the coarse screening word quantity limit to obtain the coarse screening document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

2. A search enhancement generation optimization method according to claim 1, characterized in that: Step S1 further comprises: S11: Acquire the original document, determine the target format, remove useless characters in the original document, and divide and convert the original document into a format according to the target format to obtain an intermediate document; S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, and fuse the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

3. The search enhancement generation optimization method according to claim 1, characterized in that: Step S2 further comprises: S21: Acquire a sparse searcher and a dense searcher, and construct the hybrid searcher by using the sparse searcher and the dense searcher; S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

4. The search enhancement generation optimization method according to claim 1, characterized in that: Step S3 further comprises: S31: Acquire a task instruction and a large language model, input the task instruction, the query statement, and the first candidate document into the large language model to generate the hypothetical document; S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

5. The search enhancement generation optimization method according to claim 1, characterized in that: In step S5, the coarsely screened documents are reorganized to obtain reorganized documents, paragraph lengths are determined, and the reorganized documents are segmented according to the paragraph lengths to obtain segmented documents.

6. A search enhancement generation optimization system, used to execute a search enhancement generation optimization method according to any one of claims 1 to 5, characterized in that: include: Retrieval document module: used to obtain original documents, organize and index the original documents, and obtain retrieval documents; First candidate document module: used to construct a hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; A second candidate document module is configured to generate a hypothetical document using the first candidate document, and perform a secondary hybrid search on the search document using the hybrid retriever and the hypothetical document to obtain a second candidate document; A coarse document screening module is configured to obtain a first regularized statement, a second regularized statement, and a local language model, split the second candidate document, calculate the information relevance of the second candidate document using the local language model, and sequentially calculate a first cross entropy loss and a second cross entropy loss using the first regularized statement and the second regularized statement; screen the second candidate document using the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; A vocabulary key information density score module is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words; Target document module: used to determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the retrieval enhancement generation optimization method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the retrieval enhancement generation optimization method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can execute the steps of the search enhancement generation optimization method according to any one of claims 1 to 5.

Images