Retrieval enhancement generation optimization method, system, equipment, product and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a retrieval enhancement generation optimization method, system and device, a product and a medium, and the method comprises the steps: obtaining an original document, and carrying out the arrangement and index construction of the original document to obtain a retrieval document; constructing a hybrid retriever, and obtaining a first candidate document through the hybrid retriever; generating a hypothesis document through the first candidate document, and performing secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain a second candidate document; calculating the information association degree of the second candidate document; screening the second candidate document to obtain a coarse screening document; recombining and segmenting the roughly screened document to obtain segmented documents, and calculating key information density scores of words in the segmented documents to obtain vocabulary key information density scores of the words; and screening the segmented documents according to a fine screening threshold value and the vocabulary key information density score to obtain a target document. According to the method, the reliability of retrieval enhancement and the retrieval quality are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation optimization method, system, device, product and medium. Background Art

[0002] Large language models excel in reasoning and have the potential to be widely used in real-world scenarios. However, when users need to obtain specialized responses in specific fields, general large language models often face challenges. Due to the breadth and limitations of their training data, the model may lack professional domain knowledge, resulting in "hallucinations" in its output content. That is, although the responses generated by the model are linguistically continuous and fluent, they frequently make factual errors and cannot guarantee the reliability of the response content.

[0003] Retrieval-enhanced generation is a method that combines external knowledge base information retrieval technology to inject external knowledge into model prompt sentences to enhance the performance of large language models. It solves the limitations of large language models in knowledge-intensive tasks, such as answering complex questions and generating detailed explanations, increases the transparency of the model reasoning process, and alleviates the illusion phenomenon of model generation. The introduction of retrieval documents has greatly increased the number of prompt words input to large language models, but this also brings the following difficulties to large language models: large language models have a fixed context window, so the prompt sentences need to be processed to adapt to the context window restrictions, and simple processing methods may lead to the loss of key information; long context prompts will increase the cost of calling the large language model and increase the time required for response; long context prompts have less key content, and the model is prone to ignore the information in the middle of the prompt sentence, which affects the reliability of the generated results. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a retrieval enhancement generation optimization method, system, device, product and medium, which achieves performance improvement on zero-sample document retrieval tasks through hybrid retrieval, and increases the proportion of key information in the prompt input of a large language model, thereby effectively improving the performance of the large language model.

[0005] The present invention provides a retrieval enhancement generation optimization method, comprising: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0006] According to a retrieval enhancement generation optimization method provided by the present invention, step S1 further comprises: S11: acquiring the original document, determining the target format, removing useless characters in the original document, and dividing and converting the format of the original document according to the target format to obtain an intermediate document; S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, merge the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

[0007] According to a search enhancement generation optimization method provided by the present invention, step S2 further comprises: S21: Obtain a sparse retriever and a dense retriever, and construct the hybrid retriever through the sparse retriever and the dense retriever; S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

[0008] According to a search enhancement generation optimization method provided by the present invention, step S3 further comprises: S31: Acquire a task instruction and a large language model, input the task instruction, the query statement and the first candidate document into the large language model, and generate the hypothetical document; S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

[0009] According to a search enhancement generation optimization method provided by the present invention, step S4 further comprises: S41: Decomposing the second candidate document to obtain a document sequence, a sentence sequence and an instruction sequence; S42: Acquire a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence; S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence; S44: Determine a coarse screening compression rate, calculate a coarse screening word quantity limit according to the coarse screening compression rate, calculate a document key information density score according to the information relevance, the first cross entropy loss, and the second cross entropy loss, screen the second candidate document according to the document key information density score and the coarse screening word quantity limit, and obtain the coarse screening document.

[0010] According to a retrieval enhancement generation optimization method provided by the present invention, in step S5, the coarsely screened document is reorganized to obtain a reorganized document, the paragraph length is determined, and the reorganized document is segmented according to the paragraph length to obtain the segmented document.

[0011] The present invention also provides a retrieval enhancement generation optimization system, comprising: Retrieval document module: used to obtain original documents, organize and index the original documents, and obtain retrieval documents; The first candidate document module is used to construct a hybrid retriever, and to perform a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; A second candidate document module: used to generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; A rough screening document module: used to obtain a first regularized statement, a second regularized statement and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document through the local language model, and respectively calculate a first cross entropy loss and a second cross entropy loss through the first regularized statement and the second regularized statement; the second candidate document is screened through the information relevance, the first cross entropy loss and the second cross entropy loss to obtain a rough screening document; A vocabulary key information density score module is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words; Target document module: used to determine a detailed screening threshold, and screen the segmented documents according to the detailed screening threshold and the vocabulary key information density score to obtain a target document.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a retrieval enhancement generation optimization method as described in any one of the above are implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of a retrieval enhancement generation optimization method as described in any one of the above are implemented.

[0014] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of any of the retrieval enhancement generation optimization methods described above.

[0015] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention provides a search enhancement generation optimization method, system, device, product and medium. By using hybrid search, the advantages of dense search and sparse search are combined, and information in documents is extracted from the two perspectives of semantics and keywords, which effectively improves the search quality and reduces the computing resources consumed by the search. In addition, the present invention uses a local language model to achieve automatic filtering of documents with low key information density scores, and avoids the quality degradation of search enhancement generation caused by screening, so that search enhancement generation can generate more efficient and accurate prompt content, and improves its reliability.

[0016] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a retrieval enhancement generation optimization method provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of a retrieval enhancement generation optimization system provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of a retrieval enhancement generation optimization device provided by the present invention.

[0021] Reference numerals: 100, retrieval document module; 200, first candidate document module; 300, second candidate document module; 400, coarse screening document module; 500, vocabulary key information density score module; 600, target document module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0023] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0024] In the description of the embodiments of the present invention, it should be noted that, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0025] Combine the following Figures 1 to 3 Describing embodiments of the present invention: Figure 1 A flow chart of a retrieval enhancement generation optimization method provided by the present invention, firstly, a retrieval document needs to be obtained, and then a first hybrid retrieval and a second hybrid retrieval are performed, which further includes: constructing a hybrid retriever, performing a first hybrid retrieval to obtain a first candidate document, generating a hypothesis document, and performing a second hybrid retrieval to obtain a second candidate document; then screening to obtain a coarse screening document, which further includes: splitting the second candidate document, calculating the information relevance of the second candidate document, calculating the first cross entropy loss, calculating the second cross entropy loss, screening the second candidate document to obtain a coarse screening document; and finally fine screening to obtain a target document.

[0026] The present invention provides a retrieval enhancement generation optimization method, comprising: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; Furthermore, the purpose of this stage is to organize the original documents, convert the original documents into a unified format and remove unrecognizable characters in the original documents. In addition, the original documents are indexed to facilitate subsequent mixed retrieval of the documents. Step S1 further includes: S11: acquiring the original document, determining the target format, removing useless characters in the original document, and dividing and converting the format of the original document according to the target format to obtain an intermediate document; S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, merge the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

[0027] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, obtain the original document and determine the target format. In this embodiment, the selected target format is jsonl format. The jsonl format means that each line in the file is a complete json string, and its structure is relatively clear, which has the advantage of being easy for the program to read and process. Then, filter and remove the characters that cannot be correctly recognized in the document, and divide and block the original document at the same time, and the document blocks obtained by division will be used as the basic unit of retrieval. In addition, since some document blocks include more content, it is not appropriate to search all the content, so it is necessary to divide them into fields to distinguish the content with different meanings or functions. At this time, the content of a certain field can be selected for retrieval by manual or other means. Then, the format of the cleaned and divided original document is converted into the target format to obtain the intermediate document.

[0028] Then, obtain the bag-of-words model. Through the bag-of-words model, all words in the intermediate document can be placed in a set, and the word frequency vector of the word can be obtained. Here, the word frequency vector can represent the frequency of the word in the document block of the intermediate document. Through the word frequency vector, an inverted index with a structure of word-document is constructed to query the distribution of words in the intermediate document, and the construction of the sparse retrieval index can be completed. Next, obtain the dense retrieval encoder. The dense retrieval encoder can encode part of the content in the intermediate document into a vector. Then, you can choose a variety of index calculation methods to construct a dense retrieval index, such as a simple index and a product quantization index. Finally, the sparse retrieval index and the dense retrieval index are merged with the intermediate document to obtain the retrieval document.

[0029] S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; Furthermore, the purpose of this stage is to reduce the number of documents by using a hybrid searcher, so as to perform a first hybrid search on the search documents according to the query statement, and obtain the first candidate document that is more suitable for the query statement. Among them, step S2 further includes: S21: Obtain a sparse retriever and a dense retriever, and construct the hybrid retriever through the sparse retriever and the dense retriever; S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

[0030] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, obtain a sparse retriever and a dense retriever, and construct a hybrid retriever through the sparse retriever and the dense retriever. Indicates that the search is performed through the hybrid search engine. Next, the query statement q is obtained. Here, the query statement is the query statement entered by the user when performing a search, and its common form is a keyword. The query statement q is input into the hybrid search engine, and the hybrid search engine performs the first hybrid search on the search document D, and retains the previous search results. The first candidate document can be obtained by : S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; Furthermore, the purpose of this stage is to generate a hypothetical document through a large language model, expand the query statement through the hypothetical document to enhance the query effect of the query statement, and perform a secondary hybrid search on the searched document through a hybrid searcher to further obtain a second candidate document. Among them, step S3 further includes: S31: Acquire a task instruction and a large language model, input the task instruction, the query statement and the first candidate document into the large language model, and generate the hypothetical document; S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

[0031] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, obtain the task instruction I and the large language model. () indicates retrieval through a large language model. The task instruction is an instruction indicating the type of hypothetical document generated by the large language model. For example, if the query is "Qinghai-Tibet Railway" and the task instruction is "generate a popular science article", the large language model can generate a popular science article about the Qinghai-Tibet Railway. , task instruction I and query statement q are combined into a prompt statement, and the previous setting is retained After getting the result, input the prompt sentence into the large language model to get the hypothetical document : Then, the query statement is expanded by assuming documents to obtain a sparse retrieval query statement : in, Indicates string concatenation, Indicates that the query statement copy This is to ensure that the query statement maintains a certain proportion in the sparse search query statement to ensure that it does not deviate from the query requirements.

[0032] At the same time, the query statement is expanded by the hypothetical document to obtain a dense retrieval query statement: the query statement and the hypothetical document are embedded in text, and the query statement vectors can be obtained respectively. and assume that the document vector , and then the query sentence vector is combined with Treat them as one and find the average to get the dense search query statement : Then, the sparse search query statement and intensive search queries Enter the hybrid retriever, To use the hybrid retriever for retrieval, the hybrid retriever performs a secondary hybrid retrieval on the retrieved documents and retains the previous results, you can get the initial second candidate document : Finally, the initial second candidate document is concatenated with the query statement and task instruction to obtain the second candidate document. : S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; Furthermore, the purpose of this stage is to calculate the information relevance of the second candidate document through the local language model, and to calculate the first cross entropy loss and the second cross entropy loss respectively through the first regularization statement and the second regularization statement, and finally to screen the second candidate document through the first cross entropy loss and the second cross entropy loss to obtain a coarsely screened document. Among them, step S4 further includes: S41: Decomposing the second candidate document to obtain a document sequence, a sentence sequence and an instruction sequence; S42: Acquire a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence; S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence; S44: Determine a coarse screening compression rate, calculate a coarse screening word quantity limit according to the coarse screening compression rate, calculate a document key information density score according to the information relevance, the first cross entropy loss, and the second cross entropy loss, screen the second candidate document according to the document key information density score and the coarse screening word quantity limit, and obtain the coarse screening document.

[0033] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the content of the second candidate document is disassembled to obtain multiple document sequences, one of which is used as Indicates that we can also get the statement sequence and instruction sequence Then, the local language model is obtained, the document sequences are input into the local language model respectively, and the corresponding predicted sentence sequence is generated for each document sequence. The cross entropy loss between the predicted sentence sequence and the sentence sequence is calculated. The information relevance of a single document sequence can be obtained: in, The first words, The local language model predicts the first sentence sequence based on a single document sequence The word prediction is The probability of N is the number of words in the sentence sequence, and The larger the value, the higher the correlation between the document sequence and the sentence sequence. In the local language model, when the input is determined, the local language model outputs words according to probability. For example, when the input is A, the probability of outputting B is 90%, but there is also a 1% probability of outputting C, a 2% probability of outputting D, etc.

[0034] Then obtain the first regularized statement and the second regularized statement. In this embodiment, the first regularized statement The second regularization statement is "We can get the answer to this question in the given documents." "We cannot get the answer to this question in the given documents." Input the single document sequence and sentence sequence into the local language model, generate the regularized prediction sentence, and calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence : in, is the jth word in the first regularized sentence, The local language model predicts the jth word of the regularized prediction sentence based on a single document sequence and sentence sequence as O is the number of words in the regularized prediction sentence, and the number of words in the regularized prediction sentence is the same as the number of words in the first regularized sentence and the second regularized sentence. The larger the first cross entropy loss, the higher the possibility that the document sequence solves the problem. Then calculate the second cross entropy loss of the regularized prediction sentence and the second regularized sentence : in, is the jth word in the second regularized sentence, The local language model predicts the jth word of the regularized prediction sentence based on a single document sequence and sentence sequence as The higher the second cross entropy loss, the lower the probability that the document sequence can solve the problem. The cross entropy loss of the regularized sentence is calculated because the document sequence with a high degree of correlation with the problem does not always solve the problem raised by the user. In addition to considering the correlation between the document sequence and the sentence sequence, the ability of the local language model can be more fully generalized and utilized. The sentence sequence and the document sequence are used as the input of the local language model, and the output content of the local language model reflects the judgment of the local language model on whether the document sequence can solve the problem. In order to quantify the positive or negative tendency of the local language model for the document sequence to solve the problem, the cross entropy loss of the document and question sequence and the two pre-defined regularized sentence sequences is calculated. When the local language model judges that the document sequence can solve the problem, the probability of outputting the first regularized sentence is greater, otherwise the probability of outputting the second regularized sentence is greater. Subtracting the first cross entropy loss from the second cross entropy loss can reduce noise.

[0035] The coarse filter compression ratio is then determined based on experience and the expected document size of the coarse filter documents. , calculate the limit of the number of words in the rough screening by the rough screening compression ratio : Where L is the total number of words in multiple document sequences, statement sequences, and instruction sequences. is the number of words in the instruction sequence, is the number of words in the statement sequence; here, the number of words in the statement sequence and instruction sequence is almost negligible compared to the number of words in multiple document sequences. Therefore, although the coarse screening is performed on multiple document sequences, L can be directly used to compare with the coarse screening compression rate. Multiply and subtract the number of words in the statement sequence and instruction sequence.

[0036] Then, the document key information density score of the document sequence is calculated by information relevance, the first cross entropy loss and the second cross entropy loss. : Arrange all document sequences from high to low according to the document key information density score and calculate the word length of each document sequence, so that the word length of the document sequence can be accumulated from front to back, and the document sequence is marked at the same time until the accumulated value reaches the coarse screening word number limit. The unmarked document sequences at this time are eliminated to complete the screening of the second candidate document, and the retained document sequences are used as coarse screening documents.

[0037] S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; Furthermore, the purpose of this stage is to obtain segmented documents and calculate the key information density scores of the segmented documents, thereby obtaining the key information density scores of the paragraphs. First, the coarsely screened documents are spliced ​​with the instruction sequence and the sentence sequence to complete the reorganization of the coarsely screened documents and obtain the reorganized documents. Then, the number of words contained in each paragraph, that is, the paragraph length, is determined, and the repeated documents are segmented according to the paragraph length to obtain multiple segmented documents. Finally, each word in each segmented document is input into the local language model respectively, and the document key information density score is calculated according to the method of calculating the document key information density score of the document sequence in steps S42 to S45 described above, so as to obtain the vocabulary key information density score of each word in each segmented document.

[0038] S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0039] Furthermore, the purpose of this stage is to filter the segmented documents by fine screening threshold and paragraph key information density score to obtain the target document. Specifically, first, it is necessary to calculate the first The fine-scale compression ratio of a document sequence ; in, () means taking the maximum value of the two values ​​in the brackets. () means the minimum value of the two values ​​in the brackets is taken. Indicates the number of words after fine filtering, Indicates the first The word length of the document sequence, is the total number of document sequences in the coarse filter document.

[0040] Then the compression ratio is fine-tuned based on each document sequence. The original The segmented document corresponding to the content of a document sequence determines the fine screening threshold based on empirical assignment. If the content in a segmented document includes the content of two adjacent document sequences, the fine screening threshold is determined based on the average fine screening compression rate of the two adjacent document sequences. Finally, the words in each segmented document whose vocabulary key information density score is lower than the fine screening threshold are eliminated and the remaining content is used as the target document.

[0041] The present invention also verifies the effectiveness of a retrieval enhancement generation optimization method. First, a document retrieval task is performed on the TREC DL19 and TREC DL20 datasets. This task requires retrieving up to 1,000 relevant documents from the complete document collection and sorting them according to the degree of relevance, thereby fully simulating the end-to-end retrieval process in actual applications. This experiment uses Map, nDCG@10, and R@1k as evaluation indicators on TREC DL19 and TREC DL20. Among them, Map is the category average accuracy, Ndcg@10 represents the result of taking the top 10 relevant documents when calculating the normalized discounted cumulative revenue, and R@1k represents the recall rate of the top 1,000 relevant documents; the results of the document retrieval task on the TREC DL19 and TREC DL20 datasets are shown in Table 1: Table 1 Document retrieval task results on TREC DL19 and TREC DL20 datasets

[0042] Among them, BM25, Contriever, HyDE, LameR, DPR, ANCE, SimLM, Q2D, etc. are all retrieval methods, and the bold value indicates that the value is the highest value in this column.

[0043] It can be seen from Table 1 that, when the hybrid weights are not adjusted for the data set and the parameters are fixed, compared with the baseline model of zero samples, the method of the present invention achieves the best performance of most indicators when using RRF, i.e., reciprocal sort fusion, and also achieves excellent performance on TREC DL20 when using WSF, i.e., weighted score fusion. Here, reciprocal sort fusion and weighted score fusion are methods for retrieval by hybrid retrievers. Compared with few samples and fine-tuning models, the method of the present invention also achieves competitive performance. This reflects the effectiveness of the approach of the present invention. In addition, it can be seen from the results that different hybrid algorithms have different performances on different data sets, and choosing the appropriate algorithm and parameter settings has a great impact on the retrieval performance.

[0044] In addition, the present invention has also been validated in the RAG (Retrieval Augmented Generation) scenario: an open-domain question answering task is performed on a natural question dataset. This benchmark organizes the dataset into an open-domain question answering task similar to the RAG scenario: each sample contains a question, 1 document containing the answer to the question as the correct answer document, and 19 documents that do not contain the answer to the question but are related to the question. The document containing the answer to the question can be fixed at the 1st, 5th, 10th, 15th, and 20th positions in the prompt to check the generation ability of the large language model for key information appearing in different positions.

[0045] Here, substring exact matching is used to judge the generation result of each sample, and the accuracy is used as the evaluation index. The correct answer document is placed in different positions, that is, it becomes the 1st, 5th, 10th, 15th, and 20th document in the document to form five independent test data samples, which are 1st, 5th, 10th, 15th, and 20th respectively, and the accuracy of the text generated by them is calculated. The rearrangement is the probability that the correct answer document is in the first place after the documents are reordered using the rearrangement strategy. In addition, the output accuracy of the large language model without any reference document is added as the performance lower limit of the model, that is, no retrieval document, and the output accuracy when only the correct answer document is given and no interference documents are included is used as the theoretical performance upper limit of the compression method, that is, the best document. Each compression method sets a test of 2x and 4x compression ratios. The compression ratio is the multiple of the prompt length, that is, the total number of words in the document. For example, when the total number of words in the document is 1000, when the compression ratio is 2, the total number of words in the compressed document is 500. Table 2 shows the experimental results of the present invention and other research methods on the natural problem benchmark. R@5 represents the recall rate of the correct document among the first 5 relevant documents after re-ranking, and R@10 represents the recall rate of the correct document among the first 10 relevant documents after re-ranking. The bold value indicates that the value is the highest value in the column.

[0046] Table 2 Experimental results of the present invention and other research methods on natural problem benchmarks

[0047] It can be seen from experiments that the accuracy of the present invention exceeds that of the existing methods at compression rates of 2 and 4 and when the correct documents are in different positions, and the probability of the present invention placing the correct answer document in the first place during rearrangement is also higher.

[0048] A retrieval enhancement generation optimization system provided by the present invention is described below. The retrieval enhancement generation optimization system described below and the retrieval enhancement generation optimization method described above can refer to each other.

[0049] Figure 2 The following is a schematic diagram of the structure of a retrieval enhancement generation optimization system. Figure 2 As shown, a method for performing a retrieval enhancement generation optimization method as described above includes: The document retrieval module 100 is used to obtain original documents, organize and index the original documents, and obtain retrieval documents; The first candidate document module 200 is used to construct a hybrid search engine, and to perform a first hybrid search on the search document through the hybrid search engine to obtain a first candidate document; The second candidate document module 300 is used to generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; A rough screening document module 400 is used to obtain a first regularized statement, a second regularized statement and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document through the local language model, and respectively calculate a first cross entropy loss and a second cross entropy loss through the first regularized statement and the second regularized statement; the second candidate document is screened through the information relevance, the first cross entropy loss and the second cross entropy loss to obtain a rough screening document; The vocabulary key information density score module 500 is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words; The target document module 600 is used to determine a detailed screening threshold, and screen the segmented documents according to the detailed screening threshold and the vocabulary key information density score to obtain a target document.

[0050] on the other hand, Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a retrieval enhancement generation optimization method, the method comprising: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0051] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0052] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute a retrieval enhancement generation optimization method provided by the above methods, the method comprising: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0053] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program is implemented to execute a search enhancement generation optimization method provided by the above methods, the method comprising: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A search enhancement generation optimization method, characterized in that: include: S1: Obtaining original documents, organizing and indexing the original documents to obtain retrieval documents; S2: constructing a hybrid retriever, and performing a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; S3: Generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; S4: obtaining a first regularized statement, a second regularized statement, and a local language model, splitting the second candidate document, and calculating the information relevance of the second candidate document through the local language model, and calculating a first cross entropy loss and a second cross entropy loss in sequence through the first regularized statement and the second regularized statement; screening the second candidate document through the information relevance, the first cross entropy loss, and the second cross entropy loss to obtain a coarsely screened document; S5: reorganizing and segmenting the coarsely screened documents to obtain segmented documents, calculating key information density scores of words in the segmented documents, and obtaining lexical key information density scores of the words; S6: Determine a fine screening threshold, and screen the segmented documents according to the fine screening threshold and the vocabulary key information density score to obtain a target document.

2. A search enhancement generation optimization method according to claim 1, characterized in that: Step S1 further comprises: S11: acquiring the original document, determining the target format, removing useless characters in the original document, and dividing and converting the format of the original document according to the target format to obtain an intermediate document; S12: Obtain a bag-of-words model and a dense retrieval encoder, construct a sparse retrieval index of the intermediate document through the bag-of-words model, construct a dense retrieval index of the intermediate document through the dense retrieval encoder, merge the sparse retrieval index and the dense retrieval index with the intermediate document to obtain the retrieval document.

3. A search enhancement generation optimization method according to claim 1, characterized in that: Step S2 further comprises: S21: Obtain a sparse retriever and a dense retriever, and construct the hybrid retriever through the sparse retriever and the dense retriever; S22: Obtain a query statement, input the query statement into the hybrid retriever, and perform a first hybrid search on the search document through the hybrid retriever to obtain the first candidate document.

4. The retrieval enhancement generation optimization method according to claim 1, characterized in that: Step S3 further comprises: S31: Acquire a task instruction and a large language model, input the task instruction, the query statement and the first candidate document into the large language model, and generate the hypothetical document; S32: Expand the query statement through the hypothetical document to obtain a sparse retrieval query statement and a dense retrieval query statement respectively, input the sparse retrieval query statement and the dense retrieval query statement into the hybrid retriever, and perform a secondary hybrid retrieval on the retrieval document through the hybrid retriever to obtain the second candidate document.

5. The retrieval enhancement generation optimization method according to claim 1, characterized in that: Step S4 further comprises: S41: Decomposing the second candidate document to obtain a document sequence, a sentence sequence and an instruction sequence; S42: Acquire a local language model, input the document sequence into the local language model, generate a predicted sentence sequence, and obtain the information relevance of the second candidate document by calculating the cross entropy loss between the predicted sentence sequence and the sentence sequence; S43: Obtain the first regularized sentence and the second regularized sentence, input the document sequence and the sentence sequence into the local language model, generate a regularized prediction sentence, calculate the first cross entropy loss between the regularized prediction sentence and the first regularized sentence, and calculate the second cross entropy loss between the regularized prediction sentence and the second regularized sentence; S44: Determine a coarse screening compression rate, calculate a coarse screening word quantity limit according to the coarse screening compression rate, calculate a document key information density score according to the information relevance, the first cross entropy loss, and the second cross entropy loss, screen the second candidate document according to the document key information density score and the coarse screening word quantity limit, and obtain the coarse screening document.

6. A search enhancement generation optimization method according to claim 1, characterized in that: In step S5, the coarsely screened documents are reorganized to obtain reorganized documents, paragraph lengths are determined, and the reorganized documents are segmented according to the paragraph lengths to obtain segmented documents.

7. A search enhancement generation optimization system, used to execute a search enhancement generation optimization method according to any one of claims 1 to 6, characterized in that: include: Retrieval document module: used to obtain original documents, organize and index the original documents, and obtain retrieval documents; The first candidate document module is used to construct a hybrid retriever, and to perform a first hybrid search on the search document through the hybrid retriever to obtain a first candidate document; A second candidate document module: used to generate a hypothetical document through the first candidate document, and perform a secondary hybrid search on the search document through the hybrid retriever and the hypothetical document to obtain a second candidate document; A rough screening document module: used to obtain a first regularized statement, a second regularized statement and a local language model, split the second candidate document, and calculate the information relevance of the second candidate document through the local language model, and respectively calculate a first cross entropy loss and a second cross entropy loss through the first regularized statement and the second regularized statement; the second candidate document is screened through the information relevance, the first cross entropy loss and the second cross entropy loss to obtain a rough screening document; A vocabulary key information density score module is used to reorganize and segment the coarsely screened documents to obtain segmented documents, calculate the key information density scores of the words in the segmented documents, and obtain the vocabulary key information density scores of the words; Target document module: used to determine a detailed screening threshold, and screen the segmented documents according to the detailed screening threshold and the vocabulary key information density score to obtain a target document.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of a retrieval enhancement generation optimization method as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a retrieval enhancement generation optimization method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can execute the steps of a retrieval enhancement generation optimization method as claimed in any one of claims 1 to 6.

Images