Utility evaluation-based retrieval enhancement generation method and device, equipment and medium

Abstract

The invention relates to the technical field of retrieval enhancement generation, and discloses a retrieval enhancement generation method based on utility evaluation, which comprises the following steps: acquiring a plurality of corpus blocks retrieved according to a question input by a user; performing utility evaluation on each corpus block to obtain a corresponding utility score; wherein the utility score is used for representing the useful degree of the corpus block for answering the question input by the user; sorting the plurality of corpus blocks according to the utility scores of the corpus blocks and the semantic similarity between the corpus blocks and the questions input by the user, and selecting a Top K corpus block from the plurality of sorted corpus blocks by adopting a trained Top K selection model; wherein the Top K corpus blocks are selected from a plurality of sorted corpus blocks of the questions input by the user by adopting a Top K algorithm, and the semantic similarity between the selected Top K corpus blocks and the standard answers of the questions is the highest; and answering the question input by the user through the Top K corpus block.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval enhancement generation, and in particular to a retrieval enhancement generation method, device, equipment and medium based on utility evaluation. Background Art

[0002] In recent years, due to the disadvantage that large language models do not have real-time knowledge, retrieval enhancement generation technology has developed rapidly. Current retrieval enhancement generation methods all select corpus blocks related to user input questions from the database through semantic similarity as context references for large language models.

[0003] However, the corpus blocks obtained by the retrieval method based on semantic similarity may not be closely related to the user input question, and the corpus blocks contain a lot of noise. Summary of the Invention

[0004] The purpose of the present invention is to provide a retrieval enhancement generation method, device, equipment and medium based on utility evaluation, which can solve the technical problem that the corpus blocks obtained by the retrieval method based on semantic similarity may have little relevance to the user input question and contain a lot of noise.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for generating retrieval enhancement based on utility evaluation, comprising the following steps: Obtain multiple corpus blocks retrieved based on the user input question; Performing a utility evaluation on each corpus chunk to obtain a corresponding utility score, wherein the utility score is used to represent the usefulness of the corpus chunk in answering the user input question; Sort multiple chunks based on their utility scores and the semantic similarity between the chunks and the user's input question, and use the trained Top K selection model to select the Top K chunks from the sorted chunks. The Top K selection model is trained by selecting the optimal Top K chunks from a variety of Top K chunks corresponding to each user-input question based on the standard answers to the questions. Each Top K chunk is selected using the Top K algorithm from the sorted chunks corresponding to the user-input question. The optimal Top K chunk has the highest semantic similarity with the standard answer. Answer user input questions using Top K corpus chunks.

[0006] Optionally, performing utility evaluation on each corpus block to obtain a corresponding utility score includes: The utility evaluation model trained in the following way is used to evaluate the utility of each corpus block and obtain the corresponding utility score: The actual large language model is trained using utility scores obtained by evaluating a corpus of several user input questions using a standard large language model and utility scores obtained by evaluating a corpus of several user input questions using an actual large language model; Among them, the number of parameters of the actual large language model is smaller than that of the standard large language model.

[0007] Optionally, the training steps of the Top K selection model include: Obtain multiple user input questions and the standard answer corresponding to each user input question; Select a variety of different Top K corpus blocks corresponding to each user input question; Input multiple different Top K corpora into the standard large language model respectively, so that the standard large language model can answer the user input questions based on each Top K corpus; Obtain the answer of the standard large language model based on each Top K corpus block, and calculate the information entropy between the answer of the standard large language model and the standard answer; Based on the information entropy between the answer of the standard large language model and the standard answer, for each user input question, the optimal Top K corpus chunks are determined from the corresponding multiple Top K corpus chunks; The Top K selection model is trained using the optimal Top K corpus blocks corresponding to multiple user input questions.

[0008] Optionally, calculating the information entropy between the answer result of the standard large language model and the standard answer includes: The answer results of the standard large language model and the standard answer are segmented separately to ensure that the answer results correspond to the segmented words in the standard answer one by one; Count the frequency of each pair of participles appearing simultaneously in the answer and the standard answer, and divide the frequency of each pair of participles appearing simultaneously by the total number of participle pairs in the answer and the standard answer to obtain the probability of each pair of participles appearing simultaneously. The information entropy between the answer result and the standard answer is obtained using the following formula: ; In the formula, X represents the answer result, Y represents the standard answer, represents the information entropy between X and Y, m and n are the number of possible values ​​of X and Y respectively, Indicates that the value of X is And the value of Y is The joint probability of , b is a preset constant.

[0009] Optionally, the training step of the utility evaluation model includes: Get several user input questions; Obtain multiple corpus chunks retrieved based on each user input question; Input each input question and each corresponding corpus chunk into the standard large language model to evaluate the utility score of each corpus chunk through the standard large language model; Input each input question and each corresponding corpus chunk into the actual large language model, and evaluate the utility score of each corpus chunk through the actual large language model; The actual large language model training is supervised according to the loss function used to constrain the distance between the utility score corresponding to the standard large language model and the utility score corresponding to the actual large language model.

[0010] Optionally, answering the user input question using the Top K corpus includes: A preset large language model is used to summarize the information in the Top K corpus blocks, and the information in the summarized Top K corpus blocks is used as a context reference for the preset large language model to answer user input questions through the preset large language model.

[0011] An embodiment of the present invention further provides a retrieval enhancement generation device based on utility evaluation, comprising: A corpus retrieval module is used to obtain multiple corpus blocks retrieved according to the user input question; A utility evaluation module is used to evaluate the utility of each corpus chunk and obtain a corresponding utility score, wherein the utility score is used to represent the usefulness of the corpus chunk in answering the user input question; The corpus chunk selection module is used to sort multiple corpora based on their utility scores and the semantic similarity between the corpora and the user input question, and select the top K corpora from the sorted multiple corpora using the trained Top K selection model; The Top K selection model is trained by selecting the optimal Top K chunks from a variety of Top K chunks corresponding to each user-input question based on the standard answers to the questions. Each Top K chunk is selected using the Top K algorithm from the sorted chunks corresponding to the user-input question. The optimal Top K chunk has the highest semantic similarity with the standard answer. The question answering module is used to answer user input questions using the Top K corpus.

[0012] An embodiment of the present invention also provides a computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned utility evaluation-based retrieval enhancement generation method.

[0013] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned utility evaluation-based retrieval enhancement generation method.

[0014] The utility evaluation-based search enhancement generation method provided by the present invention has at least the following beneficial effects: Faced with multiple corpora retrieved based on user input questions, we first conduct utility evaluation on each of them to obtain a utility score for each corpus that can characterize the usefulness of the corpus in answering the user input question. Then, based on the utility score of the corpus and the semantic similarity between the corpus and the user input question, we sort the multiple corpora to select the TopK corpora. The selected TopK corpora have the highest semantic similarity with the standard answer, which is equivalent to screening the retrieved corpora according to the answer to the user input question, so that the corpus finally obtained by the retrieval enhancement generation method is more closely related to the user input question and has less noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0016] Figure 1 is a flowchart of a retrieval enhancement generation method based on utility evaluation according to an embodiment of the present invention; Figure 2 3 is a schematic diagram of a retrieval enhancement generation method based on utility evaluation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in the embodiments of the present invention, many technical details are provided to enable the reader to better understand the present invention. However, even without these technical details and the various changes and modifications based on the following embodiments, the technical solutions claimed in the present invention can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with each other and referenced to each other under the premise that there is no contradiction.

[0018] An embodiment of the present invention relates to a retrieval enhancement generation method based on utility evaluation. The implementation details of the retrieval enhancement generation method based on utility evaluation of this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding and is not necessary for implementing this solution.

[0019] The specific process of the utility evaluation-based retrieval enhancement generation method of this embodiment can be as follows: Figure 1 Shown, including: Step 101: Acquire multiple corpus blocks retrieved according to a question input by a user.

[0020] Specifically, the user inputs questions to retrieve multiple related corpus blocks in a pre-established database.

[0021] Step 102 : perform utility evaluation on each corpus chunk to obtain a corresponding utility score. The utility score is used to represent the usefulness of the corpus chunk in answering the question input by the user.

[0022] In a specific implementation, the utility evaluation model trained in the following manner is used to perform utility evaluation on each corpus to obtain a corresponding utility score: the utility scores obtained by evaluating several corpora of user input questions using a standard large language model, and the utility scores obtained by evaluating several corpora of user input questions using an actual large language model, are used to train the actual large language model.

[0023] The specific training steps of the utility evaluation model are as follows: (1) Obtaining several user input questions; the user input questions can be real questions or synthetic questions. Synthetic questions are new questions generated by combining or modifying existing questions. These questions are based on variants of real questions and are formed by adding, deleting or replacing certain elements; (2) Obtaining multiple corpus blocks retrieved based on each user input question, that is, using the obtained multiple user input questions to retrieve multiple corpus blocks related to each user input question in a pre-established database; (3) Each input question and each corresponding corpus chunk is input into the standard large language model to evaluate the utility score of each corpus chunk through the standard large language model; wherein, the standard large language model is a large language model with good performance deployed locally, and the utility score of each corpus chunk in answering the user's question is evaluated by designing appropriate prompt words; (4) Input each input question and each corresponding corpus chunk into the actual large language model, and evaluate the utility score of each corpus chunk through the actual large language model; wherein the actual large language model is a large language model with a small number of parameters, and the utility score is the utility score that needs to be optimized; (5) Supervise the training of the actual large language model based on any loss function that can constrain the distance between the utility score corresponding to the standard large language model and the utility score corresponding to the actual large language model; wherein, the loss function includes but is not limited to the following: cross entropy loss function, L1 distance loss function, relative entropy (Kullback-Leibler Divergence, KL divergence) loss function, Jensen-Shannon distance (JS divergence) loss function, etc.

[0024] Step 103: sort the multiple corpora based on their utility scores and the semantic similarity between the corpora and the user input questions, and select Top K corpora from the sorted multiple corpora using a trained Top K selection model. The Top K selection model is trained by selecting the optimal Top K corpora from a plurality of different Top K corpora corresponding to each user input question based on the standard answers to the multiple user input questions. Each Top K corpus is selected from the sorted multiple corpora corresponding to the user input question using the Top K algorithm, and the optimal Top K corpus has the highest semantic similarity with the standard answer.

[0025] In the specific implementation, the training steps of the Top K selection model are as follows: (1) Construct multiple user input questions and the standard answer corresponding to each user input question; (2) Selecting a plurality of different Top K corpora corresponding to each user input question, the selection method is to use the Top K algorithm to select from the plurality of sorted corpora corresponding to the user input question; (3) Inputting a variety of different Top K corpora into the standard large language model, so that the standard large language model can answer the user input questions based on each Top K corpus; (4) Obtain the answer results of the standard large language model based on each Top K corpus block, and calculate the information entropy between the answer results of the standard large language model and the standard answer; (5) Based on the information entropy between the answer result of the standard large language model and the standard answer, for each user input question, the optimal Top K corpus block is determined from the corresponding multiple different Top K corpus blocks; (6) Using the optimal Top K corpora corresponding to multiple user input questions, a Top K selection model is trained to adaptively select the optimal Top K corpora from the sorted multiple corpora through the Top K selection model.

[0026] The information entropy between the answer of the standard large language model and the standard answer is calculated by the following steps: (1) Segment the answer results of the standard large language model and the standard answer separately so that the answer results correspond to the segmented words in the standard answer one by one; (2) Count the frequency of each pair of segmentation words appearing simultaneously in the answer result and the standard answer (one segmentation word in each pair comes from the answer result and the other comes from the standard answer), and divide the frequency of each segmentation word pair appearing simultaneously by the total number of segmentation word pairs in the answer result and the standard answer to obtain the probability of each segmentation word pair appearing simultaneously; (3) The information entropy between the answer result and the standard answer is obtained using the following formula: ; In the formula, X represents the answer result, Y represents the standard answer, represents the information entropy between X and Y, m and n are the number of possible values ​​of X and Y respectively, Indicates that the value of X is And the value of Y is The joint probability of , b is a preset constant.

[0027] Step 104: answer the user input question using the Top K corpus.

[0028] Specifically, a preset large language model is used to summarize the information in the Top K corpus blocks, and the information in the summarized Top K corpus blocks is used as a context reference for the preset large language model to answer user input questions through the preset large language model.

[0029] When using the preset large language model to summarize the information in the corpus, examples of prompt words include but are not limited to the following: “I will give you some text blocks. Please summarize the content of these text blocks. The summary should be logical and you need to remove any redundant information that is not relevant to the user’s question.

[0030] Note: Please only output the content you summarized, do not output any other content, do not output irrelevant guide words or expressions similar to subheadings, and do not include numbers, codes, or special symbols.

[0031] Let’s officially start: Text block collection:<Text Chunk> User Question:<User Query> ”.

[0032] In a specific embodiment, the utility evaluation-based retrieval enhancement generation method of the present invention can be implemented as follows: Figure 2 The schematic shown implements: The first step is to use the user input question to search for relevant corpus blocks (i.e., corpus block 1, corpus block 2, ..., corpus block N) in the pre-established database; The second step is to generate a utility score for each corpus using a trained utility evaluation model, which is trained based on Large Language Models (LLMs). The third step is to sort all retrieved chunks using the semantic similarity and utility scores between the chunks and the user input question (i.e., to obtain chunk K, chunk N, ..., chunk M). The fourth step is to use an adaptive Top K selection model to select the top K corpus chunks as the reference context for the large language model; Step 5: Use the large language model (LLM) to summarize the information in the corpus. In the sixth step, the summary of the previous step is used as the context of the large language model, and the large language model is asked to answer the user's question based on the prompt word.

[0033] The utility evaluation-based retrieval enhancement generation method of the present invention first performs utility evaluation on multiple corpora retrieved based on user input questions to obtain a utility score for each corpus that can characterize the usefulness of the corpus in answering the user input question. Then, based on the utility score of the corpus and the semantic similarity between the corpus and the user input question, the multiple corpora are sorted to select Top K corpora from them. The selected Top K corpora have the highest semantic similarity with the standard answer, which is equivalent to screening the retrieved corpora according to the answer to the user input question, so that the corpora finally obtained by the utility evaluation-based retrieval enhancement generation method are more closely related to the user input question and have less noise.

[0034] The steps of the various methods above are divided only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are within the scope of protection of the present invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of the invention.

[0035] Another embodiment of the present invention relates to a search enhancement generation device based on utility evaluation. The implementation details of the search enhancement generation device based on utility evaluation of this embodiment are described in detail below. The following content is provided only for ease of understanding and is not required for implementing this solution. The search enhancement generation device based on utility evaluation of this embodiment includes: A corpus retrieval module is used to obtain multiple corpus blocks retrieved according to the user input question; The utility evaluation module is used to evaluate the utility of each corpus block and obtain a corresponding utility score. The utility score is used to represent the usefulness of the corpus block in answering the user's input question; The corpus chunk selection module is used to sort multiple corpora based on their utility scores and the semantic similarity between the corpora and the user input question, and select the top K corpora from the sorted multiple corpora using the trained Top K selection model; The Top K selection model is trained by selecting the optimal Top K chunks from a variety of Top K chunks corresponding to each user-input question based on the standard answers to the questions. Each Top K chunk is selected using the Top K algorithm from the sorted chunks corresponding to the user-input question. The optimal Top K chunk has the highest semantic similarity with the standard answer. The question answering module is used to answer user input questions using the Top K corpus.

[0036] It is not difficult to find that this embodiment is an apparatus embodiment corresponding to the above-mentioned method embodiment, and this embodiment can be implemented in conjunction with the above-mentioned method embodiment. The relevant technical details and technical effects mentioned in the above-mentioned embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiment.

[0037] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0038] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the utility evaluation-based retrieval enhancement generation method in the above-mentioned embodiments.

[0039] The memory and processor are connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor.

[0040] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0041] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0042] That is, those skilled in the art will understand that all or part of the steps in the above-described method embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a device (such as a microcontroller or chip) or a processor to execute all or part of the steps in the method embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0043] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A retrieval enhancement generation method based on utility evaluation, characterized in that: The method comprises: Obtain multiple corpus blocks retrieved based on the user input question; Performing a utility evaluation on each corpus chunk to obtain a corresponding utility score, wherein the utility score is used to represent the usefulness of the corpus chunk in answering the user input question; Sort multiple chunks based on their utility scores and the semantic similarity between the chunks and the user's input question, and use the trained Top K selection model to select the Top K chunks from the sorted chunks. The Top K selection model is trained by selecting the optimal Top K chunks from a variety of Top K chunks corresponding to each user-input question based on the standard answers to the questions. Each Top K chunk is selected using the Top K algorithm from the sorted chunks corresponding to the user-input question. The optimal Top K chunk has the highest semantic similarity with the standard answer. Answer user input questions using Top K corpus chunks.

2. The utility evaluation-based retrieval enhancement generation method according to claim 1, characterized in that: The utility evaluation of each corpus block to obtain a corresponding utility score includes: The utility evaluation model trained in the following way is used to evaluate the utility of each corpus block and obtain the corresponding utility score: The actual large language model is trained using utility scores obtained by evaluating a corpus of several user input questions using a standard large language model and utility scores obtained by evaluating a corpus of several user input questions using an actual large language model; Among them, the number of parameters of the actual large language model is smaller than that of the standard large language model.

3. The utility evaluation-based search enhancement generation method according to claim 2, characterized in that: The training steps of the Top K selection model include: Obtain multiple user input questions and the standard answer corresponding to each user input question; Select a variety of different Top K corpus blocks corresponding to each user input question; Input multiple different Top K corpora into the standard large language model respectively, so that the standard large language model can answer the user input questions based on each Top K corpus; Obtain the answer of the standard large language model based on each Top K corpus block, and calculate the information entropy between the answer of the standard large language model and the standard answer; Based on the information entropy between the answer of the standard large language model and the standard answer, for each user input question, the optimal Top K corpus chunk is determined from the corresponding multiple Top K corpus chunks; The Top K selection model is trained using the optimal Top K corpus blocks corresponding to multiple user input questions.

4. The utility evaluation-based search enhancement generation method according to claim 3, characterized in that: The calculating of the information entropy between the answer result of the standard large language model and the standard answer includes: The answer results of the standard large language model and the standard answer are segmented separately to ensure that the answer results correspond to the segmented words in the standard answer one by one; Count the frequency of each pair of participles appearing simultaneously in the answer and the standard answer, and divide the frequency of each pair of participles appearing simultaneously by the total number of participle pairs in the answer and the standard answer to obtain the probability of each pair of participles appearing simultaneously. The information entropy between the answer result and the standard answer is obtained using the following formula: ； In the formula, X represents the answer result, Y represents the standard answer, represents the information entropy between X and Y, m and n are the number of possible values ​​of X and Y respectively, Indicates that the value of X is And the value of Y is The joint probability of , b is a preset constant.

5. The utility evaluation-based search enhancement generation method according to claim 1, characterized in that: The training steps of the utility evaluation model include: Get several user input questions; Obtain multiple corpus chunks retrieved based on each user input question; Input each input question and each corresponding corpus chunk into the standard large language model to evaluate the utility score of each corpus chunk through the standard large language model; Input each input question and each corresponding corpus chunk into the actual large language model, and evaluate the utility score of each corpus chunk through the actual large language model; The actual large language model training is supervised according to the loss function used to constrain the distance between the utility score corresponding to the standard large language model and the utility score corresponding to the actual large language model.

6. The utility evaluation-based search enhancement generation method according to claim 1, characterized in that: Answering the user input question using the Top K corpus includes: A preset large language model is used to summarize the information in the Top K corpus blocks, and the information in the summarized Top K corpus blocks is used as a context reference for the preset large language model to answer user input questions through the preset large language model.

7. A retrieval enhancement generation device based on utility evaluation, characterized in that: The device comprises: A corpus retrieval module is used to obtain multiple corpus blocks retrieved according to the user input question; A utility evaluation module is used to evaluate the utility of each corpus chunk and obtain a corresponding utility score, wherein the utility score is used to represent the usefulness of the corpus chunk in answering the user input question; The corpus chunk selection module is used to sort multiple corpora based on their utility scores and the semantic similarity between the corpora and the user input question, and select the top K corpora from the sorted multiple corpora using the trained Top K selection model; The Top K selection model is trained by selecting the optimal Top K chunks from a variety of Top K chunks corresponding to each user-input question based on the standard answers to the questions. Each Top K chunk is selected using the Top K algorithm from the sorted chunks corresponding to the user-input question. The optimal Top K chunk has the highest semantic similarity with the standard answer. The question answering module is used to answer user input questions using the Top K corpus.

8. A computer device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the utility evaluation-based retrieval enhancement generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the utility evaluation-based retrieval enhancement generation method according to any one of claims 1 to 6 is implemented.

Images