Search enhancement generation method and apparatus, device, and storage medium

Abstract

The specification provides a retrieval enhancement generation method, device, equipment and storage medium, which comprises: determining a target paragraph from a plurality of candidate paragraphs retrieved in a database according to a user question; inputting the target paragraph and the question into a first large language model to generate a first answer; inputting the target paragraph, the question and the first answer into a second large language model, and the output evaluation result comprises a paragraph evaluation result indicating that the number of target paragraphs is excessive or insufficient; adjusting the number of target paragraphs input into the second large language model according to the paragraph evaluation result, repeatedly executing the processes of generating the first answer, outputting the evaluation result and adjusting the number of target paragraphs until a preset termination condition is met; inputting the question and the target paragraph after the number is adjusted into the first large language model to obtain the target answer of the question. The scheme effectively solves the problems of noise retrieval and missing retrieval in the retrieval enhancement generation task, improves the accuracy of retrieval, and improves the generation quality.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of computer technology, and more particularly to a retrieval-augmented generation method, device, equipment and storage medium. BACKGROUND

[0002] Retrieval-augmented generation (RAG) technology significantly improves the performance of question answering systems by integrating information retrieval and text generation. The core technical path of this method is: the system first retrieves relevant document information from a large-scale corpus based on the semantic of the user's query question, and then inputs these retrieval results as context into a pre-trained language model to generate accurate and informative answers.

[0003] However, retrieval-augmented generation technology still faces several key technical challenges in practical applications, the most prominent of which is the noise interference problem in the retrieval stage. Specifically, when retrieving relevant documents, the system inevitably introduces irrelevant noise information that is not related to the query intent. The presence of these noise information may mislead the reasoning process of the language model, resulting in a decrease in the accuracy of the generated answers. On the other hand, when the retrieval system fails to obtain sufficient key context information, i.e., the missing retrieval phenomenon occurs, which will directly limit the generation ability of the language model, making it unable to provide complete and accurate answers. These technical bottlenecks seriously restrict the performance of retrieval-augmented generation methods in practical application scenarios. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide a retrieval-augmented generation method, device, equipment and storage medium to improve retrieval accuracy and thus improve generation quality.

[0005] To achieve the above-mentioned purpose, one or more embodiments of the present specification provide technical solutions as follows:

[0006] According to a first aspect of one or more embodiments of the present specification, a retrieval-augmented generation method is provided, comprising:

[0007] Retrieving in a database according to a question raised by a user to obtain a plurality of candidate paragraphs;

[0008] Determining a target paragraph from the plurality of candidate paragraphs, wherein the similarity score of the target paragraph to the question meets a set requirement;

[0009] Inputting the target paragraph and the question into a first large language model to generate a first answer;

[0010] inputting the target passage, the question and the first answer into a second large language model to make the second large language model output an evaluation result, the evaluation result including a passage evaluation result indicating that the target passage is too much or insufficient in quantity;

[0011] adjusting the quantity of the target passage input into the second large language model according to the passage evaluation result, and repeating the process of generating a first answer, outputting an evaluation result and adjusting the quantity of the target passage until a preset stopping condition is met;

[0012] inputting the question and the target passage with the adjusted quantity into the first large language model to obtain a target answer to the question.

[0013] In some embodiments, the evaluation result further includes an evaluation score of the first answer, and the preset stopping condition includes that the evaluation score meets a set condition or the number of repeated execution reaches a set number threshold,

[0014] The method further includes:

[0015] In the case that the evaluation score does not meet the set condition, the passage evaluation result is obtained, otherwise the answer to the question is obtained.

[0016] In some embodiments, the plurality of candidate passages are arranged in descending order of similarity scores with the question, and the adjusting the quantity of the target passage input into the second large language model according to the passage evaluation result includes:

[0017] In the case that the passage evaluation result indicates that the quantity is too much, the quantity of the target passage is reduced, and the target passages arranged in the front are retained;

[0018] In the case that the passage evaluation result indicates that the quantity is insufficient, the quantity of the target passage is increased according to the arrangement.

[0019] In some embodiments, the inputting the target passage, the question and the first answer into the second large language model includes:

[0020] combining the target passage, the question and the first answer according to a preset instruction template to generate a self-feedback prompt, the self-feedback prompt being used to instruct the second large language model to generate an evaluation result in a specified manner;

[0021] inputting the self-feedback prompt into the second large language model.

[0022] In some embodiments, the method further includes constructing a database, specifically including the following steps:

[0023] segmenting documents in a corpus into a plurality of semantically complete passages using a semantic segmentation model;

[0024] converting each passage into a vector representation using an embedding model;

[0025] constructing the database from the respective vector representations.

[0026] In some embodiments, the method further comprises training the semantic segmentation model, specifically comprising the steps of:

[0027] obtaining a sample dataset comprising a plurality of sentence pairs each consisting of two adjacent sentences, the sentence pairs having labels indicating whether the two sentences are in the same passage;

[0028] training the semantic segmentation model according to the sample dataset.

[0029] In some embodiments, the step of segmenting documents in a corpus into a plurality of semantically complete passages using a semantic segmentation model comprises:

[0030] inputting each pair of adjacent sentences in the corpus into the semantic segmentation model and obtaining a prediction result of the semantic segmentation model for the pair of adjacent sentences;

[0031] when the prediction result indicates that the two sentences are not in the same passage, segmenting the two sentences into different passages;

[0032] when the prediction result indicates that the two sentences are in the same passage, putting the two sentences into the same passage.

[0033] In some embodiments, the semantic segmentation model comprises a sentence encoding model, a feature enhancement model, and a multi-layer perceptron, and the method further comprises:

[0034] the sentence encoding model obtaining a first encoding vector and a second encoding vector from the two sentences;

[0035] the feature enhancement model performing subtraction and multiplication operations on the first and second encoding vectors to obtain a difference vector and a product vector;

[0036] the multi-layer perceptron obtaining the prediction result from the first and second encoding vectors, the product vector, and the difference vector.

[0037] In some embodiments, the step of determining a target passage from the plurality of candidate passages whose rate of change of similarity score to the question meets a set requirement comprises:

[0038] In a case where the plurality of candidate paragraphs are arranged in descending order of the similarity score with the question, a score drop rate of each paragraph is determined according to a score ratio of each paragraph to a previous paragraph;

[0039] A paragraph before a candidate paragraph with a score drop rate reaching a set gradient threshold is selected as the target paragraph.

[0040] According to a second aspect of one or more embodiments of the present specification, a retrieval enhancement generation apparatus is provided, comprising:

[0041] A retrieval unit is configured to perform retrieval in a database according to a question raised by a user to obtain a plurality of candidate paragraphs;

[0042] A determination unit is configured to determine, from the plurality of candidate paragraphs, a target paragraph with a change rate of a similarity score with the question meeting a set requirement;

[0043] A generation unit is configured to input the target paragraph and the question into a first large language model to generate a first answer;

[0044] An evaluation unit is configured to input the target paragraph, the question and the first answer into a second large language model to cause the second large language model to output an evaluation result, the evaluation result including a paragraph evaluation result indicating that the target paragraph is too much or insufficient in quantity;

[0045] An adjustment unit is configured to adjust the quantity of target paragraphs input into the second large language model according to the paragraph evaluation result, and repeatedly perform the processes of generating a first answer, outputting an evaluation result and adjusting the quantity of target paragraphs until a preset termination condition is met;

[0046] An obtaining unit is configured to input the question and the target paragraph with an adjusted quantity into the first large language model to obtain a target answer to the question.

[0047] In some embodiments, the evaluation result further includes an evaluation score of the first answer, and the preset termination condition includes that the evaluation score meets a set condition, or the number of repeated executions reaches a set number threshold,

[0048] The apparatus further includes a termination unit configured to:

[0049] In a case where the evaluation score does not meet the set condition, the paragraph evaluation result is obtained, otherwise an answer to the question is obtained.

[0050] In some embodiments, the adjustment unit is specifically configured to:

[0051] In a case where the paragraph evaluation result indicates that the quantity is too much, the quantity of target paragraphs is reduced, and the retained are target paragraphs ranked in the front.

[0052] In a case where the paragraph evaluation result indicates that the quantity is insufficient, the quantity of the target paragraph is increased according to the ranking.

[0053] In some embodiments, the evaluation unit, when used for inputting the target paragraph, the question and the first answer into the second large language model, is specifically used for:

[0054] combining the target paragraph, the question and the first answer according to a preset instruction template to generate a self-feedback prompt, the self-feedback prompt being used for instructing the second large language model to generate an evaluation result in a specified manner;

[0055] inputting the self-feedback prompt into the second large language model.

[0056] In some embodiments, the apparatus further comprises a construction unit configured to:

[0057] segmenting documents in a corpus into a plurality of semantically complete paragraphs by using a semantic segmentation model;

[0058] converting each paragraph into a vector representation by using an embedding model;

[0059] constructing the database according to the respective vector representations.

[0060] In some embodiments, the apparatus further comprises a training unit configured to:

[0061] obtain a sample data set, the sample data set containing a plurality of sentence pairs each consisting of two adjacent sentences, the sentence pairs having labels indicating whether the two sentences are in the same paragraph;

[0062] train the semantic segmentation model according to the sample data set.

[0063] In some embodiments, the construction unit, when used for segmenting documents in a corpus into a plurality of semantically complete paragraphs by using a semantic segmentation model, is specifically used for:

[0064] inputting each two adjacent sentences in the corpus into the semantic segmentation model and obtaining a prediction result of the semantic segmentation model on the two adjacent sentences;

[0065] in a case where the prediction result indicates that the two sentences are not in the same paragraph, segmenting the two sentences into different paragraphs;

[0066] in a case where the prediction result indicates that the two sentences are in the same paragraph, putting the two sentences into the same paragraph.

[0067] In some embodiments, the semantic segmentation model comprises a sentence encoding model, a feature enhancement model and a multi-layer perceptron, and the apparatus further comprises a prediction unit configured to:

[0068] The sentence encoding model obtains a first encoding vector and a second encoding vector according to the two sentences;

[0069] The feature enhancement model performs subtraction and multiplication operations on the first encoding vector and the second encoding vector to obtain a difference vector and a product vector;

[0070] The multi-layer perceptron obtains the prediction result according to the first encoding vector, the second encoding vector, the product vector and the difference vector.

[0071] In some embodiments, the determining unit is specifically configured to:

[0072] In a case where the plurality of candidate paragraphs are arranged in descending order of similarity scores with the question, determine a score drop rate of each paragraph according to a score ratio of each paragraph to a previous paragraph;

[0073] Select a paragraph before a candidate paragraph with a score drop rate reaching a set gradient threshold as the target paragraph.

[0074] According to a third aspect of one or more embodiments of the present specification, an electronic device is provided, comprising:

[0075] a processor;

[0076] a memory for storing processor-executable instructions;

[0077] wherein the processor implements the steps of the method proposed in the above embodiments by running the executable instructions.

[0078] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method proposed in the above embodiments.

[0079] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising computer program / instructions, and the computer program / instructions are executed by a processor to implement the steps of the method proposed in the above embodiments.

[0080] The retrieval enhancement generation method provided by the embodiment of the present specification firstly determines a target paragraph according to the relevance of a plurality of candidate paragraphs retrieved in a database to a user question, then evaluates the quantity rationality of an answer generated by a large language model by using another large language model, adjusts the quantity of the target paragraph according to the evaluation result, and finally obtains a target answer of the user question according to the target paragraph after the quantity adjustment as the context. The scheme effectively solves the problems of noisy retrieval and missing retrieval in the retrieval enhancement generation task, improves the accuracy of retrieval, and improves the generation quality. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 FIG. 1 is an application scenario diagram of a retrieval enhancement generation method provided by an example embodiment.

[0082] Figure 2 FIG. 2 is a flowchart of a retrieval enhancement generation method provided by an example embodiment.

[0083] Figure 3 FIG. 3 is a schematic diagram of a paragraph relevance score.

[0084] Figure 4 FIG. 6 is a block diagram of a retrieval enhancement generation device provided by an example embodiment.

[0085] Figure 5 FIG. 7 is a structural schematic diagram of an apparatus provided by an example embodiment. DETAILED DESCRIPTION

[0086] The example embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Instead, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the present specification, as detailed in the appended claims.

[0087] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in the present specification. In some other embodiments, the steps included in the method can be more or less than described in the present specification. In addition, a single step described in the present specification can be divided into a plurality of steps for description in other embodiments; and a plurality of steps described in the present specification can also be combined into a single step for description in other embodiments.

[0088] Existing retrieval augmentation generation methods inevitably introduce redundant information unrelated to the query intent when retrieving relevant documents. This noise can mislead the language model's reasoning process, leading to decreased accuracy in the generated answers. Furthermore, the inability to obtain sufficient key contextual information results in missing retrieval, directly limiting the language model's generation capabilities and preventing it from providing complete and accurate answers. These technical bottlenecks severely restrict the performance of retrieval augmentation generation methods in real-world applications.

[0089] To address the aforementioned issues, this application proposes a search enhancement generation scheme.

[0090] like Figure 1 The diagram shown illustrates an application scenario of a retrieval enhancement generation method as described in this application. Figure 1 It includes: client 10, database 20, first major language model 30, and second major language model 40.

[0091] Client 10 is a device with network connectivity, data processing, and user interaction capabilities. It is responsible for receiving user input requests, executing the request processing flow, and presenting the processing results to the user. Client 10 can be a client for smartphones, tablets, desktop computers, smart wearable devices, in-vehicle systems, smart home devices, learning machines, etc.

[0092] Database 20 is pre-built and can be stored either internally within client 10 or independently externally. Client 10 can interact with database 20.

[0093] The first and second largest language models 30 and 40 are large-scale neural network models in the field of Natural Language Processing (NLP). They typically consist of hundreds of millions to billions of parameters and learn complex patterns and structures through pre-training on large amounts of text data, enabling them to understand and generate complex language data. The first and second largest language models 30 and 40 are usually deployed on servers to provide backend support for the applications or services of client 10. Client 10 can communicate with the first and second largest language models 30 and 40 on the server via the Internet, sending requests and receiving processed responses.

[0094] It should be noted that adding additional modules to or removing individual modules from the illustrated environment does not change the underlying concept of the exemplary embodiments of this application; and the request processing method proposed in this application is not only applicable to... Figure 1 The application scenarios shown are also applicable to any device that has request processing needs.

[0095] Figure 2 A flowchart of a method for retrieving and enhancing generation is shown, which is performed by a client in Figure 1 . As shown in Figure 2 , the method comprises:

[0096] Step 201, retrieving in the database according to the question raised by the user to obtain a plurality of candidate paragraphs.

[0097] In some embodiments, the database can be a pre-constructed vector retrieval database, which contains a plurality of candidate paragraph vectors. By utilizing an embedding model to vectorize the user question, the question is converted into a vector, and then the vector retrieval database is queried to extract a plurality of candidate vectors closest to the question vector. Specifically, the similarity (relevance) between the question vector and the paragraph vectors in the database can be calculated, and then all paragraphs are sorted in descending order according to the similarity score (relevance score), and the top N paragraphs with the highest score are selected as the candidate results. Wherein, the number N of the plurality of candidate paragraphs can be pre-set.

[0098] Step 202, determining a target paragraph from the plurality of candidate paragraphs, which has a change rate of the similarity score with the question meeting a set requirement.

[0099] Since the plurality of candidate paragraphs obtained in step 201, it is possible to ignore useful paragraphs due to insufficient number of selections, or to include irrelevant paragraphs due to excessive number of selections, and unnecessary information in these paragraphs can confuse the large language model and make it difficult to give correct answers.

[0100] Figure 3 Two cases of paragraph relevance scores of two articles in the database are shown. As can be seen from Figure 3 , the score pattern of the paragraphs shows a sharp decline before a gradual decline. In this example, if only the first three paragraphs of article 1 and one paragraph of article 2 are selected, it is easier to get the correct answer. This shows that the relevance of the paragraphs before the sharp decline to the question is significantly higher than that of the paragraphs after the sharp decline.

[0101] Therefore, in the embodiments of the present disclosure, the change rate of the similarity of the retrieved plurality of candidate paragraphs with the question raised by the user is used to further screen the candidate paragraphs, and the candidate paragraphs with the change rate meeting the set requirement are determined as target paragraphs for subsequent processing steps.

[0102] Specifically, in the case that the plurality of candidate paragraphs are arranged in descending order of the similarity score with the question, the score drop rate of each paragraph can be determined according to the score ratio of each paragraph to the previous paragraph, and the paragraph before the candidate paragraph whose score drop rate reaches a set gradient threshold is selected as the target paragraph. That is, from the N candidate paragraphs obtained in step 201, the top K candidate paragraphs (selected from top in descending order) before the significant drop of the similarity score are selected as the target paragraphs (K < N), ensuring that only the candidate paragraphs that meet the set requirements of relevance to the question are selected for the next stage.

[0103] This method of dynamically selecting paragraphs based on the gradient of the ranked paragraph scores, rather than selecting a fixed number of paragraphs, can more accurately identify the most relevant paragraphs to a given question.

[0104] Step 203: inputting the target paragraph and the question into a first large language model to generate a first answer.

[0105] The target paragraph selected in step 202 is used as context and is output to the first large language model along with the question posed by the user, and the model generates an initial answer in response to the question, referred to herein as the first answer.

[0106] Step 204: inputting the target paragraph, the question, and the first answer into a second large language model to cause the second large language model to output an evaluation result.

[0107] The evaluation result can include a paragraph evaluation result indicating that the number of target paragraphs is excessive or insufficient. That is, the second large language model is used to evaluate whether the number of target paragraphs as context is excessive or insufficient.

[0108] Step 205: adjusting the number of target paragraphs input to the second large language model according to the paragraph evaluation result, repeating the process of generating a first answer, outputting an evaluation result, and adjusting the number of target paragraphs until a preset stopping condition is met.

[0109] Specifically, in the case that the paragraph evaluation result indicates that the number is excessive, the number of target paragraphs is reduced, and the retained target paragraphs are the ones ranked first; in the case that the paragraph evaluation result indicates that the number is insufficient, the number of target paragraphs is increased according to the ranking.

[0110] For example, assuming that N candidate passages are retrieved according to the user question in step 201, K target passages whose similarity score change rates meet the set requirements are filtered out from the N candidate passages in step 202, i.e., the top K passages in terms of relevance scores. If the passage evaluation result indicates that the number of target passages is excessive, K-1 passages are selected from the ranked passages; if the passage evaluation result indicates that the number of target passages is insufficient, K+1 passages are selected from the ranked passages. Then steps 203 to 205 are repeatedly executed, i.e., the reselected passages are input into the first large language model to generate a first answer, the second large language model is used to evaluate whether the number of passages is excessive or insufficient, and adjustment is made again according to the evaluation result until a preset termination condition is met. The termination condition can include that the evaluation score meets the set condition or the number of repeated executions reaches a set number threshold M. That is, when steps 203 to 205 have been repeatedly executed M times or the passage evaluation result is neither excessive nor insufficient (the judgment result of the second large language model is that the number of passages is unchanged), adjustment is stopped and the final target passage is obtained.

[0111] By using the second large language model to adjust the number of retrieved target passages, the problems of noise retrieval and missing retrieval are further solved.

[0112] In step 206, the question and the target passage after adjustment of the number are input into the first large language model to obtain a target answer to the question.

[0113] The retrieval enhancement generation method proposed in the embodiments of the present specification first determines target passages for the relevance of a plurality of candidate passages retrieved in a database to a user question, then uses another large language model to evaluate the rationality of the number of target passages for an answer generated by a large language model, adjusts the number of target passages according to the evaluation result, and finally obtains a target answer to the user question according to the target passage after adjustment of the number as context. This scheme effectively solves the problems of noise retrieval and missing retrieval in the retrieval enhancement generation task, improves the accuracy of retrieval, and improves the generation quality.

[0114] In some embodiments, the evaluation result further includes an evaluation score of the first answer. For example, the second large language model can score the first answer from multiple dimensions such as accuracy, relevance, completeness, flow, information richness, etc. The score value can be, for example, 1 to 10 points.

[0115] In the case where the evaluation result also contains the evaluation score of the first answer, first determine whether the evaluation score meets the set condition, for example, whether the score reaches the feedback score threshold fs, such as 9, if it meets, directly get the target answer of the question, and return it to the user; If not, further obtain the paragraph evaluation result, and adjust the number of target paragraphs according to the paragraph evaluation result.

[0116] In some embodiments, the target paragraph, the question and the first answer can be combined according to a preset instruction template to generate a self-feedback prompt, which is used to instruct the second large language model to generate the evaluation result in a specified manner; Next, the self-feedback prompt is input to the second large language model.

[0117] The self-feedback prompt can serve two purposes: 1) to evaluate the quality of the first answer, and 2) to evaluate the appropriateness of the retrieved paragraphs, that is, to evaluate whether the target paragraphs as context contain redundant paragraphs or lack the necessary paragraphs to answer the question. This feedback process produces two feedback outputs, the first being an evaluation score from 1 to 10, and the second being a paragraph evaluation result indicating context adjustment, which can be represented by -1 or 1. Where -1 indicates that there is redundant information in the target paragraphs, and 1 indicates that additional information is needed to fully answer the question.

[0118] After receiving the feedback output, first check the evaluation score of the first answer. If this score reaches or exceeds the feedback score threshold fs, the generated first answer is considered acceptable, and it is determined as the target answer and then presented to the user; If not, adjust the context according to the paragraph evaluation result. If the paragraph evaluation result indicates -1, it means the number of target paragraphs should be reduced, for example, reduced by one according to the ranking; Conversely, if the paragraph evaluation result indicates 1, it means the number of target paragraphs should be increased, for example, increased by one according to the ranking. After adjustment, steps 203 to 205 are repeated until the evaluation score reaches or exceeds the feedback score threshold fs, or the loop has been executed for a set number of times threshold M.

[0119] In the current RAG system, two main corpus segmentation methods are used. The first method segments the corpus into passages based on a predetermined number of tokens, and the second method ensures that each passage contains complete sentences based on the first method. However, the first method can result in incomplete sentences in a passage, which weakens the overall coherence and meaning of each passage, making the similarity between the calculated user question and these passages invalid; the second method segments consecutive sentences of a fixed length into a passage. According to this method, if the passage exceeds the predetermined length limit, the last sentence is transferred to the next passage in its entirety to ensure that each passage contains complete sentences. However, this method can also distort the meaning of the passage.

[0120] To solve the above problems, the embodiments of the present disclosure propose a method for constructing a database, which uses a lightweight and effective semantic segmentation model to segment the documents in the corpus into multiple semantically complete passages to ensure that each passage conveys a clear and complete meaning without containing too many tokens; then uses an embedding model to convert each passage into a vector representation, and constructs the database according to the respective vector representations.

[0121] The semantic segmentation model constructed in the embodiments of the present disclosure mainly includes three parts, namely a sentence encoding model, a feature enhancement model, and a multi-layer perception.

[0122] The sentence encoding model is a lightweight embedding model that obtains a first encoding vector x1 and a second encoding vector x2 according to the input of two sentences; the feature enhancement model receives x1 and x2, performs subtraction and multiplication operations to process these embeddings, and obtains their difference vector (x1-x2) and product vector x1*x2; the multi-layer perception receives x1, x2, (x1-x2), and (x1*x2) to produce a score representing the prediction result, which is used to determine whether the two input sentences should be segmented or kept as a continuous part.

[0123] The semantic segmentation model proposed in the embodiments of the present disclosure can better capture the semantic differences or similarities between sentences, thereby improving the semantic segmentation effect.

[0124] In some embodiments, the semantic segmentation model can be trained in the following way.

[0125] First, a sample data set is obtained, that is, a large amount of training data for semantic segmentation is collected. The sample data set contains multiple sentence pairs composed of two adjacent sentences, and the sentence pairs have labels indicating whether the two sentences are in the same passage. For example, a label of 1 indicates that the two sentences should be classified into the same passage; a label of 0 indicates that they should be segmented into different passages.

[0126] In one example, a Wikipedia dataset can be used as the data source, in which all the passages have been semantically segmented into paragraphs.

[0127] Next, all the sentence pairs are input into the semantic segmentation model, a loss function is constructed according to the output of the model and the label of the sentence pair itself, the parameters of the semantic segmentation model are trained using the loss function, and the trained semantic segmentation model is obtained until the loss function converges.

[0128] In some embodiments, each two adjacent sentences in the corpus can be input into the semantic segmentation model, and the prediction result of the two adjacent sentences by the semantic segmentation model is obtained; when the prediction result indicates that the two sentences are not in the same paragraph, the two sentences are segmented into different paragraphs; when the prediction result indicates that the two sentences are in the same paragraph, the two sentences are put into the same paragraph.

[0129] The prediction result can be a score, if the score is lower than a predetermined segmentation score threshold ss (the threshold is between 0 and 1, for example 0.5), the two adjacent sentences will be segmented into different paragraphs; on the contrary, if the score is higher than or equal to ss, the two adjacent sentences will remain in the same paragraph.

[0130] For any given corpus, the segmentation process can be quickly performed, because the lightweight semantic segmentation model can be run in parallel in GPU. For example, all the sentence pairs in the corpus can be collected and organized into multiple batches, and then the semantic segmentation model is called to perform inference in parallel.

[0131] Figure 4 is a block diagram of a retrieval enhancement generation apparatus provided by an exemplary embodiment. As shown in Figure 4 The apparatus includes:

[0132] The retrieval unit 401 is configured to perform retrieval in the database according to a question raised by a user, and obtain a plurality of candidate paragraphs.

[0133] The determination unit 402 is configured to determine a target paragraph from the plurality of candidate paragraphs, wherein a change rate of a similarity score of the target paragraph with respect to the question meets a set requirement.

[0134] The generation unit 403 is configured to input the target paragraph and the question into a first large language model, and generate a first answer.

[0135] The evaluation unit 404 is configured to input the target paragraph, the question and the first answer into a second large language model, so that the second large language model outputs an evaluation result, wherein the evaluation result includes a paragraph evaluation result indicating that the number of the target paragraph is excessive or insufficient.

[0136] The adjusting unit 405 is configured to adjust the number of target paragraphs input to the second large language model according to the paragraph evaluation result, repeatedly perform the process of generating a first answer, outputting an evaluation result, and adjusting the number of target paragraphs until a preset stopping condition is met.

[0137] The obtaining unit 406 is configured to input the question and the target paragraphs after adjustment into the first large language model to obtain a target answer to the question.

[0138] In some embodiments, the evaluation result further includes an evaluation score of the first answer, and the preset stopping condition includes that the evaluation score meets a set condition or the number of repeated executions reaches a set number threshold.

[0139] The device further includes a stopping unit configured to:

[0140] In a case where the evaluation score does not meet the set condition, the paragraph evaluation result is obtained, otherwise, the answer to the question is obtained.

[0141] In some embodiments, the adjusting unit is specifically configured to:

[0142] In a case where the paragraph evaluation result indicates that the number is too large, the number of target paragraphs is reduced, and the target paragraphs that are retained are the target paragraphs ranked in the front;

[0143] In a case where the paragraph evaluation result indicates that the number is insufficient, the number of target paragraphs is increased according to the ranking.

[0144] In some embodiments, when the evaluation unit is configured to input the target paragraphs, the question, and the first answer into the second large language model, the evaluation unit is specifically configured to:

[0145] combine the target paragraphs, the question, and the first answer according to a preset instruction template to generate a self-feedback prompt, the self-feedback prompt being used to instruct the second large language model to generate an evaluation result in a specified manner;

[0146] input the self-feedback prompt into the second large language model.

[0147] In some embodiments, the device further includes a constructing unit configured to:

[0148] segment documents in a corpus into a plurality of semantically complete paragraphs by using a semantic segmentation model;

[0149] convert each paragraph into a vector representation by using an embedding model;

[0150] construct the database according to the respective vector representations.

[0151] In some embodiments, the apparatus further comprises a training unit configured to:

[0152] obtain a sample dataset comprising a plurality of sentence pairs each consisting of two adjacent sentences, the sentence pairs having labels indicating whether the two sentences are in the same paragraph;

[0153] train the semantic segmentation model according to the sample dataset.

[0154] In some embodiments, the constructing unit is specifically configured to:

[0155] input each two adjacent sentences in the corpus into the semantic segmentation model and obtain a prediction result of the semantic segmentation model on the two adjacent sentences;

[0156] when the prediction result indicates that the two sentences are not in the same paragraph, segment the two sentences into different paragraphs;

[0157] when the prediction result indicates that the two sentences are in the same paragraph, put the two sentences into the same paragraph.

[0158] In some embodiments, the semantic segmentation model comprises a sentence encoding model, a feature enhancement model and a multi-layer perception, and the apparatus further comprises a prediction unit configured to:

[0159] the sentence encoding model obtains a first encoding vector and a second encoding vector according to the two sentences;

[0160] the feature enhancement model performs subtraction and multiplication operations on the first encoding vector and the second encoding vector to obtain a difference vector and a product vector;

[0161] the multi-layer perception obtains the prediction result according to the first encoding vector, the second encoding vector, the product vector and the difference vector.

[0162] In some embodiments, the determining unit is specifically configured to:

[0163] when the plurality of candidate paragraphs are arranged in descending order of the similarity scores with the question, determine a score drop rate of each paragraph according to a score ratio of each paragraph to a previous paragraph;

[0164] select a paragraph before the candidate paragraph whose score drop rate reaches a set gradient threshold as the target paragraph.

[0165] Figure 5 FIG. 1 is a schematic structural diagram of an apparatus provided by an example embodiment. Please refer to Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and of course can also include other hardware required by the business. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 510 into the memory 508 by the processor 502 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0166] The system, apparatus, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] In a typical configuration, the computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0168] The memory can include non-permanent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0169] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage, quantum memory, graphene-based storage medium or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition in this paper, the computer readable medium does not include transitory computer readable media, such as modulated data signals and carriers.

[0170] It is also important to note that the terms "comprises", "comprising", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0171] The above description of certain examples of the disclosure has been presented for the purposes of illustration and description. Other examples are within the scope and spirit of the disclosure. In some cases, the actions or processes can be performed in an order different from that described in the examples. Also, in some instances, well-known structures have not been described in order to avoid obscuring the concepts of the examples. Additionally, in some instances, the processes described have been presented in an order that is not necessarily the order in which the processes are performed. Furthermore, in some instances, the processes described have been presented in an order that is not necessarily the order in which the processes are performed. In some embodiments, multitasking and parallel processing can be advantageous.

[0172] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit one or more embodiments of the description. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this description, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0173] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are used only to distinguish one piece of information from another, but do not imply a relationship between the information. For example, without departing from the scope of one or more embodiments, the first information can be called the second information, and similarly, the second information can be called the first information. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "in response to determining."

[0174] The foregoing description of certain examples of the disclosure has been presented for the purposes of illustration and description. Other examples are within the scope and spirit of the disclosure. In some cases, the actions or processes can be performed in an order different from that described in the examples. Also, in some instances, well-known structures have not been described in order to avoid obscuring the concepts of the examples. Additionally, in some instances, the processes described have been presented in an order that is not necessarily the order in which the processes are performed. Furthermore, in some instances, the processes described have been presented in an order that is not necessarily the order in which the processes are performed. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A method for retrieval augmentation generation, the method comprising: The method comprises: retrieving a plurality of candidate passages from a database according to a question raised by a user; determining a target passage from the plurality of candidate passages, wherein a rate of change of a similarity score of the target passage to the question meets a set requirement; inputting the target passage and the question into a first large language model to generate a first answer; inputting the target passage, the question, and the first answer into a second large language model to cause the second large language model to output an evaluation result, wherein the evaluation result comprises a passage evaluation result indicating that the target passage is excessive or insufficient in quantity; adjusting the quantity of target passages input into the second large language model according to the passage evaluation result, and repeating the processes of generating a first answer, outputting an evaluation result, and adjusting the quantity of target passages until a preset stopping condition is met; inputting the question and the target passage with the adjusted quantity into the first large language model to obtain a target answer to the question.

2. The method of claim 1, wherein, The evaluation result further comprises an evaluation score of the first answer, and the preset stopping condition comprises that the evaluation score meets a set condition or the number of repeated executions reaches a set number threshold, The method further comprises: if the evaluation score does not meet the set condition, obtaining the passage evaluation result, otherwise obtaining an answer to the question.

3. The method according to claim 1 or 2, characterized in that, The plurality of candidate passages are arranged in descending order of similarity scores to the question, and the adjusting of the quantity of target passages input into the second large language model according to the passage evaluation result comprises: if the passage evaluation result indicates that the quantity is excessive, reducing the quantity of target passages, and retaining target passages arranged in the front; if the passage evaluation result indicates that the quantity is insufficient, increasing the quantity of target passages according to arrangement.

4. The method of claim 1, wherein, The inputting of the target passage, the question, and the first answer into the second large language model comprises: combining the target passage, the question, and the first answer according to a preset instruction template to generate a self-feedback prompt, wherein the self-feedback prompt is used to instruct the second large language model to generate an evaluation result in a specified manner; inputting the self-feedback prompt into the second large language model.

5. The method of claim 1, wherein, The method further comprises constructing a database, specifically comprising the following steps: segmenting documents in a corpus into a plurality of semantically complete passages by using a semantic segmentation model; converting each passage into a vector representation by using an embedding model; constructing the database according to each vector representation.

6. The method of claim 5, wherein, The method further comprises training the semantic segmentation model, specifically comprising the following steps: obtaining a sample data set, wherein the sample data set comprises a plurality of sentence pairs each consisting of two adjacent sentences, and each sentence pair has a label indicating whether the two sentences are in the same passage; training the semantic segmentation model according to the sample data set.

7. The method of claim 6, wherein, The segmenting of documents in a corpus into a plurality of semantically complete passages by using a semantic segmentation model comprises: inputting each two adjacent sentences in the corpus into the semantic segmentation model to obtain a prediction result of the semantic segmentation model for the two adjacent sentences; when the prediction result indicates that the two sentences are not in the same paragraph, segmenting the two sentences into different paragraphs; when the prediction result indicates that the two sentences are in the same paragraph, putting the two sentences into the same paragraph.

8. The method of claim 7, wherein, The semantic segmentation model comprises a sentence encoding model, a feature enhancement model and a multi-layer perception machine, and the method further comprises: The sentence encoding model obtains a first encoding vector and a second encoding vector according to the two sentences; The feature enhancement model performs subtraction and multiplication operations on the first encoding vector and the second encoding vector to obtain a difference vector and a product vector; The multi-layer perception machine obtains the prediction result according to the first encoding vector, the second encoding vector, the product vector and the difference vector.

9. The method of claim 1, wherein, The target paragraph determined from the plurality of candidate paragraphs includes: In the case where the plurality of candidate paragraphs are arranged in descending order of the similarity score with the question, determining the score drop rate of each paragraph according to the score ratio of each paragraph to the previous paragraph; selecting the paragraph before the candidate paragraph whose score drop rate reaches the set gradient threshold as the target paragraph.

10. A retrieval augmentation generation apparatus characterized by comprising: It comprises: a retrieval unit configured to retrieve a plurality of candidate paragraphs from a database according to a question raised by a user; a determination unit configured to determine a target paragraph from the plurality of candidate paragraphs, wherein the change rate of the similarity score of the target paragraph with the question meets a set requirement; a generation unit configured to input the target paragraph and the question into a first large language model to generate a first answer; an evaluation unit configured to input the target paragraph, the question and the first answer into a second large language model to make the second large language model output an evaluation result, wherein the evaluation result comprises a paragraph evaluation result indicating that the number of target paragraphs is excessive or insufficient; an adjustment unit configured to adjust the number of target paragraphs input into the second large language model according to the paragraph evaluation result, and repeat the process of generating a first answer, outputting an evaluation result and adjusting the number of target paragraphs until a preset termination condition is met; a obtaining unit configured to input the question and the target paragraph after the number is adjusted into the first large language model to obtain a target answer to the question.

11. An electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1-9 by running the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method of any one of claims 1-9.

13. A computer program product comprising computer program / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1-9.

Images