Search enhancement generation method based on sequence generation

Abstract

The application provides a retrieval enhancement generation method based on sequence generation, which comprises the following steps: retrieving a query text to obtain a candidate relevant text; generating a current predicted text block vector based on an initial context sequence and a sequence generation model, and matching a most relevant text sequence from the candidate relevant text based on the current predicted text block vector; updating the initial context sequence based on the current most relevant text sequence, generating a next predicted text block vector based on the updated initial context sequence and the sequence generation model, and obtaining a next round of most relevant text sequence until the obtained most relevant text sequence meets a preset condition; and determining a target prompt word based on the obtained most relevant text sequence. The method provided by the application optimizes the candidate relevant text obtained through retrieval by using a sequence generation model, realizes the generation of a coherent and high-quality prompt word, and further improves the performance of a downstream task of fine-tuning or prompt learning based on the prompt word.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a retrieval enhancement generation method based on sequence generation. BACKGROUND

[0002] With the rapid development of natural language processing and deep learning, more and more tasks begin to use pre-trained language models to improve the effect, such as information retrieval, question and answer system, dialogue generation, etc. In order to adapt to the downstream task, the pre-trained language model usually needs to be fine-tuned or prompted learning combined with the task related prompt. The current construction method of the prompt mainly searches for the information related to the input in the large-scale corpus, and directly splices the relevant information obtained by retrieval to obtain the prompt.

[0003] However, directly splicing the retrieval result to the input ignores the noise and redundancy of the retrieval text, resulting in low quality of the generated prompt, which further affects the training effect of fine-tuning or prompt learning based on the prompt. SUMMARY

[0004] The present application provides a retrieval enhancement generation method based on sequence generation to solve the defect that the quality of the generated prompt based on the direct splicing of the retrieval result in the prior art is not high.

[0005] The present application provides a retrieval enhancement generation method based on sequence generation, comprising:

[0006] Obtaining a query text, retrieving the query text to obtain a candidate relevant text;

[0007] Based on the query text, obtaining an initialization context sequence of the current round, and based on the initialization context sequence and a sequence generation model, generating a current predicted text block vector, to match the current most relevant text sequence of the current round from the candidate relevant text based on the current predicted text block vector;

[0008] Updating the initialization context sequence based on the current most relevant text sequence, generating a next predicted text block vector based on the updated initialization context sequence and the sequence generation model, to match the next most relevant text sequence of the next round from the candidate relevant text based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition;

[0009] Based on the obtained most relevant text sequence, determining a target prompt of the query text, to determine an enhanced reply text of the query text based on the target prompt; the sequence generation model is obtained by training an initial sequence generation model based on sample query texts.

[0010] According to the retrieval enhancement generation method based on sequence generation provided by the application, the initialization context sequence of the current round is obtained based on the query text, and the initialization context sequence of the current round comprises:

[0011] Based on the query text, the candidate relevant text and the sequence generation model, a reply text vector of the query text is generated;

[0012] Based on the reply text vector, the initialization context sequence of the current round is obtained.

[0013] According to the retrieval enhancement generation method based on sequence generation provided by the application, the candidate relevant text comprises a plurality of candidate text segments with different text sources;

[0014] The most relevant text sequence of the current round is matched from the candidate relevant text based on the current predicted text block vector, and the most relevant text sequence of the current round comprises:

[0015] The semantic similarity between the current predicted text block vector and each candidate text segment is calculated respectively;

[0016] Based on the calculated semantic similarity, the most relevant text sequence of the current round is matched from the candidate text segments.

[0017] According to the retrieval enhancement generation method based on sequence generation provided by the application, the query text is retrieved to obtain a candidate relevant text, and the retrieval comprises:

[0018] Based on the keywords in the query text, sparse retrieval is performed to obtain key relevant text;

[0019] Based on the query text, dense retrieval is performed to obtain semantic relevant text;

[0020] Based on the key relevant text and the semantic relevant text, the candidate relevant text is obtained.

[0021] According to the retrieval enhancement generation method based on sequence generation provided by the application, the training step of the sequence generation model comprises:

[0022] The sample query text and a plurality of sample relevant texts corresponding to the sample query text are obtained;

[0023] Based on the sample query text and the plurality of sample relevant texts, mutual information calculation is performed to obtain the optimal context segment of the sample query text;

[0024] The sample query text and the optimal context segment are input into the initial sequence generation model to obtain a sample predicted text block output by the initial sequence generation model;

[0025] Based on the sample prediction text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model.

[0026] According to the sequence generation-based retrieval enhancement generation method provided by the application, mutual information calculation is performed based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text, which comprises the following steps:

[0027] The plurality of sample related texts are respectively subjected to segmentation processing to obtain sample related text segments of the sample related texts;

[0028] Mutual information calculation is performed based on the sample query text and the sample related text segments to obtain contribution values of the sample related text segments;

[0029] Based on the contribution values of the sample related text segments, preferred context segments are selected from the sample related texts;

[0030] The preferred context segments in the sample related texts are aggregated to obtain the optimal context segment of the sample query text.

[0031] The application further provides a sequence generation-based retrieval enhancement generation device, which comprises:

[0032] An acquisition unit is configured to acquire a query text, and perform retrieval on the query text to obtain candidate related texts;

[0033] An initial prediction unit is configured to obtain an initialization context sequence of a current round based on the query text, and generate a current prediction text block vector based on the initialization context sequence and a sequence generation model, so as to match a most relevant text sequence of the current round from the candidate related texts based on the current prediction text block vector;

[0034] A loop prediction unit is configured to update the initialization context sequence based on the current most relevant text sequence, generate a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, so as to match a most relevant text sequence of a next round from the candidate related texts based on the next prediction text block vector, until the most relevant text sequence meets a preset condition;

[0035] A generation unit is configured to determine a target prompt word of the query text based on the obtained most relevant text sequence, and determine an enhanced reply text of the query text based on the target prompt word; the sequence generation model is obtained by training an initial sequence generation model based on sample query texts.

[0036] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the sequence generation based search enhancement generation method according to any one of the above when executing the program.

[0037] The application further provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the sequence generation based search enhancement generation method according to any one of the above.

[0038] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the sequence generation based search enhancement generation method according to any one of the above.

[0039] The sequence generation based search enhancement generation method provided by the application can ensure the diversity of prompt words by searching the query text to obtain candidate relevant texts. Then, the sequence generation model is used to optimize the candidate relevant texts obtained by searching, the candidate text segments obtained by searching are used as basic units of similar words, and the sequence generation algorithm is used to optimize and combine the candidate relevant texts, so that coherent and high-quality prompt words can be obtained, and the performance of downstream tasks based on prompt word fine-tuning or prompt learning can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is one of the flowcharts of the sequence generation based search enhancement generation method provided by the application;

[0042] Figure 2 is another flowchart of the sequence generation based search enhancement generation method provided by the application;

[0043] Figure 3 is a training flowchart of the sequence generation model provided by the application;

[0044] Figure 4 is a method flowchart for constructing sample data provided by the application;

[0045] Figure 5 is a structural diagram of the sequence generation based search enhancement generation device provided by the application;

[0046] Figure 6 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described below in conjunction with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] Current prompt word construction methods mainly rely on manual design or heuristic rules, which not only consume time and effort, but also are difficult to cover diversified query intentions, resulting in generated prompt words being difficult to accurately depict the real needs of users. In addition, prompt word learning for downstream tasks also lacks effective optimization mechanisms, making it difficult to fully exploit the potential of pre-trained language models. To solve this problem, a method of automatically constructing prompt words using retrieval technology has emerged. This retrieval-enhanced method can obtain diversified background knowledge by retrieving information related to the input in a large-scale corpus, reducing the dependence on manually annotated data. However, existing retrieval-enhanced methods usually directly concatenate the retrieval results to the input, ignoring the noise and redundancy of the retrieval text, resulting in low-quality generated prompt words.

[0049] To solve this problem, the present application provides a retrieval-enhanced generation method based on sequence generation to generate more coherent, high-quality and user-intention-compliant prompt words. Figure 1 is one of the flowcharts of the retrieval-enhanced generation method based on sequence generation provided by the present application, as shown in Figure 1 The method comprises the following steps:

[0050] Step 110: obtaining a query text, retrieving the query text to obtain a candidate relevant text;

[0051] Step 120: obtaining an initial context sequence of the current round based on the query text, and generating a current predicted text block vector based on the initial context sequence and a sequence generation model, to match a most relevant text sequence of the current round from the candidate relevant text based on the current predicted text block vector;

[0052] Step 130: updating the initial context sequence based on the current most relevant text sequence, generating a next predicted text block vector based on the updated initial context sequence and the sequence generation model, to match a most relevant text sequence of the next round from the candidate relevant text based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition;

[0053] At step 140, based on the obtained most relevant text sequence, a target prompt word of the query text is determined, so as to determine an enhanced reply text of the query text based on the target prompt word; the sequence generation model is obtained by training an initial sequence generation model based on sample query texts.

[0054] Here, the query text refers to the text contained in the user's query request, which can include question text, instructions, or text information related to the query. For example, the user's historical interaction, relevant document, or text information in the external knowledge base. Therefore, the query text here can reflect the user's query intention. The candidate relevant text here refers to the background knowledge related to the query intention in the query text, which can include multiple candidate relevant text segments from different sources.

[0055] In addition, the initialization context sequence here refers to the query text and the predicted text block vector generated based on the sequence generation model. It can be understood that with the step-by-step prediction and generation of the prompt word, the predicted text block vector in the initialization context sequence becomes more and more rich.

[0056] The sequence generation model here can be a decoder-only generation model. Therefore, the query text and the retrieved candidate relevant text can be first vector-encoded by an encoder such as Bert to obtain the query text vector and the relevant text segment vector.

[0057] Specifically, first, the text information in the query request can be obtained by acquiring the user's query request, and the text information in the query request can be taken as the query text. The text in the query request can be preprocessed, including text cleaning, word segmentation, and removal of stop words, etc. The processed text can be taken as the query text to facilitate the subsequent execution of the query text. Then, the query text can be retrieved in a large-scale corpus, and the background knowledge related to the query text can be taken as the candidate relevant text. It should be noted that the number and object of the large-scale corpus are not specifically limited, which can be a professional knowledge base in a certain specific field, or a basic knowledge base such as a large language model. Therefore, the candidate relevant text obtained here contains extensive and rich background knowledge, which further ensures the diversification of the prompt word.

[0058] Further, the query text vector obtained after vectorizing the query text can be used as the initial context sequence. Alternatively, the query text vector and the reply text vector corresponding to the query text vector can also be used as the initial context sequence. Then, the initial context sequence can be input into the sequence generation model, and the next text block with the highest probability can be predicted by the sequence generation model based on the initial context sequence. Then, the most relevant text sequence of the current round can be matched from the candidate relevant text based on the current predicted text block vector. For example, the text segment vector with the highest semantic similarity to the current predicted text block vector can be matched from the candidate relevant text by calculating the semantic similarity between the current predicted text block vector and each text segment vector in the candidate relevant text.

[0059] It can be understood that the current predicted text block vector generated by the sequence model filters the candidate relevant text obtained by retrieval, selects the most relevant text segment, further improves the quality of the prompt word on the basis of ensuring the coherence of the prompt word, and avoids interference caused by text information with low relevance.

[0060] Further, the initial context sequence can be updated by the current most relevant text sequence. For example, the text vector of the current most relevant text sequence of the current round can be added to the initial context sequence after the query text vector to obtain an updated initial context sequence. Then, the updated initial context sequence can be input into the sequence generation model, and the next predicted text block vector can be generated by the sequence generation model based on the updated initial context sequence. For example, the next predicted text block vector can be generated by the following formula:

[0061]

[0062] In the formula, pred_text_block_next represents the next predicted text block vector; pred_text_block_next represents the next predicted text block vector; pred_text_block_next represents the next predicted text block vector; pred_text_block_next represents the next predicted text block vector.

[0063] Similarly, the text segment vector with the highest semantic similarity to the next predicted text block vector can be matched from the candidate relevant text by calculating the semantic similarity between the next predicted text block vector and each text segment vector in the candidate relevant text. The text segment vector with the highest semantic similarity to the next predicted text block vector can be used as the most relevant text sequence of the next round. The above steps are repeated until the most relevant text sequence meets the preset condition. For example, the most relevant text sequence generated after multiple rounds reaches the preset maximum generation length, or the decoder generates a specific termination symbol, which can be a quotation mark “”. The preset condition can be set based on the actual requirements of the prompt word, and the embodiments of the present application do not make specific limitations.

[0064] Then, the most relevant text sequences obtained in each round can be arranged in the order of the most relevant text sequences in each round in the initialization context sequence to obtain a final coherent text sequence. Then, a final prompt word is obtained by integrating output based on the coherent text sequence. The process of integrating output can include text processing operations such as grammar correction and style adjustment to improve the naturalness of the text and the satisfaction of the user. Finally, an enhanced reply text corresponding to the query text can be generated by indicating the target prompt word. The enhanced reply text here refers to the reply text of the question obtained based on the above retrieval enhancement method, which can be regarded as a more appropriate reply text to the query intent of the user.

[0065] It should be noted that, compared with the prior art in which the candidate text segments are directly spliced to obtain a prompt word, resulting in an incoherent prompt word. The embodiments of the present application treat the candidate relevant text segments obtained by retrieval as basic units similar to words, assign a weight or generation probability to each text segment, and then use a sequence generation algorithm to optimize the combination of these text segments, so that a coherent prompt word can be obtained. The introduction of the sequence generation algorithm into the prompt word construction process creates a new optimization paradigm for retrieval enhancement and prompt word generation.

[0066] The method provided by the embodiments of the present application ensures the diversity of the prompt word by retrieving the query text to obtain candidate relevant text. Then, the candidate relevant text obtained by retrieval is optimized by a sequence generation model, the candidate text segments obtained by retrieval are treated as basic units similar to words, and a sequence generation algorithm is used to optimize the combination of the candidate relevant text, so that a coherent and high-quality prompt word can be obtained, and the performance of the downstream task of fine-tuning or prompt learning based on the prompt word is improved.

[0067] To further improve the quality of the generated prompt word, so that the prompt word can more accurately depict the real needs of the user. Based on any of the above embodiments, in step 120, an initialization context sequence of the current round is obtained based on the query text, including:

[0068] Based on the query text, the candidate relevant text, and the sequence generation model, a reply text vector of the query text is generated;

[0069] Based on the reply text vector, an initialization context sequence of the current round is obtained.

[0070] The reply text vector here refers to a text vector for replying to the query text. When the query text is a question, the reply text can be an answer corresponding to the question. When the query text is a command, the reply text can be an execution result corresponding to the command.

[0071] Specifically, the reply text vector of the query text can be generated by inputting the query text vector generated in advance based on the query text into the sequence generation model, taking the relevant text segment vector of the retrieved candidate relevant text as background knowledge, and generating the reply text vector of the query text through the sequence generation model. Then, the reply text vector can be added to the query text vector to obtain the initialization context sequence of the current round, so as to ensure that the predicted text block vector generated based on the sequence generation model in the subsequent generation process is closely related to the user query.

[0072] It should be noted that the reply text corresponding to the query text can be the main query intention of the user, and the reply text vector is generated as the starting point of the generation process of the sequence generation model, which ensures that the predicted text block vector generated based on the sequence generation model in the subsequent generation process is closely related to the query intention of the user, so that the prompt word can more accurately depict the real demand of the user, and the generated prompt word can achieve better performance in the downstream task.

[0073] According to any of the above embodiments, the candidate relevant text includes a plurality of candidate text segments with different text sources;

[0074] The matching of the most relevant text sequence of the current round from the candidate relevant text based on the current predicted text block vector includes:

[0075] The semantic similarity between the current predicted text block vector and each candidate text segment is calculated respectively;

[0076] The most relevant text sequence of the current round is matched from the candidate text segments based on the calculated semantic similarity.

[0077] Here, the candidate relevant text can include a plurality of candidate text segments with different text sources. Therefore, there may be problems such as noise, redundancy, and incoherence between the candidate text segments, which affect the quality of the text. Therefore, in order to further optimize and screen the retrieved candidate relevant text, the semantic similarity between the current predicted text block vector and each candidate text segment can be calculated respectively. For example, the cosine similarity between the current predicted text block vector and each candidate text segment vector can be calculated as the semantic similarity. The cosine similarity here can be calculated by the following formula:

[0078]

[0079] In the formula, denotes the cosine similarity between the current predicted text block vector u and the candidate text segment vector v; denotes the dot product of two vectors; ||u|| and ||v|| denote the Euclidean norm of the current predicted text block vector u and the candidate text segment vector v, respectively.

[0080] Then, the candidate text segment vector corresponding to the maximum semantic similarity between the current predicted text block vector and each candidate text segment vector can be selected as the most relevant text sequence in the current round. Here, the most relevant text sequence can be obtained by the following formula:

[0081]

[0082] In the formula, represents the most relevant text sequence; represents the candidate text segment vector; represents the semantic similarity between the current predicted text block vector and the candidate text segment vector.

[0083] The method provided by the embodiment of the application optimizes and selects each candidate text segment based on retrieval by using the current predicted text block vector generated by the sequence generation model, matches the most relevant text sequence in the current round, eliminates redundant information in the candidate relevant text based on retrieval, takes the text most consistent with the user query intention as the most relevant text, and generates the final prompt word based on the most relevant text in each round, so that the text quality of the prompt word is higher and more consistent with the user query intention.

[0084] Based on any of the above embodiments, in step 110, the query text is retrieved to obtain candidate relevant text, including:

[0085] Sparse retrieval is performed based on the keywords in the query text to obtain key relevant text;

[0086] Dense retrieval is performed based on the query text to obtain semantic relevant text;

[0087] The candidate relevant text is obtained based on the key relevant text and the semantic relevant text.

[0088] Specifically, each keyword in the query text can be indexed by an inverted index to obtain key relevant text related to each keyword to complete sparse retrieval. It should be noted that sparse retrieval can quickly respond and accurately match background knowledge related to each keyword in the query text, and can retrieve more rich and diverse background knowledge. It can be understood that the key relevant text obtained by sparse retrieval contains more rich and diverse background knowledge, but may lack in the semantic understanding of the context.

[0089] ​​Thus, the query text vector of the query text can be input into the deep learning model, and the similar text vector to the query text vector can be searched through vector similarity search. The text obtained by decoding the text vector is the semantic related text. The semantic related text is the background knowledge similar to the query text in semantics.

[0090] Finally, the key related text and the semantic related text can be used as the candidate related text. It should be noted that the combination of sparse retrieval and dense retrieval ensures the speed and basic relevance of the retrieval, and improves the richness and diversity of the candidate related text. In addition, the accuracy and depth understanding of the retrieval result are also improved to achieve comprehensive, accurate and efficient retrieval.

[0091] In an embodiment, Figure 2 is a flowchart of the retrieval enhancement generation method based on sequence generation provided by the present application, as Figure 2 shown, first, the query text vector (Query Embedding) and each candidate text segment vector (Context 1 Embedding, Context 2 Embedding, …) are input into the sequence generation model (Transformer Decoder) to reduce the calculation amount of the sequence generation model. In detail, the reply text vector is output by the sequence generation model, and the initialization context sequence of the current round is obtained based on the reply text vector and the query text vector. Then, based on the initialization context sequence, the sequence generation model generates the current predicted text block vector (Predict Embedding). Then, based on the most relevant text sequence of the current round matched from each candidate text segment, the initialization context sequence is updated, and based on the updated initialization context sequence, the sequence generation model generates the next predicted text block vector. Based on the next predicted text block vector, the most relevant text sequence of the next round is matched from the candidate related text, until the most relevant text sequence meets the preset condition, and all the most relevant text sequences are obtained, that is, all the relevant context blocks Embedding in Figure 2 . Then, based on the text generation strategy, all the relevant context blocks Embedding are integrated and output to form the final answer or prompt word sequence. The output text is properly post-processed, such as grammar correction, style adjustment, etc., to improve the naturalness of the text and the satisfaction of the user.

[0092] It should be noted that the method provided by the embodiment of the application realizes rapid positioning of the optimal context of the query in a large number of candidate related texts by fusing the latest technologies of deep learning and natural language processing, and outputs high-quality prompt words through an efficient sequence generation method, thereby significantly improving the response quality of the system and user satisfaction. This method of generating prompt words is not only suitable for question and answer systems, but also can be extended to various dialogue systems, and has a wide application prospect.

[0093] Based on any of the above embodiments, the training step of the sequence generation model includes:

[0094] obtaining the sample query text and a plurality of sample related texts corresponding to the sample query text;

[0095] calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text;

[0096] inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample predicted text block output by the initial sequence generation model;

[0097] based on the sample predicted text block and the optimal context segment, obtaining a prediction loss, and iteratively training the initial sequence generation model based on the prediction loss to obtain the sequence generation model.

[0098] Specifically, first, the query in the question and answer pair in different fields such as medical treatment, law, and technology can be collected as the sample query text, and the corresponding context can be collected as the plurality of sample related texts corresponding to the sample query text. In detail, the collected question and answer pair and the corresponding context can be preprocessed, which can include data cleaning to remove low-quality texts and irrelevant text information, and can also include standardization format arrangement to ensure that each sample has a complete "query-context-answer" format. In addition, the sample related texts can be processed by block, and the context content of each sample can be segmented into several smaller and semantically complete text segments by using a natural language processing tool such as a sentence segmenter. The segmented text segments should maintain the complete meaning expression as much as possible, and the length of each segment can be adjusted according to actual needs, usually not more than a few sentences.

[0099] Then, mutual information calculation can be performed according to the sample query text and each sample related text respectively, so as to select a text with a higher contribution degree to the answer from the plurality of sample related texts as the optimal context segment. The mutual information calculation here can be calculated by the following formula:

[0100]

[0101] In the formula, denotes the probability difference of the expected output answer o in the case of containing the sample query text and the first sample related text, and the first text segment; wherein, denotes the probability of the model generating the answer o in the case of containing the first sample query text and the first sample related text, and the first text segment; denotes the probability of the model generating the answer o in the case of containing the sample query text .

[0102] Further, the sample query text and the optimal context segment can be input to an initial sequence generation model to obtain a sample predicted text block output by the initial sequence generation model. The initial sequence generation model here can be a model containing only a decoder, and then the sample query text and the optimal context segment can be vectorized based on BERT, GPT or other natural language processing pre-training models in advance to obtain a sample query text vector and an optimal context segment vector. Thus, the optimal context segment vector can be appended to the sample query text vector and input to the initial sequence generation model together, and the embedding vector of the sample predicted text block is output by the initial sequence generation model.

[0103] Then, the mean square error calculation can be performed through the embedding vector of the sample predicted text block and the optimal context segment vector to obtain a prediction loss. The prediction loss here can be calculated by the following formula:

[0104]

[0105] In the formula, denotes the prediction loss; denotes the total number of sample query texts; denotes the first sample query text in the training set; denotes the embedding vector of the first sample predicted text block; denotes the optimal context segment vector corresponding to the first sample query text.

[0106] Finally, the initial sequence generation model is trained by minimizing the prediction loss. The model weights can be adjusted by a backpropagation algorithm and a selected optimizer (e.g., Adam or SGD) to reduce the error between the predicted embedding vectors and the true embedding vectors, resulting in a sequence generation model with accurate predictions.

[0107] It should be noted that the method provided by the embodiments of the present application trains an initial sequence generation model based on a TransformerDecoder to generate embedding vectors of text blocks, so that the generated sample prediction text block embedding vectors are as close as possible to the optimal context segment vectors, which helps to build an efficient generation model, especially for natural language generation tasks that require coherence and rich semantics.

[0108] In an embodiment, Figure 3 is a schematic diagram of the training process of the sequence generation model provided by the present application, as Figure 3 shown, first, the sample query text vector (Query Embedding) and each related text segment vector (Context 1 Embedding, Context 2 Embedding...) are input into the sequence generation model (Transformer Decoder), and the sample prediction text block vector (Predict Embedding) is generated by the sequence generation model. The loss Loss is calculated by the sample prediction text block vector generated by prediction and the optimal context segment that is most helpful to the target answer based on the sample query text, and the initial sequence generation model is trained and iterated based on the goal of minimizing the loss Loss to obtain the final sequence generation model. In an embodiment, Figure 4 is a schematic diagram of the method for constructing sample data provided by the present application, as Figure 4 shown, first, the text blocks can be obtained by text blocking the sample related text in the training database (Query, Context, Answer), and the text blocks are vectorized. Then, the optimal context sequence can be determined by conditional mutual information, that is, from multiple text block Embedding, the optimal context sequence based on the query is determined to obtain the answer that is most helpful to the query. Thus, the training corpus (Query, Context, bestContext, Answer) is obtained. Thus, the sample query text and the sample related text can be selected from the training corpus, and the optimal context segment can be used as a label.

[0109] Based on any of the above embodiments, the mutual information calculation based on the sample query text and the plurality of sample related texts to obtain the optimal context segment of the sample query text comprises:

[0110] The multiple sample-related texts are segmented to obtain sample-related text fragments for each sample-related text.

[0111] Mutual information is calculated based on the sample query text and the relevant text fragments of each sample to obtain the contribution value of the relevant text fragments of each sample;

[0112] Based on the contribution value of the relevant text fragments of each sample, the preferred context fragments are selected from the relevant texts of each sample;

[0113] The optimal context fragment of the sample query text is obtained by aggregating the preferred context fragments from the relevant text of each sample.

[0114] Specifically, firstly, the relevant texts of multiple samples are segmented to obtain the relevant text fragments of each sample; then, mutual information is calculated based on the sample query text and the relevant text fragments to obtain the contribution value of each relevant text fragment.

[0115] Understandably, the contribution value here reflects the degree to which the relevant text fragment contributes to the corresponding sample answer obtained based on the sample query text, and thus reflects the degree to which the relevant text fragment matches the user's query intent. For example, the higher the contribution value of the relevant text fragment, the higher the degree to which the relevant text fragment matches the user's query intent; conversely, the lower the contribution value, the lower the degree to which the relevant text fragment matches the user's query intent.

[0116] Then, the contribution value of each sample-related text fragment can be selected based on a preset threshold. Specifically, for a single sample-related text, for each sample-related text fragment's contribution value, those fragments below the preset threshold are removed. This is understood to mean that a CXMI score is calculated for each text fragment. Only when the score of a sample-related text fragment is higher than the threshold is that fragment considered significantly helpful in generating the answer. Then, the sample-related text fragment with the highest CXMI score among the remaining fragments is selected as the preferred context fragment for that sample-related text. For example, this can be achieved using the following formula:

[0117]

[0118] In the formula, Indicates from the first The first sample related text in the first sample Each sample-related text fragment; Indicates from the first Select the text fragment with the highest CXMI score from the relevant texts of each sample. .

[0119] Similarly, the preferred context fragments of each sample relevant text corresponding to the sample query text are selected. Finally, the preferred context fragments of each sample relevant text corresponding to the sample query text are aggregated to obtain the optimal context fragment of the sample query text.

[0120] It should be noted that the method provided by the embodiments of the present application can not only systematically evaluate the contribution of each sample relevant text fragment, but also accurately select the sample relevant text fragment that has a significant impact on the performance of the sequence generation model. Therefore, it has obvious advantages in ensuring the relevance and accuracy of the generated content, especially in tasks that require highly accurate information support, such as machine translation, automatic question answering systems, etc.

[0121] Based on any of the above embodiments, Figure 5 is a structural schematic diagram of the retrieval enhancement generation device based on sequence generation provided by the present application, as Figure 5 shown, the device comprises:

[0122] The acquisition unit 510 acquires the query text, retrieves the query text, and obtains the candidate relevant text;

[0123] The initial prediction unit 520 obtains the initialization context sequence of the current round based on the query text, and generates the current predicted text block vector based on the initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the current round from the candidate relevant text based on the current predicted text block vector.

[0124] The cycle prediction unit 530 updates the initialization context sequence based on the current most relevant text sequence, generates the next predicted text block vector based on the updated initialization context sequence and the sequence generation model, and matches the most relevant text sequence of the next round from the candidate relevant text based on the next predicted text block vector, until the obtained most relevant text sequence meets the preset condition.

[0125] The generation unit 540 determines the target prompt word of the query text based on the obtained most relevant text sequence, and determines the enhanced reply text of the query text based on the target prompt word; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0126] The device provided by the embodiment of the application can obtain candidate relevant texts by searching the query text, so as to ensure the diversity of the prompt words. Then, the sequence generation model is used to optimize the candidate relevant texts obtained by searching, the candidate text segments obtained by searching are used as basic units of similar words, and the sequence generation algorithm is used to optimize and combine the candidate relevant texts, so that coherent and high-quality prompt words can be obtained, and the performance of the downstream task based on the prompt word fine-tuning or prompt learning is improved.

[0127] Based on any of the above embodiments, the initial prediction unit is specifically configured to:

[0128] generate a reply text vector of the query text based on the query text, the candidate relevant text, and the sequence generation model;

[0129] obtain an initial context sequence of the current round based on the reply text vector.

[0130] Based on any of the above embodiments, the candidate relevant text includes multiple candidate text segments with different text sources;

[0131] The initial prediction unit is specifically configured to:

[0132] calculate the semantic similarity between the current prediction text block vector and each candidate text segment, respectively;

[0133] match the most relevant text sequence of the current round from the candidate text segments based on the calculated semantic similarity.

[0134] Based on any of the above embodiments, the obtaining unit is specifically configured to:

[0135] perform sparse retrieval based on the keywords in the query text to obtain key relevant texts;

[0136] perform dense retrieval based on the query text to obtain semantic relevant texts;

[0137] obtain the candidate relevant text based on the key relevant texts and the semantic relevant texts.

[0138] Based on any of the above embodiments, the device further includes a training unit, and the training unit is specifically configured to:

[0139] obtain the sample query text and a plurality of sample relevant texts corresponding to the sample query text;

[0140] perform mutual information calculation based on the sample query text and the plurality of sample relevant texts to obtain an optimal context segment of the sample query text;

[0141] inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample predicted text block output by the initial sequence generation model;

[0142] based on the sample predicted text block and the optimal context segment, obtaining a prediction loss, and iteratively training the initial sequence generation model based on the prediction loss to obtain the sequence generation model.

[0143] Based on any of the above embodiments, the training unit is further specifically configured to:

[0144] segmenting the plurality of sample related texts respectively to obtain sample related text segments of each sample related text;

[0145] Based on the sample query text and each sample related text segment, mutual information is calculated to obtain a contribution value of each sample related text segment;

[0146] Based on the contribution value of each sample related text segment, an optimal context segment is selected from each sample related text;

[0147] Aggregating the optimal context segments in the sample related texts to obtain the optimal context segment of the sample query text.

[0148] Figure 6 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 6As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logic instruction in the memory 630 to execute a retrieval enhancement generation method based on sequence generation, which includes: obtaining query text, performing retrieval on the query text to obtain candidate relevant text; based on the query text, obtaining an initial context sequence of the current round, and based on the initial context sequence and a sequence generation model, generating a current predicted text block vector to match the current most relevant text sequence of the current round from the candidate relevant text based on the current predicted text block vector; updating the initial context sequence based on the current most relevant text sequence, generating a next predicted text block vector based on the updated initial context sequence and the sequence generation model, to match the next most relevant text sequence of the next round from the candidate relevant text based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition; based on the obtained most relevant text sequence, determining a target prompt word of the query text, to determine an enhanced reply text of the query text based on the target prompt word; and the sequence generation model is obtained by training an initial sequence generation model based on sample query text.

[0149] In addition, the logic instruction in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0150] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and which, when executed by a processor, enables a computer to perform the sequence generation-based search enhancement generation method provided by the above-mentioned methods. The method comprises: obtaining a query text, performing a search on the query text to obtain candidate relevant texts; based on the query text, obtaining an initial context sequence of a current round, and based on the initial context sequence and a sequence generation model, generating a current predicted text block vector, to match a most relevant text sequence of the current round from the candidate relevant texts based on the current predicted text block vector; updating the initial context sequence based on the current most relevant text sequence, and based on the updated initial context sequence and the sequence generation model, generating a next predicted text block vector, to match a most relevant text sequence of a next round from the candidate relevant texts based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition; based on the obtained most relevant text sequence, determining a target prompt word of the query text, to determine an enhanced reply text of the query text based on the target prompt word; and the sequence generation model is obtained by training an initial sequence generation model based on sample query texts.

[0151] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sequence generation-based search enhancement generation method provided by the above-mentioned methods. The method comprises: obtaining a query text, performing a search on the query text to obtain candidate relevant texts; based on the query text, obtaining an initial context sequence of a current round, and based on the initial context sequence and a sequence generation model, generating a current predicted text block vector, to match a most relevant text sequence of the current round from the candidate relevant texts based on the current predicted text block vector; updating the initial context sequence based on the current most relevant text sequence, and based on the updated initial context sequence and the sequence generation model, generating a next predicted text block vector, to match a most relevant text sequence of a next round from the candidate relevant texts based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition; based on the obtained most relevant text sequence, determining a target prompt word of the query text, to determine an enhanced reply text of the query text based on the target prompt word; and the sequence generation model is obtained by training an initial sequence generation model based on sample query texts.

[0152] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0154] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sequence generation based retrieval augmentation generation method, characterized by, The method comprises the following steps: obtaining a query text, retrieving the query text to obtain a candidate relevant text; based on the query text, obtaining an initial context sequence of the current round, and based on the initial context sequence and a sequence generation model, generating a current predicted text block vector, to match the current predicted text block vector from the candidate relevant text to obtain the most relevant text sequence of the current round; based on the most relevant text sequence of the current round, updating the initial context sequence, based on the updated initial context sequence and the sequence generation model, generating a next predicted text block vector, to match the next predicted text block vector from the candidate relevant text to obtain the most relevant text sequence of the next round, until the obtained most relevant text sequence meets the preset condition; based on the obtained most relevant text sequence, determining the target prompt word of the query text, and based on the target prompt word, determining the enhanced reply text of the query text; the sequence generation model is obtained by training an initial sequence generation model based on sample query texts; the training steps of the sequence generation model comprise: obtaining the sample query texts and a plurality of sample relevant texts corresponding to the sample query texts; based on the sample query texts and the plurality of sample relevant texts, mutual information is calculated to obtain the optimal context segment of the sample query text; input the sample query text and the optimal context segment into the initial sequence generation model to obtain the sample predicted text block output by the initial sequence generation model; based on the sample predicted text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model; the mutual information calculation based on the sample query text and the plurality of sample relevant texts to obtain the optimal context segment of the sample query text comprises: segment processing is performed on the plurality of sample relevant texts respectively to obtain sample relevant text segments of each sample relevant text; based on the sample query text and each sample relevant text segment, mutual information is calculated to obtain the contribution value of each sample relevant text segment; based on the contribution value of each sample relevant text segment, the preferred context segment is selected from each sample relevant text; the preferred context segments in the plurality of sample relevant texts are aggregated to obtain the optimal context segment of the sample query text.

2. The sequence generation based retrieval augmentation generation method of claim 1, wherein, based on the query text, the initial context sequence of the current round is obtained, which comprises: based on the query text, the candidate relevant text and the sequence generation model, a reply text vector of the query text is generated; based on the reply text vector, the initial context sequence of the current round is obtained.

3. The sequence generation based retrieval augmentation generation method of claim 1, wherein, The candidate relevant text comprises a plurality of candidate text segments with different text sources; based on the current predicted text block vector, the most relevant text sequence of the current round is matched from the candidate relevant text, which comprises: the semantic similarity between the current predicted text block vector and each candidate text segment is calculated respectively; Match the most relevant text sequence of the current round from the candidate texts based on the calculated semantic similarities.

4. The sequence generation based retrieval augmentation generation method of any one of claims 1 to 3, wherein, The searching the query text to obtain candidate relevant texts comprises: Performing sparse search based on keywords in the query text to obtain key relevant texts; Performing dense search based on the query text to obtain semantic relevant texts; Obtaining the candidate relevant texts based on the key relevant texts and the semantic relevant texts.

5. A sequence generation based retrieval augmentation generation apparatus, characterized by, Comprise: An acquisition unit that acquires a query text, searches the query text, and obtains candidate relevant texts; An initial prediction unit that obtains an initial context sequence of the current round based on the query text, generates a current predicted text block vector based on the initial context sequence and a sequence generation model, and matches the current predicted text block vector to obtain the most relevant text sequence of the current round from the candidate relevant texts; A loop prediction unit that updates the initial context sequence based on the most relevant text sequence of the current round, generates a next predicted text block vector based on the updated initial context sequence and the sequence generation model, and matches the next predicted text block vector to obtain the most relevant text sequence of the next round from the candidate relevant texts until the obtained most relevant text sequence meets a preset condition; A generation unit that determines a target prompt word of the query text based on the obtained most relevant text sequence, and determines an enhanced reply text of the query text based on the target prompt word; The sequence generation model is obtained by training an initial sequence generation model based on sample query texts; Further comprise a training unit, and the training unit is specifically used for: Acquiring the sample query texts and a plurality of sample relevant texts corresponding to the sample query texts; Performing mutual information calculation based on the sample query texts and the plurality of sample relevant texts to obtain an optimal context segment of the sample query texts; Inputting the sample query texts and the optimal context segment into the initial sequence generation model to obtain a sample predicted text block output by the initial sequence generation model; Obtaining a prediction loss based on the sample predicted text block and the optimal context segment, and iteratively training the initial sequence generation model based on the prediction loss to obtain the sequence generation model; The training unit is further specifically used for: Segmenting the plurality of sample relevant texts respectively to obtain sample relevant text segments of the sample relevant texts; Performing mutual information calculation based on the sample query texts and the sample relevant text segments to obtain contribution values of the sample relevant text segments; Selecting optimal context segments from the sample relevant texts based on the contribution values of the sample relevant text segments; Aggregating the optimal context segments in the sample relevant texts to obtain the optimal context segment of the sample query texts.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the computer program to achieve the sequence generation-based search enhancement generation method in any one of claims 1 to 4. The processor implements the computer program to achieve the sequence generation-based search enhancement generation method in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the search enhancement generation method based on sequence generation as claimed in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program, which is executed by a processor, implements the search enhancement generation method based on sequence generation as claimed in any one of claims 1 to 4.

Images