Sequence generation-based retrieval enhancement generation method

Abstract

The invention provides a sequence generation-based retrieval enhancement generation method. The method comprises the following steps of: retrieving a query text to obtain a candidate related text; based on the initialized context sequence and a sequence generation model, generating a current prediction text block vector, and based on the current prediction text block vector, matching from the candidate related texts to obtain a most related text sequence; updating the initialized context sequence based on the current most relevant text sequence, and generating a next prediction text block vector based on the updated initialized context sequence and a sequence generation model to obtain the most relevant text sequence of the next round until the obtained most relevant text sequence meets a preset condition; and determining a target cue word based on the obtained most relevant text sequence. According to the method provided by the invention, the candidate related texts obtained through retrieval are optimized through the sequence generation model, coherent and high-quality cue words are generated, and then the performance of a downstream task for fine tuning or prompt learning based on the cue words is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a retrieval enhancement generation method based on sequence generation. Background Art

[0002] With the rapid development of natural language processing and deep learning, more and more tasks are beginning to use pre-trained language models to improve results, such as information retrieval, question-answering systems, and dialogue generation. In order to adapt to downstream tasks, pre-trained language models usually need to be fine-tuned or prompted with task-related prompts. The current method of constructing prompts mainly retrieves information related to the input in a large-scale corpus and directly concatenates the retrieved related information to obtain prompts.

[0003] However, directly concatenating the search results to the input ignores the noise and redundancy of the search text, resulting in low quality of the generated prompt words, which in turn affects the training effect of fine-tuning or prompt learning based on the prompt words. Summary of the invention

[0004] The present invention provides a retrieval enhancement generation method based on sequence generation, which is used to solve the defect in the prior art that the quality of prompt words generated based on the retrieval results obtained by direct splicing is not high.

[0005] The present invention provides a retrieval enhancement generation method based on sequence generation, comprising: Obtaining a query text, searching the query text, and obtaining candidate related texts; Based on the query text, an initialization context sequence of the current round is obtained, and based on the initialization context sequence and a sequence generation model, a current prediction text block vector is generated, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; The initialization context sequence is updated based on the current most relevant text sequence, and a next predicted text block vector is generated based on the updated initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the next round from the candidate relevant texts based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition; Based on the obtained most relevant text sequence, the target prompt word of the query text is determined, and based on the target prompt word, the enhanced response text of the query text is determined; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0006] According to a retrieval enhancement generation method based on sequence generation provided by the present invention, obtaining the initialization context sequence of the current round based on the query text includes: Generate a reply text vector of the query text based on the query text, the candidate related text and the sequence generation model; Based on the reply text vector, an initialization context sequence of the current round is obtained.

[0007] According to a retrieval enhancement generation method based on sequence generation provided by the present invention, the candidate related texts include a plurality of candidate text segments with different text sources; The matching of obtaining the most relevant text sequence of the current round from the candidate relevant texts based on the current predicted text block vector includes: Calculating the semantic similarity between the current predicted text block vector and each candidate text segment respectively; Based on the calculated semantic similarities, the most relevant text sequence of the current round is matched from the candidate text segments.

[0008] According to a retrieval enhancement generation method based on sequence generation provided by the present invention, the query text is searched to obtain candidate related texts, including: Perform sparse search based on keywords in the query text to obtain key related text; Performing intensive retrieval based on the query text to obtain semantically relevant text; The candidate related text is obtained based on the key related text and the semantic related text.

[0009] According to a retrieval enhancement generation method based on sequence generation provided by the present invention, the training step of the sequence generation model includes: Acquire the sample query text and a plurality of sample related texts corresponding to the sample query text; Calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text; Inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample prediction text block output by the initial sequence generation model; Based on the sample prediction text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model.

[0010] According to a retrieval enhancement generation method based on sequence generation provided by the present invention, the mutual information calculation is performed based on the sample query text and the multiple sample related texts to obtain the optimal context segment of the sample query text, including: Segmenting the plurality of sample-related texts respectively to obtain sample-related text segments of each sample-related text; Calculating mutual information based on the sample query text and each sample-related text segment to obtain a contribution value of each sample-related text segment; Selecting a preferred context segment from each sample-related text based on the contribution value of each sample-related text segment; Aggregate the preferred context segments in the sample related texts to obtain the optimal context segment of the sample query text.

[0011] The present invention also provides a retrieval enhancement generation device based on sequence generation, comprising: An acquisition unit acquires a query text, searches the query text, and obtains candidate related texts; An initial prediction unit obtains an initialization context sequence of the current round based on the query text, and generates a current prediction text block vector based on the initialization context sequence and a sequence generation model, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; A cyclic prediction unit, which updates the initialization context sequence based on the current most relevant text sequence, generates a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, and matches the next round of most relevant text sequences from the candidate relevant texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; The generation unit determines the target prompt word of the query text based on the obtained most relevant text sequence, and determines the enhanced response text of the query text based on the target prompt word; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the retrieval enhancement generation method based on sequence generation as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the retrieval enhancement generation method based on sequence generation as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned sequence-based retrieval enhancement generation methods.

[0015] The retrieval enhancement generation method based on sequence generation provided by the present invention retrieves the query text to obtain candidate related texts to ensure the diversity of prompt words. Then, the candidate related texts obtained by retrieval are optimized through the sequence generation model, and the candidate text fragments obtained by retrieval are used as basic units similar to words. The sequence generation algorithm is used to optimize and combine the candidate related texts, so that coherent and high-quality prompt words can be obtained, thereby improving the performance of downstream tasks based on prompt words for fine-tuning or prompt learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the retrieval enhancement generation method based on sequence generation provided by the present invention; Figure 2 This is the second flow chart of the retrieval enhancement generation method based on sequence generation provided by the present invention; Figure 3 It is a schematic diagram of the training process of the sequence generation model provided by the present invention; Figure 4 It is a flow chart of the method for constructing sample data provided by the present invention; Figure 5 It is a structural schematic diagram of a retrieval enhancement generation device based on sequence generation provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The current prompt word construction method mainly relies on manual design or heuristic rules, which is not only time-consuming and laborious, but also difficult to cover diverse query intents, resulting in the generated prompt words being difficult to accurately portray the user's real needs. In addition, the prompt word learning for downstream tasks also lacks an effective optimization mechanism, making it difficult to fully tap the potential of pre-trained language models. In order to solve this problem, a method of automatically constructing prompt words using retrieval technology has emerged. This retrieval enhancement method can obtain diverse background knowledge and reduce dependence on manually annotated data by retrieving information related to the input in a large-scale corpus. However, existing retrieval enhancement methods usually directly splice the retrieval results into the input, ignoring the noise and redundancy of the retrieval text, resulting in low quality of the generated prompt words.

[0020] To address this problem, the present invention provides a retrieval enhancement generation method based on sequence generation to achieve more coherent, high-quality prompt words that meet user intentions. Figure 1 is one of the flow charts of the retrieval enhancement generation method based on sequence generation provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining a query text, searching the query text, and obtaining candidate related texts; Step 120, based on the query text, obtain an initialization context sequence of the current round, and based on the initialization context sequence and a sequence generation model, generate a current prediction text block vector, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; Step 130, updating the initialization context sequence based on the current most relevant text sequence, generating a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, so as to match the next round of most relevant text sequences from the candidate relevant texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; Step 140, based on the obtained most relevant text sequence, determine the target prompt word of the query text, and determine the enhanced response text of the query text based on the target prompt word; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0021] Here, the query text refers to the text contained in the user's query request, which can include question text, instructions, or text information related to the query. For example, the user's historical interactions, related documents, or text information in an external knowledge base. Therefore, the query text here can reflect the user's query intent. The candidate related text here refers to the background knowledge related to the query intent in the query text, and can include multiple candidate related text fragments from different sources.

[0022] In addition, the initialization context sequence here refers to the query text and the predicted text block vector generated based on the sequence generation model. It can be understood that as the prompt words are gradually predicted and generated, the predicted text block vectors in the initialization context sequence become more and more abundant.

[0023] The sequence generation model here can be a generation model based on a decoder only. Therefore, the query text and the retrieved candidate related texts can be preferentially vectorized by an encoder such as Bert to obtain a query text vector and a related text segment vector.

[0024] Specifically, first, by obtaining a user query request, the text information in the query request can be used as the query text. The text in the query request can be subjected to data preprocessing including text cleaning, word segmentation, and removal of stop words, and the processed text can be used as the query text to facilitate the subsequent execution steps of the query text. Next, the query text can be searched in a large-scale corpus, and the background knowledge related to the query text can be used as candidate related text. It should be noted that the number and objects of the large-scale corpus are not specifically limited here, and can be a professional knowledge base in a specific field, or a basic knowledge base such as a large language model. As a result, the candidate related text obtained here contains extensive and rich background knowledge, thereby ensuring the diversification of the prompt words.

[0025] Furthermore, the query text vector obtained after vectorizing the query text can be used as the initial context sequence. Alternatively, the query text vector and the reply text vector corresponding to the query text vector can also be used as the initial context sequence. Then, the initialized context sequence can be input into the sequence generation model, and the next text block can be predicted based on the initial context sequence by the sequence generation model, and the next text block with the highest probability can be used as the current predicted text block vector. Then, based on the current predicted text block vector, the most relevant text sequence of the current round is matched from the candidate related texts. For example, the semantic similarity between the current predicted text block vector and each related text segment vector in the candidate related text can be calculated, and the related text segment vector with the highest semantic similarity to the current predicted text block vector can be used as the most relevant text sequence.

[0026] It can be understood that the current predicted text block vector generated by the sequence model is used to screen the retrieved candidate related texts, select the most relevant text fragments, and further improve the quality of the prompt words on the basis of ensuring the coherence of the prompt words to avoid interference caused by text information with little relevance.

[0027] Next, the initialization context sequence is updated by the current most relevant text sequence, which can be to add the text vector of the current most relevant text sequence of the current round to the initialization context sequence and arrange it after the query text vector to obtain an updated initialization context sequence. Then, the updated initialization context sequence is input into the sequence generation model, and the next predicted text block vector is generated based on the updated initialization context sequence by the sequence generation model. For example, the generation of the next predicted text block vector can be achieved by the following formula:

[0028] In the formula, Represents the next predicted text block vector; represents a sequence generation model containing only a decoder; Represents an initial context sequence.

[0029] Similarly, by calculating the semantic similarity between the next predicted text block vector and each related text segment vector in the candidate related text, the related text segment vector with the highest semantic similarity to the next predicted text block vector can be used as the most relevant text sequence for the next round. The above steps are looped until the obtained most relevant text sequence meets the preset conditions. For example, the most relevant text sequence generated after multiple rounds may reach a preset maximum generation length, or the decoder may generate a specific termination symbol, which may be a quotation mark "". The preset conditions here may be set accordingly based on the actual requirements for the prompt words, and the embodiments of the present invention do not specifically limit this.

[0030] Next, the most relevant text sequences obtained in each round can be arranged in the order of the most relevant text sequences in each round in the initial context sequence to obtain a final coherent text sequence. Then, the final prompt words are obtained by integrating and outputting based on the coherent text sequence. The integration and output process can include text processing operations such as grammar correction and style adjustment to improve the naturalness of the text and user satisfaction. Finally, the enhanced response text corresponding to the query text can be generated through the target prompt word indication. The enhanced response text here refers to the response text of the question obtained based on the above-mentioned retrieval enhancement method, which can be regarded as a response text that is more appropriate to the user's query intention.

[0031] It should be noted that compared with the prior art that directly splices the candidate text segments to obtain prompt words, the generated prompt words are incoherent. The embodiment of the present invention regards the candidate related text segments obtained by retrieval as basic units similar to words, assigns a weight or generation probability to each text segment, and then uses a sequence generation algorithm to optimize and combine these text segments so that coherent prompt words can be obtained. Introducing the sequence generation algorithm into the prompt word construction process creates a new optimization paradigm for retrieval-enhanced prompt word generation.

[0032] The method provided by the embodiment of the present invention retrieves the query text to obtain candidate related texts to ensure the diversity of prompt words. Then, the retrieved candidate related texts are optimized through the sequence generation model, and the retrieved candidate text fragments are used as basic units similar to words. The sequence generation algorithm is used to optimize and combine the candidate related texts, so that coherent and high-quality prompt words can be obtained, thereby improving the performance of downstream tasks based on prompt word fine-tuning or prompt learning.

[0033] In order to further improve the quality of the generated prompt words so that the prompt words can more accurately describe the real needs of the user. Based on any of the above embodiments, in step 120, based on the query text, the initialization context sequence of the current round is obtained, including: Generate a reply text vector of the query text based on the query text, the candidate related text and the sequence generation model; Based on the reply text vector, an initialization context sequence of the current round is obtained.

[0034] The reply text vector here refers to the text vector used to reply to the query text. When the query text is a question, the reply text can be the answer corresponding to the question; when the query text is a command, the reply text can be the execution result corresponding to the command.

[0035] Specifically, the query text vector pre-generated based on the query text can be input into the sequence generation model, and the related text segment vector of the candidate related text retrieved can be used as background knowledge, and the response text vector of the query text can be generated through the sequence generation model. Then, the response text vector can be added to the query text vector to obtain the initialization context sequence of the current round, so as to ensure that the subsequent prediction text block vector generated based on the sequence generation model is closely related to the user query.

[0036] It should be noted that the reply text corresponding to the query text can be used as the user's most important query intention. The generated reply text vector is used as the starting point of the generation process of the sequence generation model, which ensures that the predicted text block vector generated in the subsequent generation process based on the sequence generation model is closely related to the user's query intention, so that the prompt word can more accurately portray the user's real needs, and then the generated prompt word can achieve better performance in downstream tasks.

[0037] Based on any of the above embodiments, the candidate related texts include multiple candidate text segments with different text sources; The matching of obtaining the most relevant text sequence of the current round from the candidate relevant texts based on the current predicted text block vector includes: Calculating the semantic similarity between the current predicted text block vector and each candidate text segment respectively; Based on the calculated semantic similarities, the most relevant text sequence of the current round is matched from the candidate text segments.

[0038] Here, the candidate related texts may include multiple candidate text segments with different text sources. Therefore, there may be noise, redundancy, incoherence and other problems that affect the text quality between the candidate text segments. Therefore, in order to further optimize the screening of the retrieved candidate related texts, the semantic similarity between the current predicted text block vector and each candidate text segment can be calculated respectively. For example, the cosine similarity between the current predicted text block vector and each candidate text segment vector can be calculated, and the cosine similarity can be used as the semantic similarity. The cosine similarity here can be calculated by the following formula:

[0039] In the formula, Represents the cosine similarity between the current predicted text block vector u and the candidate text segment vector v; represents the dot product of two vectors; ||u|| and ||v|| represent the Euclidean norm of the current predicted text block vector u and the candidate text segment vector v, respectively.

[0040] Then, from the semantic similarity between the current predicted text block vector and each candidate text segment vector, the candidate text segment vector corresponding to the maximum similarity can be selected as the most relevant text sequence of the current round. Here, the most relevant text sequence can be obtained by the following formula:

[0041] In the formula, represents the most relevant text sequence; Indicates candidate text segment vectors; Represents the current predicted text block vector With The semantic similarity between the candidate text segment vectors.

[0042] The method provided by the embodiment of the present invention optimizes and selects the candidate text segments obtained based on the retrieval through the current predicted text block vector generated by the sequence generation model, matches and obtains the most relevant text sequence of the current round, removes redundant information in the candidate relevant texts obtained based on the retrieval, and uses the text that best meets the user's query intention as the most relevant text, so as to generate the final prompt word based on the most relevant text of each round, so that the text quality of the prompt word is higher and more in line with the user's query intention.

[0043] Based on any of the above embodiments, in step 110, the query text is searched to obtain candidate related texts, including: Perform sparse search based on keywords in the query text to obtain key related text; Performing intensive retrieval based on the query text to obtain semantically relevant text; The candidate related text is obtained based on the key related text and the semantic related text.

[0044] Specifically, each keyword in the query text can be indexed through an inverted index to obtain key related texts related to each keyword to complete sparse retrieval. It should be noted that sparse retrieval can respond quickly and accurately match the background knowledge related to each keyword in the query text, and can retrieve richer and more diverse background knowledge. It is understandable that the key related text obtained through sparse retrieval contains richer and more diverse background knowledge, but it may lack semantic understanding of the context.

[0045] Therefore, the query text vector of the query text can be input into the deep learning model, and the text vector similar to the query text vector can be searched through vector similarity, and the text obtained by decoding the found text vector is used as the semantically related text. The semantically related text here is the background knowledge retrieved that is semantically similar to the query text.

[0046] Finally, key related texts and semantic related texts can be used as candidate related texts. It should be noted that by combining sparse retrieval and dense retrieval, the retrieval speed and basic relevance are guaranteed, and the richness and diversity of candidate related texts are improved. In addition, the accuracy and in-depth understanding of the retrieval results are also improved to achieve comprehensive, accurate and efficient retrieval.

[0047] In one embodiment, Figure 2 FIG. 2 is a flow chart of the retrieval enhancement generation method based on sequence generation provided by the present invention. Figure 2As shown, first, the query text vector (Query Embedding) and each candidate text segment vector (Context 1 Embedding, Context 2 Embedding...) are input into the sequence generation model (TransformerDecoder) to reduce the computational complexity of the sequence generation model. In detail, the reply text vector is output by the sequence generation model, and the initialization context sequence of the current round is obtained based on the reply text vector and the query text vector. Then, based on the sequence generation model, the current prediction text block vector (Predict Embedding) is generated based on the initialization context sequence. Then, based on the most relevant text sequence of the current round matched from each candidate text segment, the initialization context sequence is updated, and based on the sequence generation model, the next prediction text block vector is generated based on the updated initialization context sequence, so as to match the most relevant text sequence of the next round from the candidate related texts based on the next prediction text block vector, until the most relevant text sequence obtained meets the preset conditions, and all the most relevant text sequences are obtained, that is, Figure 2 All relevant context blocks are embedded in the text. Then, based on the text generation strategy, all the generated relevant context blocks are integrated and output to form the final answer or prompt word sequence. The output text is properly post-processed, such as grammar correction and style adjustment, to improve the naturalness of the text and user satisfaction.

[0048] It should be noted that the method provided by the embodiment of the present invention integrates the latest technologies of deep learning and natural language processing, realizes the rapid positioning of the optimal context of questions in large-scale candidate related texts, and outputs high-quality prompt words through an efficient sequence generation method, which significantly improves the response quality of the system and user satisfaction. This method of generating prompt words is not only applicable to question-answering systems, but can also be extended to various dialogue systems, and has broad application prospects.

[0049] Based on any of the above embodiments, the training step of the sequence generation model includes: Acquire the sample query text and a plurality of sample related texts corresponding to the sample query text; Calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text; Inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample prediction text block output by the initial sequence generation model; Based on the sample prediction text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model.

[0050] Specifically, first, by collecting question-answer pairs and corresponding contexts in different fields such as medicine, law, and technology, the questions in the question-answer pairs can be used as sample query texts, and the corresponding contexts can be used as multiple sample related texts corresponding to the sample query texts. In detail, the obtained question-answer pairs and corresponding contexts can be preprocessed, which can include data cleaning to remove low-quality texts and irrelevant text information; it can also include standardized formatting to ensure that each sample has a complete "question (Query)-context (Context)-answer (Answer)" format. In addition, the sample related texts can be processed in blocks, and natural language processing tools such as sentence segmenters can be used to segment the context content of each sample into several smaller, semantically complete text fragments. The segmented text fragments should be as independent as possible to maintain complete meaning expression, and the length of each fragment can be adjusted according to actual needs, usually not more than a few sentences.

[0051] Next, mutual information can be calculated based on the sample query text and each sample related text, so as to select the text with higher contribution to the answer from multiple sample related texts as the optimal context fragment. The mutual information calculation here can be calculated by the following formula:

[0052] In the formula, Indicates that the sample query text is included and The first A text segment, and then contains the sample query text In the case of , the expected probability difference of the output answer o; Representation Model Given a sample query text and The first The probability of generating answer o in the case of text segments; Representation Model Given a sample query text The probability of generating the answer o under the condition .

[0053] Furthermore, the sample query text and the optimal context fragment can be input into the initial sequence generation model to obtain a sample prediction text block output by the initial sequence generation model. The initial sequence generation model here can be a model containing only a decoder, and the sample query text and the optimal context fragment can be vectorized in advance based on BERT, GPT or other natural language processing pre-training models to obtain a sample query text vector and an optimal context fragment vector. Thus, the optimal context fragment vector can be attached to the sample query text vector and input into the initial sequence generation model together, and the embedding vector of the sample prediction text block can be output by the initial sequence generation model.

[0054] Then, the mean square error can be calculated by the embedding vector of the sample prediction text block and the optimal context segment vector to obtain the prediction loss. The prediction loss here can be calculated by the following formula:

[0055] In the formula, represents the predicted loss; Indicates the total number of sample query texts; Indicates the training set Sample query texts; Indicates The embedding vector of the sample predicted text block; Indicates The optimal context segment vector corresponding to the sample query text.

[0056] Finally, the prediction loss is minimized to train the initial sequence generation model. The model weights can be continuously adjusted through the back-propagation algorithm and the selected optimizer (such as Adam or SGD) to reduce the error between the predicted embedding vector and the true embedding vector, thus obtaining a sequence generation model with accurate prediction.

[0057] It should be noted that the method provided in the embodiment of the present invention generates embedding vectors of text blocks by training an initial sequence generation model based on TransformerDecoder, so that the embedding vectors of these generated sample prediction text blocks are as close as possible to the optimal context fragment vector, which helps to build an efficient generation model, especially in processing natural language generation tasks that require coherence and rich semantics.

[0058] In one embodiment, Figure 3 It is a schematic diagram of the training process of the sequence generation model provided by the present invention, such as Figure 3As shown, first, the sample query text vector (Query Embedding) and each related text segment vector (Context 1 Embedding, Context 2 Embedding...) are input into the sequence generation model (Transformer Decoder), and the sample prediction text block vector (Predict Embedding) is generated by the sequence generation model. The loss Loss is calculated by predicting the generated sample prediction text block vector and the optimal context segment that is most helpful in obtaining the target answer based on the sample query text, and the initial sequence generation model is trained iteratively based on the goal of minimizing the loss Loss to obtain the final sequence generation model. In one embodiment, Figure 4 is a flow chart of the method for constructing sample data provided by the present invention, such as Figure 4 As shown, first, the sample related text in the training database (Query, Context, Answer) can be segmented and the segmented text blocks can be vectorized to obtain text block Embeddings. Then, the optimal context sequence can be determined by conditional mutual information, that is, the optimal context sequence that helps the most in obtaining the answer corresponding to the query based on the query can be determined from multiple text block Embeddings. Thus, the training corpus (Query, Context, bestContext, Answer) is obtained. Thus, sample query text and sample related text can be selected from the training corpus, and the optimal context fragment can be used as a label.

[0059] Based on any of the above embodiments, the step of calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text includes: Segmenting the plurality of sample-related texts respectively to obtain sample-related text segments of each sample-related text; Calculating mutual information based on the sample query text and each sample-related text segment to obtain a contribution value of each sample-related text segment; Selecting a preferred context segment from each sample-related text based on the contribution value of each sample-related text segment; Aggregate the preferred context segments in the sample related texts to obtain the optimal context segment of the sample query text.

[0060] Specifically, first, multiple sample-related texts are segmented to obtain sample-related text segments of each sample-related text; mutual information is calculated based on the sample query text and each sample-related text segment to obtain a contribution value of each sample-related text segment.

[0061] It can be understood that the contribution value here can reflect the contribution of the sample-related text segment to the corresponding sample answer based on the sample query text, and can reflect the degree of conformity between the sample-related text segment and the user's query intention. For example, the higher the contribution value of the sample-related text segment, the higher the degree of conformity between the sample-related text segment and the user's query intention; conversely, the lower the degree of conformity between the sample-related text segment and the user's query intention.

[0062] Then, the contribution value of each sample-related text fragment can be selected by a preset threshold. In detail, for a single sample-related text, for the contribution value of each sample-related text fragment, the sample-related text fragments below the preset threshold are eliminated. It can be understood that the CXMI score is calculated for each text fragment. Only when the score of the sample-related text fragment is higher than the threshold value, the sample-related text fragment is considered to be significantly helpful in generating answers. Then, the sample-related text fragment with the highest CXMI score among the remaining is selected as the preferred context fragment of this sample-related text. For example, it can be achieved by the following formula:

[0063] In the formula, Indicates that from The first of the sample related texts Sample related text fragments; Indicates that from Select the sample-related text segment with the highest CXMI score from the sample-related texts .

[0064] Similarly, the preferred context segments of each sample related text corresponding to the sample query text are selected. Finally, the preferred context segments of each sample related text corresponding to the sample query text are aggregated to obtain the optimal context segment of the sample query text.

[0065] It should be noted that the method provided by the embodiment of the present invention can not only systematically evaluate the contribution of each sample-related text segment, but also accurately select sample-related text segments that have a significant impact on the performance of the sequence generation model. Therefore, it has obvious advantages in ensuring the relevance and accuracy of the generated content, especially in tasks that require highly accurate information support, such as machine translation, automatic question answering systems and other fields.

[0066] Based on any of the above embodiments, Figure 5 is a schematic diagram of the structure of the retrieval enhancement generation device based on sequence generation provided by the present invention, such as Figure 5 As shown, the device comprises: An acquisition unit 510 acquires a query text, searches the query text, and obtains candidate related texts; The initial prediction unit 520 obtains an initialization context sequence of the current round based on the query text, and generates a current prediction text block vector based on the initialization context sequence and a sequence generation model, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; The loop prediction unit 530 updates the initialization context sequence based on the current most relevant text sequence, generates a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, and matches the next round of most relevant text sequences from the candidate relevant texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; The generation unit 540 determines the target prompt word of the query text based on the obtained most relevant text sequence, and determines the enhanced response text of the query text based on the target prompt word; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0067] The device provided by the embodiment of the present invention retrieves the query text to obtain candidate related texts to ensure the diversity of prompt words. Then, the retrieved candidate related texts are optimized through the sequence generation model, and the retrieved candidate text fragments are used as basic units similar to words. The sequence generation algorithm is used to optimize and combine the candidate related texts, so that coherent and high-quality prompt words can be obtained, thereby improving the performance of downstream tasks based on prompt word fine-tuning or prompt learning.

[0068] Based on any of the above embodiments, the initial prediction unit is specifically used for: Generate a reply text vector of the query text based on the query text, the candidate related text and the sequence generation model; Based on the reply text vector, an initialization context sequence of the current round is obtained.

[0069] Based on any of the above embodiments, the candidate related texts include multiple candidate text segments with different text sources; The initial prediction unit is specifically used for: Calculating the semantic similarity between the current predicted text block vector and each candidate text segment respectively; Based on the calculated semantic similarities, the most relevant text sequence of the current round is matched from the candidate text segments.

[0070] Based on any of the above embodiments, the acquiring unit is specifically used for: Perform sparse search based on keywords in the query text to obtain key related text; Performing intensive retrieval based on the query text to obtain semantically relevant text; The candidate related text is obtained based on the key related text and the semantic related text.

[0071] Based on any of the above embodiments, the device further includes a training unit, and the training unit is specifically used for: Acquire the sample query text and a plurality of sample related texts corresponding to the sample query text; Calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text; Inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample prediction text block output by the initial sequence generation model; Based on the sample prediction text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model.

[0072] Based on any of the above embodiments, the training unit is further specifically used for: Segmenting the plurality of sample-related texts respectively to obtain sample-related text segments of each sample-related text; Calculating mutual information based on the sample query text and each sample-related text segment to obtain a contribution value of each sample-related text segment; Selecting a preferred context segment from each sample-related text based on the contribution value of each sample-related text segment; Aggregate the preferred context segments in the sample related texts to obtain the optimal context segment of the sample query text.

[0073] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute the retrieval enhancement generation method based on sequence generation, which includes: obtaining a query text, retrieving the query text, and obtaining candidate related texts; based on the query text, obtaining the initialization context sequence of the current round, and based on the initialization context sequence and the sequence generation model, generating a current prediction text block vector, so as to match the most relevant text sequence of the current round from the candidate related texts based on the current prediction text block vector; updating the initialization context sequence based on the current most relevant text sequence, and generating a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the next round from the candidate related texts based on the next prediction text block vector, until the obtained most relevant text sequence meets the preset conditions; based on the obtained most relevant text sequence, determining the target prompt word of the query text, so as to determine the enhanced response text of the query text based on the target prompt word; the sequence generation model is obtained by training the initial sequence generation model based on the sample query text.

[0074] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the retrieval enhancement generation method based on sequence generation provided by the above methods, which includes: obtaining a query text, retrieving the query text, and obtaining candidate related texts; based on the query text, obtaining an initialization context sequence of the current round, and based on the initialization context sequence and a sequence generation model, generating a current prediction text block vector, so as to match the most relevant text sequence of the current round from the candidate related texts based on the current prediction text block vector; updating the initialization context sequence based on the current most relevant text sequence, and generating a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the next round from the candidate related texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; based on the obtained most relevant text sequence, determining a target prompt word of the query text, so as to determine an enhanced response text of the query text based on the target prompt word; the sequence generation model is obtained by training an initial sequence generation model based on a sample query text.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the sequence-based retrieval enhancement generation method provided by the above methods when executed, the method comprising: obtaining a query text, retrieving the query text, and obtaining candidate related texts; based on the query text, obtaining an initialization context sequence of the current round, and generating a current prediction text block vector based on the initialization context sequence and a sequence generation model, so as to match the most relevant text sequence of the current round from the candidate related texts based on the current prediction text block vector; updating the initialization context sequence based on the current most relevant text sequence, and generating a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the next round from the candidate related texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; determining a target prompt word of the query text based on the obtained most relevant text sequence, so as to determine an enhanced reply text of the query text based on the target prompt word; the sequence generation model is obtained by training an initial sequence generation model based on a sample query text.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A retrieval enhancement generation method based on sequence generation, characterized in that: include: Obtaining a query text, searching the query text, and obtaining candidate related texts; Based on the query text, an initialization context sequence of the current round is obtained, and based on the initialization context sequence and a sequence generation model, a current prediction text block vector is generated, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; The initialization context sequence is updated based on the current most relevant text sequence, and a next predicted text block vector is generated based on the updated initialization context sequence and the sequence generation model, so as to match the most relevant text sequence of the next round from the candidate relevant texts based on the next predicted text block vector, until the obtained most relevant text sequence meets a preset condition; Determine a target prompt word of the query text based on the obtained most relevant text sequence, and determine an enhanced response text of the query text based on the target prompt word; The sequence generation model is obtained by training an initial sequence generation model based on sample query text.

2. The retrieval enhancement generation method based on sequence generation according to claim 1, characterized in that: The obtaining of the initialization context sequence of the current round based on the query text includes: Generate a reply text vector of the query text based on the query text, the candidate related text and the sequence generation model; Based on the reply text vector, an initialization context sequence of the current round is obtained.

3. The retrieval enhancement generation method based on sequence generation according to claim 1, characterized in that: The candidate related texts include a plurality of candidate text segments with different text sources; The matching of obtaining the most relevant text sequence of the current round from the candidate relevant texts based on the current predicted text block vector includes: Calculating the semantic similarity between the current predicted text block vector and each candidate text segment respectively; Based on the calculated semantic similarities, the most relevant text sequence of the current round is matched from the candidate text segments.

4. The retrieval enhancement generation method based on sequence generation according to any one of claims 1 to 3, characterized in that: The step of retrieving the query text to obtain candidate related texts includes: Perform sparse search based on keywords in the query text to obtain key related text; Performing intensive retrieval based on the query text to obtain semantically relevant text; The candidate related text is obtained based on the key related text and the semantic related text.

5. The retrieval enhancement generation method based on sequence generation according to any one of claims 1 to 3, characterized in that: The training steps of the sequence generation model include: Acquire the sample query text and a plurality of sample related texts corresponding to the sample query text; Calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text; Inputting the sample query text and the optimal context segment into the initial sequence generation model to obtain a sample prediction text block output by the initial sequence generation model; Based on the sample prediction text block and the optimal context segment, a prediction loss is obtained, and the initial sequence generation model is iterated based on the prediction loss to obtain the sequence generation model.

6. The retrieval enhancement generation method based on sequence generation according to claim 5, characterized in that: The step of calculating mutual information based on the sample query text and the plurality of sample related texts to obtain an optimal context segment of the sample query text includes: Segmenting the plurality of sample-related texts respectively to obtain sample-related text segments of each sample-related text; Calculating mutual information based on the sample query text and each sample-related text segment to obtain a contribution value of each sample-related text segment; Selecting a preferred context segment from each sample-related text based on the contribution value of each sample-related text segment; Aggregate the preferred context segments in the sample related texts to obtain the optimal context segment of the sample query text.

7. A retrieval enhancement generation device based on sequence generation, characterized in that: include: An acquisition unit acquires a query text, searches the query text, and obtains candidate related texts; An initial prediction unit obtains an initialization context sequence of the current round based on the query text, and generates a current prediction text block vector based on the initialization context sequence and a sequence generation model, so as to match the most relevant text sequence of the current round from the candidate relevant texts based on the current prediction text block vector; A cyclic prediction unit, which updates the initialization context sequence based on the current most relevant text sequence, generates a next prediction text block vector based on the updated initialization context sequence and the sequence generation model, and matches the next round of most relevant text sequences from the candidate relevant texts based on the next prediction text block vector, until the obtained most relevant text sequence meets a preset condition; A generating unit, which determines a target prompt word of the query text based on the obtained most relevant text sequence, and determines an enhanced response text of the query text based on the target prompt word; The sequence generation model is obtained by training an initial sequence generation model based on sample query text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the retrieval enhancement generation method based on sequence generation according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the retrieval enhancement generation method based on sequence generation as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the retrieval enhancement generation method based on sequence generation as claimed in any one of claims 1 to 6 is implemented.

Images