Abstract consistency evaluation method and device based on adaptive prompt template

Abstract

The invention provides an abstract consistency evaluation method and device based on a self-adaptive prompt template. The abstract consistency evaluation method based on the self-adaptive prompt template comprises the steps of obtaining a source document and a to-be-evaluated abstract and inputting the source document and the to-be-evaluated abstract into a template construction layer; the template construction layer selects an optimal template form based on a source document and a to-be-evaluated abstract; the template construction layer determines a prompt template based on the optimal template form, wherein the prompt template at least comprises discontinuous blank information; splicing the source document, the abstract to be evaluated and the prompt template to form mixed input of a pre-training abstract consistency evaluation model; inputting the mixture into a mask language model layer, predicting discontinuous blank information in a prompt template, and outputting a learning result; and determining a consistency evaluation result of the source document and the abstract to be evaluated through the label mapping layer and the learning result. According to the method and the device, the text abstract fact consistency problem can be accurately evaluated under the condition of few manually labeled samples.

Description

Technical Field

[0001] The present application relates to the technical field of natural language understanding, and in particular to a summary consistency evaluation method and device. Background Art

[0002] In recent years, generative summarization has been used to generate summaries through natural language generation technologies such as language models, information fusion, and information compression. In particular, the sequence-to-sequence encoder-decoder framework has achieved good results in generative summarization models, and can generate fluent and readable summaries to a certain extent. However, there is an obvious problem with generative summaries. In the process of extracting key information and restating it, there may be a large number of inconsistencies with the original facts, and the generated summaries need to be evaluated for consistency with the original facts.

[0003] At present, the academic and industrial communities mainly adopt the "pre-training-fine-tuning" paradigm to apply pre-trained models to downstream tasks of natural language reasoning, including automatic text summarization. This model first pre-trains the model for a specific task through a large-scale corpus to learn semantic relations and knowledge, and then uses the corresponding dataset of the downstream task for fine-tuning training. However, with the continuous increase in the scale of pre-trained language models and the sharp increase in the number of parameters, the requirements for computing resources for fine-tuning of downstream tasks are also increasing. Moreover, the variety of downstream tasks in natural language processing far exceeds the way the model learns in pre-training, which leads to differences between pre-training tasks and downstream tasks, thereby reducing the effect of fine-tuning in downstream tasks. Pre-trained language models also require a large amount of data for self-supervised learning in the fine-tuning stage, but the factual consistency evaluation task of generative text summarization lacks large-scale manually annotated datasets, making it difficult to automatically discover a large number of factual inconsistencies in text summaries. Summary of the invention

[0004] In view of this, the present application provides a summary consistency evaluation method and device based on an adaptive prompt template, which can provide prompts for the evaluation process by extracting knowledge from the text and matching it to the optimal template in the case of a small number of manually annotated samples, thereby more accurately evaluating the factual consistency problem of the text summary.

[0005] Specifically, the present application is implemented through the following technical solutions: In a first aspect, the present application provides a summary consistency evaluation method, the method is implemented based on a summary consistency evaluation model, the summary consistency evaluation model at least includes a template construction layer, a masked language model (Masked Language Model, MLM) layer and a label mapping layer; The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer; The template construction layer selects an optimal template form based on the source document and the summary to be evaluated; The template construction layer determines a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information; splicing the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model; Inputting the mixture into a masked language model layer, predicting discontinuous blank information in the prompt template, and outputting a learning result; The consistency evaluation result of the source document and the summary to be evaluated is determined through the label mapping layer and the learning result.

[0006] The second aspect of the present application provides a summary consistency evaluation device, the device comprising an acquisition module, a selection module, a determination module, a processing module, a prediction module and an evaluation module; wherein: The acquisition module is used to acquire the source document and the summary to be evaluated and input them into the template construction layer; The selection module is used for the template construction layer to select the optimal template form based on the source document and the summary to be evaluated; The determination module is used for the template construction layer to determine a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information; The processing module is used to splice the source document, the summary to be evaluated and the prompt template to form a mixed input of the pre-trained summary consistency evaluation model; The prediction module is used to input the mixture into the mask language model layer, predict the discontinuous blank information in the prompt template, and output the learning result; The evaluation module is used to determine the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result.

[0007] The summary consistency evaluation method and device provided in the present application are implemented based on a summary consistency evaluation model, and the summary consistency evaluation model at least includes a template construction layer, a mask language model layer and a label mapping layer; the summary consistency evaluation method specifically includes: by obtaining a source document and a summary to be evaluated and inputting them into a template construction layer, the template construction layer then selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a prompt template based on the optimal template form, the prompt template at least includes discontinuous blank information, and then splices the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model, and the mixed input is input to the mask language model layer, predicts the discontinuous blank information in the prompt template, outputs a learning result, and finally determines the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed in the template construction layer according to the optimal template form, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new field can be prompted as input data close to the field during pre-training, thereby reconstructing a form that can make better use of the pre-trained summary consistency evaluation model, and predict the output learning results through the masked language model layer. Finally, the text summary fact consistency evaluation result is obtained by comparing with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt specific prediction tasks to tasks that are learned in large quantities in the pre-training stage through the prompt of the optimal template, so that the text summary fact consistency problem can be evaluated more accurately without obtaining a large number of learning samples in specific fields. The present invention constructs the optimal prompt template by acquiring knowledge from the text, and obtains the best model evaluation prompt information based on the template, thereby improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Flow chart of Embodiment 1 of the summary consistency evaluation method based on the adaptive prompt template provided in the present application; Figure 2 This is a structural diagram of Embodiment 1 of a summary consistency assessment device based on an adaptive prompt template provided by the present application. DETAILED DESCRIPTION

[0009] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0010] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0011] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0012] The present application provides a summary consistency evaluation method and device based on an adaptive prompt template, which can more accurately evaluate the factual consistency problem of a text summary in the case of a small number of manually annotated samples.

[0013] The summary consistency evaluation method and device based on adaptive prompt template provided by the present application are implemented based on a summary consistency evaluation model, and the summary consistency evaluation model at least includes a template construction layer, a mask language model layer and a label mapping layer; the summary consistency evaluation method specifically includes: by obtaining a source document and a summary to be evaluated and inputting them into a template construction layer, and then the template construction layer selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information, and then splices the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model, and the mixed input is input to the mask language model layer, predicts the discontinuous blank information in the prompt template, outputs a learning result, and finally determines the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed in the template construction layer according to the optimal template form, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new field can be prompted as input data close to the field during pre-training, thereby reconstructing a form that can make better use of the pre-trained summary consistency evaluation model, and the masked language model layer is used to predict and output the learning results. Finally, the text summary fact consistency evaluation result is obtained by comparing with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt specific prediction tasks to tasks that are learned in large quantities in the pre-training stage through the prompt of the optimal template, so that the text summary fact consistency problem can be evaluated more accurately without obtaining a large number of learning samples in specific fields. And the prompt template form is given priority to the cloze form, which can have a priori advantage during pre-training of the masked language model layer.

[0014] Specific embodiments are given below to introduce the technical solution of the present application in detail.

[0015] Figure 1 This is a flowchart of the first embodiment of the summary consistency evaluation method based on the adaptive prompt template provided by this application. Please refer to Figure 1 The method is implemented based on a summary consistency evaluation model, which includes at least a template construction layer, a mask language model layer, and a label mapping layer; the method provided in this embodiment may include: S101, obtaining a source document and a summary to be evaluated and inputting them into a template construction layer.

[0016] It should be noted that the template construction layer is constructed using a hybrid template, wherein the hybrid template combines template elements of multiple different types or sources. Specifically, the template elements included in the hybrid template are selected according to actual needs and are not limited in this embodiment. For example, in one embodiment, the template elements included in the hybrid template may come from a predefined template library, a template automatically generated from data, a template generated based on a rule or heuristic method, etc.

[0017] Specifically, the construction of a hybrid template may involve operations such as splicing, superposition, and insertion of template elements to ensure that the generated hybrid template has good applicability and effect in different tasks and text situations.

[0018] S102: The template construction layer selects an optimal template form based on the source document and the summary to be evaluated.

[0019] The template construction layer is used to determine the form of the optimal template based on the semantic information of the source document and the abstract to be evaluated, and to determine the optimal template according to the form of the optimal template and the content of the source document and the abstract to be evaluated. The optimal template is used to describe the evaluation task of the source document and the abstract to be evaluated as a pre-training task in the pre-training stage of the abstract evaluation consistency model. It should be noted that the template construction layer will have an in-depth understanding of the source document and the abstract to be evaluated, considering its context, theme, key information and other factors to determine the content and objectives that the template needs to cover. Furthermore, the template construction layer designs templates of different forms based on contextual understanding. Specifically, the template is designed according to the characteristics and requirements of the task, which is not limited in this embodiment. For example, in one embodiment, the template may be in the form of a cloze test; for another example, in one embodiment, the template may be in the form of a question and answer; for another example, in one embodiment, the template may be in the form of a structured expression.

[0020] After designing the templates, the template construction layer evaluates each template to determine its applicability and effectiveness. Specifically, the evaluation may include the degree of matching between the template and the source document, the summary to be evaluated, the accuracy of information extraction, the clarity of language expression, etc. Based on the evaluation results, the template construction layer selects the template form that best suits the current task, that is, the optimal template form.

[0021] Specifically, the optimal template form includes at least a cloze form, a question-and-answer form, and a structured expression form.

[0022] The prior art usually uses prompts in the learning process, so that tasks with a small number of samples are predicted during the learning and training process, so that the model has the ability to perform tasks in other fields. However, when the downstream tasks are executed, if tasks in new fields are encountered, the performance of the model is still poor. To this end, the method provided by the present invention uses a template containing prompt information when executing the task, and based on the optimal template, the currently executed downstream task can be converted into a training task with a large number of samples in the pre-training stage, thereby making full use of the capabilities that the model has learned, and then obtaining accurate evaluation results, without the need to use a large number of samples for training in advance.

[0023] It should be noted that the template construction layer selects the optimal template form based on the source document and the summary to be evaluated, and further includes: (1) Match the suggested template form based on the pre-training method of the summary generation model.

[0024] It should be noted that if the pre-training method of the summary generation model is bidirectional training, so that the summary generation model has bidirectional encoding capabilities, then the suggested template form is matched based on the pre-training method of the summary generation model as a cloze form. Bidirectional encoding is good at completing cloze-type tasks, so different model training methods enable the model to have different capabilities. In order to maximize the capabilities of the model, the present invention also determines the suggested template form through a training method, and further optimizes the initially determined template form, so that the provided template can not only complete the translation of the task, but also maximize the capabilities of the model, that is, let the model do the task it is best at, thereby improving the accuracy and efficiency of model evaluation.

[0025] (2) Matching candidate template forms based on the source document and the summary to be evaluated.

[0026] It should be noted that the candidate template form can be pre-defined or dynamically generated according to a specific task, which is not limited in this embodiment.

[0027] (3) Determine the optimal template form based on the candidate template forms and the suggested template form.

[0028] It should be noted that a matching or scoring mechanism can be used to compare the similarity or adaptability between the candidate template form and the suggested template form. Specifically, the optimal template form can be determined based on the comparison of semantics, structure, etc. between the template forms.

[0029] As an optional embodiment, determining the optimal form at least includes: Compare the fields of the candidate template form and the suggested template form, and mark the difference fields; Calculate the importance score of the difference field based on the difference field and the source document; Based on a preset threshold, the difference fields with importance scores higher than the preset threshold are screened and added to the candidate template form to form the optimal form. Furthermore, the supplement position is determined based on the semantics of the difference fields to be supplemented and the semantics of the candidate template. The method provided by the present invention not only considers the task translation, but also fully considers the capabilities of the model, thereby constructing a template based on two factors, allowing the model to do what it is familiar with and good at, thereby maximizing the accuracy of the evaluation and giving full play to the capabilities of the model.

[0030] S103: The template construction layer determines a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information.

[0031] In combination with the above example, if the template construction layer determines that the prompt template is a cloze form based on the optimal template form, then the prompt template at least includes discontinuous blank information, and the discontinuous blank information is used to indicate the position to be filled in the template.

[0032] Specifically, these discontinuous blank information can be regarded as markers or placeholders of the prompt template, which are used to indicate the location that the model needs to pay attention to when processing the text. It should be noted that the specific content that needs to be filled in the discontinuous blank information is determined according to the specific task and text content. For example, the discontinuous blank information can be filled with entities, keywords, phrases, etc.

[0033] In specific implementation, the template construction layer determines the prompt template based on the optimal template form, specifically including: (1) A template task is defined according to the source document and the summary to be evaluated.

[0034] Define the language model as M, and the input source document is represented as , the summary to be evaluated is represented as , the template is ,definition Denote the word embedding of D, defined Representing the word embedding of S, the complete input consists of: , in Part of it is a solvable template vector, whose relative position to the original text and label words in the input can be freely determined.

[0035] (2) Randomly initializing a mixed template vector; the mixed template vector is a vector composed of a mixture of the source document, the summary to be evaluated, and a vector of a template defined according to the source document and the summary to be evaluated.

[0036] Initialize the mixed template vector to generate a vector of dimension (x, 768).

[0037] (3) By optimizing the loss function, some parameters in the mixed template vector are learned to obtain the prompt template vector.

[0038] Specifically, using the downstream loss function and the stochastic gradient descent method, the parameters corresponding to the template part are continuously optimized through training to obtain the prompt template vector: .

[0039] It should be noted that the downstream loss function refers to the loss function defined on a specific task, such as the cross entropy loss function or mean square error loss function that may be used in the text summary consistency evaluation task. This loss function is used to measure the difference between the model prediction result and the true label, and is a performance indicator of the model on this task. Stochastic gradient descent is an optimization algorithm used to minimize the loss function. It updates the model parameters iteratively so that the loss function gradually decreases. In each iteration, stochastic gradient descent calculates the gradient of the loss function to the parameters and updates the parameter values ​​in the opposite direction of the gradient to reduce the loss function. The optimal template generated is a document containing discontinuous blanks. The blank positions are used for prediction based on the source document and the summary. The positions with content are also obtained based on the source document and the summary, thereby making the consistency evaluation task of the source document and the summary a task that the model is better at through the description of the template, thereby improving the accuracy of the evaluation.

[0040] S104: Concatenate the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency assessment model.

[0041] After obtaining the prompt template vector, the source document, the summary to be evaluated and the prompt template are concatenated to form a mixed input of the pre-trained summary consistency evaluation model. In this way, these information can be combined into a complete input sequence so that the summary consistency evaluation model can simultaneously consider the content of the source document, the summary information of the summary and the guiding role of the prompt template, which can better guide the summary consistency evaluation model to perform summary consistency evaluation tasks and improve the performance and effect of the summary consistency evaluation model.

[0042] S105: Input the mixture into a mask language model layer, predict discontinuous blank information in the prompt template, and output a learning result.

[0043] It should be noted that the masked language model layer includes a BERT-MLM layer and a RoBERTa-MLM layer, and the BERT-MLM layer and the RoBERTa-MLM layer are used to complete the cloze task, specifically including: (1) The source document and the summary to be evaluated are taken as input sequences and input into the BERT-MLM layer and the RoBERTa-MLM layer after performing a mask operation.

[0044] It should be noted that the source document and the summary to be evaluated are used as input sequences, where the vacant position is represented by a special [MASK] tag. For example, in one embodiment, if the template is "the [MASK] jumped over the fence", the vacant position is [MASK]. These input sequences will be input into the BERT-MLM layer or the RoBERTa-MLM layer for processing.

[0045] It should be noted that the BERT-MLM layer completes two tasks, namely the MLM task and the NSP task. The MLM task is similar to the "cloze test". Specifically, the BERT-MLM layer masks 15% of the words in a sentence, completes the embedding encoding of each word through bidirectional transformer+feedforward+rediual_add+layer_norm, and then predicts the masked word. The prediction process is equivalent to multi-classification. The number of categories is the total number of words. The embedding of the masked word is transformed by MLP to generate the logits probability on each category of words. The label is the one-hot encoding of the real word at the mask position in the entire vocabulary. The cross entropy of logits and labels is calculated, and the weighted average is performed to obtain the MLM loss.

[0046] The RoBERTa-MLM layer changes BERT's static mask to a dynamic mask. Specifically, BERT stores the corpus after masking it, and the model repeatedly learns the same data in each epoch. RoBERTa copies the data 10 times, performs different random masks on each copy, and inputs the processed data into the model for learning.

[0047] (2) Define a set of candidate words and calculate the scores corresponding to the candidate words through the BERT-MLM layer and the RoBERTa-MLM layer.

[0048] Specifically, define the candidate word set For each source document and the summary to be evaluated is represented as , the template is , It represents the score of the language model when the vacant position is W. Therefore, the score corresponding to each candidate word is: .

[0049] (3) The scores corresponding to the candidate words obtained by the calculation are converted into probability distribution, and the candidate word with the highest probability is selected to fill in the blank to complete the cloze task.

[0050] Specifically, the calculated scores corresponding to the candidate words can be converted into probability distributions through the Softmax function, and the candidate words with the highest probability are selected to fill in the blanks to complete the cloze task.

[0051] It should be noted that BERT-MLM and RoBERTa-MLM are both pre-trained language models based on the Transformer architecture. In this embodiment, by using BERT-MLM and RoBERTa-MLM at the same time, the calculated corresponding scores of candidate words can be more accurate, providing more accurate guarantee for the judgment of the summary fact consistency model.

[0052] S106: Determine a consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result.

[0053] After obtaining the words for the missing parts, the results are input into the label mapping layer. The main function of the label mapping layer is to define the correspondence between candidate words and labels in the masked language model task. For example, according to different tasks, sets such as {YES, NO}, {CORRECT, INCORRECT} are used as candidate words for cloze test. For example, the consistent candidate words are: "yes", "correct", "consistent", and the inconsistent candidate words are: "no", "incorrect", "inconsistent". Therefore, the fact consistency evaluation task can select words from the candidate word set by prediction, and then obtain the final label through the mapping relationship.

[0054] It should be noted that the definition of the correspondence between candidate words and labels through the label mapping layer includes: (1) The label mapping layer receives candidate words for completing the cloze task in the masked language model layer; (2) Presetting a candidate word set and an evaluation label set; wherein the candidate word set is the same as the candidate word set defined in the masked language model layer; (3) The probability of the candidate word calculated in the masked language model layer on the candidate word set is mapped to the probability of the evaluation label set.

[0055] When the masked language model layer predicts that the [MASK] part is in the candidate word set When the probability distribution on , the word probability in the label word set W is mapped to the probability of the evaluation label set. The specific calculation method is as follows: .

[0056] In addition, the summary consistency evaluation method at least further comprises: pre-training the masked language model layer and the label mapping layer based on the label samples, and the pre-training process of the masked language model layer and the label mapping layer comprises: (1) Randomly select a preset number of positions to perform masking operations on the input sequence of input samples of the masked language model layer.

[0057] Combined with the above description, during the pre-training process, a preset number of positions are randomly selected to perform mask operations on the input sequence of the input sample. These positions are usually replaced with a special "[MASK]" tag.

[0058] (2) The masked input sequence is input into a masked language model layer, and the masked language model layer predicts the masked position.

[0059] It should be noted that the masked input sequence is input into the masked language model layer, which will predict the masked position, that is, try to predict the word at the masked position. The output of the masked language model layer is the probability distribution of the word at each masked position.

[0060] (3) Inputting the predicted result of the masked language model layer into the label mapping layer to obtain the predicted label.

[0061] (4) Compare the predicted label with the actual label word of the label sample, calculate the loss function, and optimize the pre-trained parameters of the masked language model layer and the label mapping layer.

[0062] The predicted label is compared with the true label word of the label sample, and a cross entropy loss function is usually used to measure the difference between the predicted result and the true label.

[0063] Specifically, after obtaining the pre-trained parameters, the method further includes: (1) Fixing the pre-trained parameters of the label mapping layer.

[0064] (2) Taking the minimization of the loss function of the masked language model layer as the optimization goal, the back-propagation algorithm and optimizer are used to update the parameters of the masked language model layer, and the training is repeated until the pre-trained summary consistency evaluation model converges or reaches a pre-set number of training rounds.

[0065] In specific implementation, the parameters of the masked language model layer are updated by minimizing the cross entropy loss between the factual consistency labels mapped by the [MASK] candidate set in the prediction template and the true label words: , The cross entropy loss is calculated by the above formula , where c represents the number of factual consistency categories, represents the manually annotated consistency category, Indicates the category after label mapping.

[0066] The summary consistency evaluation method based on adaptive prompt template provided in this embodiment is implemented based on a summary consistency evaluation model, and the summary consistency evaluation model at least includes a template construction layer, a mask language model layer and a label mapping layer; the summary consistency evaluation method specifically includes: by obtaining a source document and a summary to be evaluated and inputting them into a template construction layer, then the template construction layer selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a prompt template based on the optimal template form, the prompt template at least includes discontinuous blank information, and then splices the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model, and the mixed input is input to the mask language model layer, predicts the discontinuous blank information in the prompt template, outputs a learning result, and finally determines the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed in the template construction layer according to the optimal template form, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new field can be prompted as input data close to the field during pre-training, thereby reconstructing a form that can make better use of the pre-trained summary consistency evaluation model, and the learning result is predicted and output through the masked language model layer. Finally, the text summary fact consistency evaluation result is obtained by comparing with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt a specific prediction task to a task that has been learned in large quantities in the pre-training stage through the prompt of the optimal template, so that the text summary fact consistency problem can be evaluated more accurately, and at the same time, there is no need to obtain a large number of learning samples in a specific field. In addition, the method provided in this embodiment also uses a new paradigm of pre-training summary consistency model (Pre-train, Prompt, and Predict) to detect the problem of text summary fact inconsistency. By designing "prompt text" and reformulating downstream tasks to make them close to the training method of language models during pre-training, pre-training can be performed on a large amount of original text. By designing a prompt function, the model can act on few-shot or zero-shot samples and adapt to small amounts of or no labeled data in other scenarios. This can solve many tasks, reduce the difference between the pre-training stage and downstream tasks, and achieve significant results in the task of detecting factual consistency in text summaries.

[0067] Corresponding to the aforementioned embodiment of a summary consistency evaluation method based on an adaptive prompt template, the present application also provides an embodiment of a summary consistency evaluation device based on an adaptive prompt template.

[0068] Figure 2 This is a structural diagram of the first embodiment of the summary consistency evaluation device based on the adaptive prompt template provided by this application. Figure 2 The device provided in this embodiment includes an acquisition module 210, a selection module 220, a determination module 230, a processing module 240, a prediction module 250 and an evaluation module 260; wherein, The acquisition module 210 is used to acquire the source document and the summary to be evaluated and input them into the template construction layer; The selection module 220 is used for the template construction layer to select the optimal template form based on the source document and the summary to be evaluated; The determination module 230 is used for the template construction layer to determine a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information; The processing module 240 is used to splice the source document, the summary to be evaluated and the prompt template to form a mixed input of the pre-trained summary consistency evaluation model; The prediction module 250 is used to input the mixture into the mask language model layer, predict the discontinuous blank information in the prompt template, and output the learning result; The evaluation module 260 is used to determine the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result.

[0069] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0070] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0071] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0072] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A summary consistency evaluation method based on an adaptive prompt template, characterized in that: The method is implemented based on a summary consistency evaluation model, which at least includes a template construction layer, a mask language model layer and a label mapping layer; The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer; The template construction layer selects an optimal template form based on the source document and the summary to be evaluated; The template construction layer determines a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information; splicing the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model; Inputting the mixture into a masked language model layer, predicting discontinuous blank information in the prompt template, and outputting a learning result; The consistency evaluation result of the source document and the summary to be evaluated is determined through the label mapping layer and the learning result.

2. The method according to claim 1, characterized in that: The optimal template form at least includes a cloze form, a question-and-answer form, and a structured expression form; The template construction layer selects the optimal template form based on the source document and the summary to be evaluated, and further includes: The suggested template form is matched based on the pre-training method of the summary generation model; Matching candidate template forms based on the source document and the summary to be evaluated; An optimal template form is determined based on the candidate template forms and the suggested template form.

3. The method according to claim 2, characterized in that If the pre-training method of the summary generation model is bidirectional training, so that the summary generation model has bidirectional encoding capability, then the suggested template form is matched based on the pre-training method of the summary generation model as a cloze form.

4. The method according to claim 1, characterized in that: The template construction layer determines the prompt template based on the optimal template form, specifically including: defining a template task according to the source document and the summary to be evaluated; Randomly initialize a mixed template vector; the mixed template vector is a vector composed of a mixture of the source document, the summary to be evaluated, and a vector defining a template according to the source document and the summary to be evaluated; Part of the parameters in the mixed template vector are learned by optimizing the loss function to obtain the prompt template vector.

5. The method according to claim 1, characterized in that The masked language model layer includes a BERT-MLM layer and a RoBERTa-MLM layer, and the BERT-MLM layer and the RoBERTa-MLM layer are used to complete the cloze task, specifically including: The source document and the summary to be evaluated are used as input sequences and input into the BERT-MLM layer and the RoBERTa-MLM layer after performing mask operations; Define a candidate word set and calculate the scores corresponding to the candidate words through the BERT-MLM layer and the RoBERTa-MLM layer; The scores corresponding to the candidate words obtained by the calculation are converted into probability distribution, and the candidate word with the largest probability is selected to fill in the blank to complete the cloze task.

6. The method according to claim 1, characterized in that Defining the correspondence between candidate words and labels through the label mapping layer includes: The label mapping layer receives candidate words for completing the cloze task in the masked language model layer; Preset a candidate word set and an evaluation label set; wherein the candidate word set is the same as the candidate word set defined in the masked language model layer; The probability of the candidate words calculated in the masked language model layer on the candidate word set is mapped to the probability of the evaluation label set.

7. The method according to claim 1, characterized in that The summary consistency evaluation method at least further comprises: pre-training the masked language model layer and the label mapping layer based on the label samples, wherein the pre-training process of the masked language model layer and the label mapping layer comprises: Randomly select a preset number of positions to perform a masking operation on the input sequence of the input samples of the masked language model layer; Inputting the masked input sequence into a masked language model layer, wherein the masked language model layer predicts the masked position; Inputting the predicted result of the masked language model layer into the label mapping layer to obtain a predicted label; The predicted label is compared with the actual label word of the label sample, and the loss function is calculated to obtain the pre-trained parameters of the masked language model layer and the label mapping layer.

8. The method according to claim 7, characterized in that After obtaining the pre-trained parameters, the method further includes: Fixing the pre-trained parameters of the label mapping layer; Taking the minimization of the loss function of the masked language model layer as the optimization goal, the back-propagation algorithm and optimizer are used to update the parameters of the masked language model layer, and the training is repeated until the pre-trained summary consistency evaluation model converges or reaches a preset number of training rounds.

9. The method according to claim 2, characterized in that: Determining the optimal form at least includes: Compare the fields of the candidate template form and the suggested template form, and mark the difference fields; Calculate the importance score of the difference field based on the difference field and the source document; Based on the preset threshold, the difference fields with importance scores higher than the preset threshold are filtered and added to the suggested template form to form the optimal form.

10. A summary consistency evaluation device based on an adaptive prompt template, characterized in that: The device includes an acquisition module, a selection module, a determination module, a processing module, a prediction module and an evaluation module; wherein, The acquisition module is used to acquire the source document and the summary to be evaluated and input them into the template construction layer; The selection module is used for the template construction layer to select the optimal template form based on the source document and the summary to be evaluated; The determination module is used for the template construction layer to determine a prompt template based on the optimal template form, and the prompt template at least includes discontinuous blank information; The processing module is used to splice the source document, the summary to be evaluated and the prompt template to form a mixed input of the pre-trained summary consistency evaluation model; The prediction module is used to input the mixture into the mask language model layer, predict the discontinuous blank information in the prompt template, and output the learning result; The evaluation module is used to determine the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result.

Images