A summary consistency evaluation method and device based on adaptive prompt template

Abstract

The present application provides a summary consistency evaluation method and device based on an adaptive prompt template. The summary consistency evaluation method based on an adaptive prompt template provided by the present application includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer; the template construction layer selects the optimal template form based on the source document and the summary to be evaluated; the template construction layer determines a prompt template based on the optimal template form, and the prompt template includes at least discontinuous blank information; splicing the source document, the summary to be evaluated and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model; inputting the mixed input to a masked language model layer, predicting the discontinuous blank information in the prompt template, and outputting the learning result; determining the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. The present application can more accurately evaluate the factual consistency problem of text summaries in the case of a small number of manually annotated samples.

Description

Technical Field

[0001] The present application relates to the technical field of natural language understanding, and in particular to a method and device for evaluating summary consistency. Background Art

[0002] In recent years, generative summarization has been developed through natural language generation techniques such as language models, information fusion, and information compression. In particular, the sequence-to-sequence encoder-decoder framework has achieved promising results in generative summarization models, capable of generating fluent and readable summaries to a certain extent. However, a significant issue with generative summarization is that the process of extracting key information and restating it may contain numerous inconsistencies with the original text, necessitating an assessment of the factual consistency of the generated summaries with the original text.

[0003] Currently, academia and industry primarily employ a "pretrain-fine-tune" paradigm to apply pretrained models to downstream tasks of natural language inference, including automatic text summarization. This model first pretrains the model for a specific task using a large-scale corpus to learn semantic relationships and knowledge, and then fine-tunes the model using the corresponding dataset for the downstream task. However, with the increasing size and rapid growth in the number of parameters of pretrained language models, the computational resources required for fine-tuning downstream tasks are also increasing. Furthermore, the variety of downstream tasks in natural language processing far exceeds the types of learning models performed during pretraining, leading to discrepancies between pretrained and downstream tasks and reducing the effectiveness of fine-tuning in downstream tasks. Pretrained language models also require a large amount of data for self-supervised learning during the fine-tuning phase. However, the factual consistency assessment task of generative text summarization lacks large-scale, manually annotated datasets, making it difficult to automatically detect numerous factual inconsistencies in text summaries. Summary of the Invention

[0004] In view of this, the present application provides a summary consistency evaluation method and device based on an adaptive prompt template, which can provide prompts for the evaluation process by extracting knowledge from the text and matching it to the optimal template in the case of a small number of manually annotated samples, thereby more accurately evaluating the factual consistency problem of the text summary.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] In a first aspect, the present application provides a summary consistency evaluation method, which is implemented based on a summary consistency evaluation model. The summary consistency evaluation model includes at least a template construction layer, a masked language model (MLM) layer, and a label mapping layer.

[0007] The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer;

[0008] The template construction layer selects an optimal template form based on the source document and the abstract to be evaluated;

[0009] The template construction layer determines a prompt template based on the optimal template form, wherein the prompt template at least includes discontinuous blank information;

[0010] splicing the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model;

[0011] Inputting the mixture into the masked language model layer, predicting discontinuous blank information in the prompt template, and outputting a learning result;

[0012] The consistency evaluation result of the source document and the summary to be evaluated is determined through the label mapping layer and the learning result.

[0013] A second aspect of the present application provides a summary consistency evaluation device, the device comprising an acquisition module, a selection module, a determination module, a processing module, a prediction module and an evaluation module; wherein,

[0014] The acquisition module is used to acquire the source document and the summary to be evaluated and input them into the template construction layer;

[0015] The selection module is used for the template construction layer to select the optimal template form based on the source document and the abstract to be evaluated;

[0016] The determining module is used for the template construction layer to determine a prompt template based on the optimal template form, wherein the prompt template at least includes discontinuous blank information;

[0017] The processing module is configured to combine the source document, the summary to be evaluated, and the prompt template to form a mixed input for a pre-trained summary consistency assessment model;

[0018] The prediction module is used to input the mixture into the masked language model layer, predict the discontinuous blank information in the prompt template, and output the learning result;

[0019] The evaluation module is configured to determine a consistency evaluation result between the source document and the summary to be evaluated through a label mapping layer and the learning result.

[0020] The summary consistency evaluation method and device provided in the present application are implemented based on a summary consistency evaluation model, which includes at least a template construction layer, a mask language model layer, and a label mapping layer. The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into the template construction layer, then the template construction layer selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a prompt template based on the optimal template form, the prompt template includes at least discontinuous blank information, and then splicing the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model, and the mixed input is input to the mask language model layer, predicting the discontinuous blank information in the prompt template, outputting a learning result, and finally determining the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed in the template construction layer according to the optimal template form, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new field can be prompted as input data close to the field during pre-training, thereby reconstructing a form that can make better use of the pre-trained summary consistency evaluation model, and the learning result is predicted and output through the masked language model layer. Finally, the text summary fact consistency evaluation result is obtained by comparing with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt a specific prediction task to a task that has been learned a lot in the pre-training stage through the prompt of the optimal template, so that the text summary fact consistency problem can be evaluated more accurately without obtaining a large number of learning samples in a specific field. The present invention constructs the optimal prompt template by acquiring knowledge from the text, and obtains the best model evaluation prompt information based on the template, thereby improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flowchart of Example 1 of the summary consistency assessment method based on the adaptive prompt template provided in this application;

[0022] Figure 2 This is a structural diagram of the first embodiment of the summary consistency evaluation device based on the adaptive prompt template provided by this application. DETAILED DESCRIPTION

[0023] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0024] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] The present application provides a summary consistency assessment method and device based on an adaptive prompt template, which can more accurately assess the factual consistency of text summaries with a small number of manually annotated samples.

[0027] The present application provides a summary consistency evaluation method and device based on an adaptive prompt template, which are implemented based on a summary consistency evaluation model. The summary consistency evaluation model includes at least a template construction layer, a mask language model layer, and a label mapping layer. The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer, then the template construction layer selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a prompt template based on the optimal template form, the prompt template includes at least discontinuous blank information, and then splicing the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model, and the mixed input is input to the mask language model layer, predicting the discontinuous blank information in the prompt template, outputting a learning result, and finally determining the consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed according to the optimal template form at the template construction layer, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new field can be prompted as input data close to the field during pre-training, thereby reconstructing a form that can make better use of the pre-trained summary consistency evaluation model, and the learning result is predicted and output through the masked language model layer. Finally, the text summary fact consistency evaluation result is obtained by comparing with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt a specific prediction task to a task that has been learned a lot in the pre-training stage through the prompt of the optimal template, so that the text summary fact consistency problem can be evaluated more accurately without obtaining a large number of learning samples in a specific field. And the prompt template form is given priority to be a cloze form, which can have a priori advantage when pre-training the masked language model layer.

[0028] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0029] Figure 1 This is a flowchart of the first embodiment of the summary consistency evaluation method based on the adaptive prompt template provided by this application. Please refer to Figure 1 The method is implemented based on a summary consistency evaluation model, which includes at least a template construction layer, a mask language model layer, and a label mapping layer. The method provided in this embodiment may include:

[0030] S101: Obtain a source document and a summary to be evaluated and input them into a template construction layer.

[0031] It should be noted that the template construction layer is constructed using a hybrid template, wherein the hybrid template combines template elements of multiple different types or sources. Specifically, the template elements included in the hybrid template are selected based on actual needs and are not limited in this embodiment. For example, in one embodiment, the template elements included in the hybrid template can come from a predefined template library, a template automatically generated from data, a template generated based on rules or heuristic methods, etc.

[0032] Specifically, the construction of a hybrid template may involve operations such as splicing, overlaying, and inserting template elements to ensure that the generated hybrid template has good applicability and effectiveness in different tasks and text contexts.

[0033] S102: The template construction layer selects an optimal template form based on the source document and the abstract to be evaluated.

[0034] The template construction layer is used to determine the form of the optimal template based on the semantic information of the source document and the abstract to be evaluated, and to determine the optimal template according to the form of the optimal template and the content of the source document and the abstract to be evaluated. The optimal template is used to describe the evaluation task of the source document and the abstract to be evaluated as a pre-training task in the pre-training stage of the abstract evaluation consistency model. It should be noted that the template construction layer will have an in-depth understanding of the source document and the abstract to be evaluated, considering its context, theme, key information and other factors to determine the content and objectives that the template needs to cover. Furthermore, the template construction layer designs templates of different forms based on contextual understanding. Specifically, the template is designed according to the characteristics and requirements of the task, which is not limited in this embodiment. For example, in one embodiment, the template may be in the form of a cloze test; for another example, in one embodiment, the template may be in the form of a question and answer; for another example, in one embodiment, the template may be in the form of a structured expression.

[0035] After designing the templates, the template construction layer evaluates each one to determine its suitability and effectiveness. Specifically, the evaluation may include aspects such as the template's match with the source document and the abstract being evaluated, the accuracy of information extraction, and the clarity of language expression. Based on the evaluation results, the template construction layer selects the template form that best suits the task at hand—the optimal template form.

[0036] Specifically, the optimal template form includes at least a cloze form, a question-and-answer form, and a structured expression form.

[0037] Existing technologies typically use hints during the learning process, enabling predictions for tasks with a small number of samples during the learning and training process, thereby enabling the model to perform tasks in other domains. However, when executing downstream tasks, if a task in a new domain is encountered, the model's performance is still poor. To this end, the method provided by the present invention uses a template containing hint information during task execution. Based on the optimal template, the currently executed downstream task can be converted into a training task with a large number of samples in the pre-training phase, thereby fully utilizing the capabilities already learned by the model and obtaining accurate evaluation results without the need for prior training with a large number of samples.

[0038] It should be noted that the template construction layer selects the optimal template form based on the source document and the abstract to be evaluated, and further includes:

[0039] (1) Matching suggestion templates based on the pre-training method of the summary generation model.

[0040] It should be noted that if the pre-training method of the summary generation model is bidirectional training, which enables the summary generation model to have bidirectional encoding capabilities, then the suggested template form matched based on the pre-training method of the summary generation model is a cloze form. Bidirectional encoding is good at completing cloze-type tasks, so different model training methods give the model different capabilities. To maximize the model's capabilities, the present invention also determines the suggested template form through training methods and further optimizes the initially determined template form. This ensures that the provided template can both complete the translation of the task and maximize the model's capabilities, that is, allowing the model to perform the tasks it is best at, thereby improving the accuracy and efficiency of model evaluation.

[0041] (2) Matching candidate template forms based on the source document and the summary to be evaluated.

[0042] It should be noted that the candidate template format may be pre-defined or dynamically generated according to a specific task, which is not limited in this embodiment.

[0043] (3) Determine the optimal template form based on the candidate template forms and the recommended template form.

[0044] It should be noted that a matching or scoring mechanism can be used to compare the similarity or adaptability between the candidate template forms and the suggested template forms. Specifically, the optimal template form can be determined based on the semantics, structure, etc. of the template forms.

[0045] As an optional embodiment, determining the optimal form at least includes:

[0046] Comparing the fields of the candidate template form and the suggested template form, and marking the different fields;

[0047] Calculating an importance score of the difference field based on the difference field and the source document;

[0048] Based on a preset threshold, difference fields with importance scores higher than the preset threshold are screened and added to the candidate template form to form the optimal form. Furthermore, the addition position is determined based on the semantics of the difference fields to be added and the semantics of the candidate template. The method provided by the present invention not only considers task translation but also fully considers the model's strengths. Thus, a template is constructed based on these two factors, allowing the model to do what it is familiar with and good at, thereby maximizing the accuracy of the evaluation and giving full play to the model's capabilities.

[0049] S103: The template construction layer determines a prompt template based on the optimal template form, where the prompt template at least includes discontinuous blank information.

[0050] In combination with the above example, if the template construction layer determines that the prompt template is a cloze form based on the optimal template form, then the prompt template at least includes discontinuous blank information, and the discontinuous blank information is used to indicate the position to be filled in the template.

[0051] Specifically, these discontinuous blanks can be considered markers or placeholders that indicate the locations the model should focus on when processing text. It should be noted that the specific content that fills these discontinuous blanks depends on the specific task and text content. For example, discontinuous blanks can be filled with entities, keywords, phrases, etc.

[0052] In a specific implementation, the template construction layer determines the prompt template based on the optimal template form, specifically including:

[0053] (1) A template task is defined based on the source document and the summary to be evaluated.

[0054] Define the language model as M, and the input source document is represented as , the summary to be evaluated is represented as , the template is ,definition Denote the word embedding of D, defined Representing the word embedding of S, the complete input structure is:

[0055] ,

[0056] in Part of it is a solvable template vector, and its relative position to the original text and label words in the input can be freely determined.

[0057] (2) Randomly initializing a mixed template vector; the mixed template vector is a vector composed of a mixture of the source document, the summary to be evaluated, and a vector defining a template based on the source document and the summary to be evaluated.

[0058] Initialize the mixed template vector to generate a vector of (x, 768) dimensions.

[0059] (3) Learning some parameters of the mixed template vector by optimizing the loss function to obtain the prompt template vector.

[0060] Specifically, using the downstream loss function and the stochastic gradient descent method, the parameters corresponding to the template part are continuously optimized through training to obtain the prompt template vector:

[0061] .

[0062] It should be noted that the downstream loss function refers to the loss function defined on a specific task, such as the cross-entropy loss function or mean squared error loss function that may be used in the text summary consistency evaluation task. This loss function is used to measure the difference between the model's prediction results and the true label, and is an indicator of the model's performance on this task. Stochastic gradient descent is an optimization algorithm used to minimize the loss function. It iteratively updates the model parameters so that the loss function gradually decreases. In each iteration, stochastic gradient descent calculates the gradient of the loss function with respect to the parameters, and updates the parameter values ​​in the opposite direction of the gradient to reduce the loss function. The optimal template generated is a document containing discontinuous blanks. The blank positions are used for prediction based on the source document and summary. The positions with content are also obtained based on the source document and summary. The consistency evaluation task of the source document and summary is made a task that the model is better at through the description of the template, thereby improving the accuracy of the evaluation.

[0063] S104: Concatenate the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency assessment model.

[0064] After obtaining the hint template vector, the source document, the summary to be evaluated, and the hint template are concatenated to form the mixed input for the pre-trained summary consistency assessment model. This allows this information to be combined into a complete input sequence, allowing the summary consistency assessment model to simultaneously consider the content of the source document, the summary summary information, and the guidance of the hint template. This can better guide the summary consistency assessment model in performing summary consistency assessment tasks, improving the performance and effectiveness of the summary consistency assessment model.

[0065] S105 , inputting the mixture into a mask language model layer, predicting discontinuous blank information in the prompt template, and outputting a learning result.

[0066] It should be noted that the masked language model layer includes the BERT-MLM layer and the RoBERTa-MLM layer, and the BERT-MLM layer and the RoBERTa-MLM layer are used to complete the cloze task, specifically including:

[0067] (1) The source document and the summary to be evaluated are used as input sequences and masked and then input into the BERT-MLM layer and the RoBERTa-MLM layer.

[0068] It should be noted that the source document and the summary to be evaluated are used as input sequences, where the vacant position is represented by a special [MASK] tag. For example, in one embodiment, if the template is "the [MASK] jumped over the fence", the vacant position is [MASK]. These input sequences are then fed into the BERT-MLM layer or the RoBERTa-MLM layer for processing.

[0069] It should be noted that the BERT-MLM layer completes two tasks, namely the MLM task and the NSP task. The MLM task is similar to the "cloze test". Specifically, the BERT-MLM layer masks 15% of the words in a sentence, completes the embedding encoding of each word through a bidirectional transformer+feedforward+rediual_add+layer_norm, and then predicts the masked word. The prediction process is equivalent to multi-classification. The number of categories is the total number of words. The embedding of the masked word is transformed through the MLP to generate the logits probability on each category of words. The label is the one-hot encoding of the real word at the mask position in the entire vocabulary. The cross entropy of the logits and label is calculated, and a weighted average is performed to obtain the MLM loss.

[0070] The RoBERTa-MLM layer replaces BERT's static mask with a dynamic mask. Specifically, BERT masks the corpus and stores it, allowing the model to repeatedly learn from the same data in each epoch. RoBERTa replicates the data 10 times, applies different random masks to each, and then feeds the processed data into the model for learning.

[0071] (2) Define a set of candidate words and calculate the scores corresponding to the candidate words through the BERT-MLM layer and the RoBERTa-MLM layer.

[0072] Specifically, define the candidate word set For each source document and the summary to be evaluated is represented as , the template is , It represents the score of the language model when the vacant position is W. Therefore, the score corresponding to each candidate word is:

[0073] .

[0074] (3) The scores corresponding to the candidate words obtained by the calculation are converted into probability distribution, and the candidate word with the highest probability is selected to fill in the blank to complete the cloze task.

[0075] Specifically, the scores corresponding to the calculated candidate words can be converted into probability distribution through the Softmax function, and the candidate word with the highest probability is selected to fill in the blank to complete the cloze task.

[0076] It should be noted that BERT-MLM and RoBERTa-MLM are both pre-trained language models based on the Transformer architecture. In this embodiment, by using BERT-MLM and RoBERTa-MLM simultaneously, the calculated candidate word correspondence scores can be made more accurate, providing more accurate guarantees for the judgment of the summary fact consistency model.

[0077] S106 : Determine a consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result.

[0078] After obtaining the missing words, the results are input into the label mapping layer. The label mapping layer primarily defines the correspondence between candidate words and labels in the masked language model task. For example, depending on the task, sets such as {YES, NO} and {CORRECT, INCORRECT} are used as candidate words for cloze tests. For example, consistent candidate words are "yes," "correct," and "consistent," while inconsistent candidate words are "no," "incorrect," and "inconsistent." Therefore, the fact consistency assessment task can predict and select words from the candidate word set, and then obtain the final label through mapping relationships.

[0079] It should be noted that the definition of the correspondence between candidate words and labels through the label mapping layer includes:

[0080] (1) The label mapping layer receives candidate words for completing the cloze task from the masked language model layer;

[0081] (2) Preset a candidate word set and an evaluation label set; wherein the candidate word set is the same as the candidate word set defined in the masked language model layer;

[0082] (3) The probability of the candidate word calculated in the masked language model layer on the candidate word set is mapped to the probability of the evaluation label set.

[0083] When the masked language model layer predicts the [MASK] part in the candidate word set When the probability distribution on , the word probability in the label word set W is mapped to the probability of the evaluation label set. The specific calculation method is as follows:

[0084] .

[0085] In addition, the summary consistency evaluation method further includes at least: pre-training the masked language model layer and the label mapping layer based on the label samples, and the pre-training process of the masked language model layer and the label mapping layer includes:

[0086] (1) Randomly select a preset number of positions to perform masking operations on the input sequence of the input samples of the masked language model layer.

[0087] Combined with the above description, during the pre-training process, a preset number of positions are randomly selected to perform mask operations on the input sequence of the input sample. These positions are usually replaced with a special "[MASK]" mark.

[0088] (2) The masked input sequence is input into the masked language model layer, which predicts the masked position.

[0089] It should be noted that the masked input sequence is fed into the masked language model layer, which predicts the masked positions, attempting to predict the word at those positions. The output of the masked language model layer is the probability distribution of the word at each masked position.

[0090] (3) Inputting the predicted result of the masked language model layer into the label mapping layer to obtain the predicted label.

[0091] (4) Compare the predicted labels with the true label words of the label samples, calculate the loss function, and optimize the pre-trained parameters of the masked language model layer and the label mapping layer.

[0092] The predicted label is compared with the true label word of the label sample, and a cross entropy loss function is usually used to measure the difference between the predicted result and the true label.

[0093] Specifically, after obtaining the pre-trained parameters, the method further includes:

[0094] (1) Fix the pre-trained parameters of the label mapping layer.

[0095] (2) Taking the minimization of the loss function of the masked language model layer as the optimization goal, the backpropagation algorithm and optimizer are used to update the parameters of the masked language model layer, and the training is repeated until the pre-trained summary consistency evaluation model converges or reaches a pre-set number of training rounds.

[0096] In specific implementation, the parameters of the masked language model layer are updated by minimizing the cross entropy loss between the factual consistency labels mapped by the [MASK] candidate set in the prediction template and the true label words:

[0097] ,

[0098] The cross entropy loss is calculated by the above formula , where c represents the number of fact consistency categories, represents the manually labeled consistency category, Indicates the category after label mapping.

[0099] The summary consistency evaluation method based on an adaptive hint template provided in this embodiment is implemented based on a summary consistency evaluation model. The summary consistency evaluation model includes at least a template construction layer, a masked language model layer, and a label mapping layer. The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into the template construction layer, then the template construction layer selects an optimal template form based on the source document and the summary to be evaluated, so that the template construction layer determines a hint template based on the optimal template form, the hint template including at least discontinuous blank information, and then splicing the source document, the summary to be evaluated, and the hint template to form a mixed input of a pre-trained summary consistency evaluation model. The mixed input is then input to the masked language model layer, the discontinuous blank information in the hint template is predicted, and a learning result is output. Finally, the consistency evaluation result of the source document and the summary to be evaluated is determined through the label mapping layer and the learning result. In this way, the source document and the summary to be evaluated are processed according to the optimal template form at the template construction layer, and the source document, the summary to be evaluated and the processing results are spliced. Through the prompt of the optimal template, a new domain can be prompted as input data close to the domain during pre-training, thereby reconstructing it into a form that can better utilize the pre-trained summary consistency evaluation model, and the learning result is predicted and output through the masked language model layer. Finally, the text summary fact consistency evaluation result is obtained by comparing it with the label of the label mapping layer. Even on the basis of fewer training samples, the present invention can still prompt a specific prediction task to a task that has been learned a lot in the pre-training stage through the prompt of the optimal template, thereby more accurately evaluating the text summary fact consistency problem, without having to obtain a large number of learning samples in a specific domain. In addition, the method provided in this embodiment also uses a new paradigm of pre-training, prompting and predicting (Pre-train, Prompt, and Predict) pre-trained summary consistency model to detect text summary fact inconsistency problems. By designing "prompt text" and reformulating downstream tasks to make them closer to the training method of language models during pre-training, pre-training can be performed on a large amount of original text. By designing a prompt function, the model can act on few-shot or zero-shot samples and adapt to small amounts of or no-label data in other scenarios. This can solve many tasks, reduce the gap between the pre-training stage and downstream tasks, and achieve significant results in the task of detecting factual consistency in text summarization.

[0100] Corresponding to the aforementioned embodiment of a summary consistency evaluation method based on an adaptive prompt template, the present application also provides an embodiment of a summary consistency evaluation device based on an adaptive prompt template.

[0101] Figure 2This is a structural diagram of the first embodiment of the summary consistency evaluation device based on the adaptive prompt template provided by this application. Figure 2 The device provided in this embodiment includes an acquisition module 210, a selection module 220, a determination module 230, a processing module 240, a prediction module 250 and an evaluation module 260; wherein,

[0102] The acquisition module 210 is used to acquire the source document and the summary to be evaluated and input them into the template construction layer;

[0103] The selection module 220 is used for the template construction layer to select the optimal template form based on the source document and the abstract to be evaluated;

[0104] The determining module 230 is configured to determine a prompt template based on the optimal template form at the template construction layer, wherein the prompt template at least includes discontinuous blank information;

[0105] The processing module 240 is configured to combine the source document, the summary to be evaluated, and the prompt template to form a mixed input for a pre-trained summary consistency assessment model;

[0106] The prediction module 250 is used to input the mixture into the masked language model layer, predict the discontinuous blank information in the prompt template, and output the learning result;

[0107] The evaluation module 260 is configured to determine a consistency evaluation result between the source document and the summary to be evaluated through the label mapping layer and the learning result.

[0108] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0109] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0110] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0111] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A summary consistency evaluation method based on adaptive prompt template, characterized in that: The method is implemented based on a summary consistency evaluation model, which includes at least a template construction layer, a mask language model layer and a label mapping layer; The summary consistency evaluation method specifically includes: obtaining a source document and a summary to be evaluated and inputting them into a template construction layer; The template construction layer selects an optimal template form based on the source document and the abstract to be evaluated; The template construction layer determines a prompt template based on the optimal template form, wherein the prompt template at least includes discontinuous blank information; splicing the source document, the summary to be evaluated, and the prompt template to form a mixed input of a pre-trained summary consistency evaluation model; Inputting the mixture into the masked language model layer, predicting discontinuous blank information in the prompt template, and outputting a learning result; Determining a consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result; The template construction layer selects the optimal template form based on the source document and the summary to be evaluated, and further includes: Matching suggestion templates based on the pre-training method of the summary generation model; Matching candidate template forms based on the source document and the summary to be evaluated; Determining an optimal template form based on the candidate template forms and the suggested template form; Determining the optimal form at least includes: Comparing the fields of the candidate template form and the suggested template form, and marking the different fields; Calculating an importance score of the difference field based on the difference field and the source document; Based on the preset threshold, the difference fields with importance scores higher than the preset threshold are filtered and added to the recommended template form to form the optimal form; The supplementation position is determined based on the semantics of the difference field to be supplemented and the semantics of the candidate template.

2. The method according to claim 1, characterized in that If the pre-training mode of the summary generation model is bidirectional training, so that the summary generation model has bidirectional encoding capability, then the suggested template form matched based on the pre-training mode of the summary generation model is a cloze form.

3. The method according to claim 1, characterized in that The template construction layer determines the prompt template based on the optimal template form, specifically including: defining a template task according to the source document and the summary to be evaluated; Randomly initializing a mixed template vector; the mixed template vector is a vector composed of a mixture of the source document, the summary to be evaluated, and a vector defining a template according to the source document and the summary to be evaluated; Part of the parameters in the hybrid template vector are learned by optimizing the loss function to obtain the prompt template vector.

4. The method according to claim 1, wherein The masked language model layer includes a BERT-MLM layer and a RoBERTa-MLM layer, and the BERT-MLM layer and the RoBERTa-MLM layer are used to complete the cloze task, specifically including: The source document and the summary to be evaluated are used as input sequences and masked before being input into the BERT-MLM layer and the RoBERTa-MLM layer; Define a set of candidate words and calculate the scores corresponding to the candidate words through the BERT-MLM layer and the RoBERTa-MLM layer; The scores corresponding to the candidate words obtained by the calculation are converted into probability distribution, and the candidate word with the largest probability is selected to fill in the blank to complete the cloze task.

5. The method according to claim 1, wherein Defining the correspondence between candidate words and labels through the label mapping layer includes: The label mapping layer receives candidate words for completing the cloze task in the masked language model layer; Preset a candidate word set and an evaluation label set; wherein the candidate word set is the same as the candidate word set defined in the masked language model layer; The probability of the candidate word calculated in the masked language model layer on the candidate word set is mapped to the probability of the evaluation label set.

6. The method according to claim 1, wherein The summary consistency evaluation method further includes at least: pre-training the masked language model layer and the label mapping layer based on the label samples, wherein the pre-training process of the masked language model layer and the label mapping layer includes: Randomly select a preset number of positions to perform masking operations on the input sequence of the input samples of the masked language model layer; Inputting the masked input sequence into the masked language model layer, which predicts the masked positions; Inputting the predicted result of the masked language model layer into the label mapping layer to obtain a predicted label; The predicted label is compared with the true label word of the label sample, and a loss function is calculated to obtain pre-trained parameters of the masked language model layer and the label mapping layer.

7. The method according to claim 6, characterized in that After obtaining the pre-trained parameters, the method further includes: Fixing the pre-trained parameters of the label mapping layer; With the minimization of the loss function of the masked language model layer as the optimization goal, the backpropagation algorithm and optimizer are used to update the parameters of the masked language model layer, and the training is repeated until the pre-trained summary consistency evaluation model converges or reaches a pre-set number of training rounds.

8. A summary consistency evaluation device based on an adaptive prompt template, characterized in that: The device includes an acquisition module, a selection module, a determination module, a processing module, a prediction module and an evaluation module; wherein, The acquisition module is used to acquire the source document and the summary to be evaluated and input them into the template construction layer; The selection module is used for the template construction layer to select the optimal template form based on the source document and the abstract to be evaluated; The determining module is used for the template construction layer to determine a prompt template based on the optimal template form, wherein the prompt template at least includes discontinuous blank information; The processing module is configured to combine the source document, the summary to be evaluated, and the prompt template to form a mixed input for a pre-trained summary consistency assessment model; The prediction module is used to input the mixture into the masked language model layer, predict the discontinuous blank information in the prompt template, and output the learning result; The evaluation module is configured to determine a consistency evaluation result of the source document and the summary to be evaluated through the label mapping layer and the learning result; The template construction layer selects the optimal template form based on the source document and the summary to be evaluated, and further includes: Matching suggestion templates based on the pre-training method of the summary generation model; Matching candidate template forms based on the source document and the summary to be evaluated; Determining an optimal template form based on the candidate template forms and the suggested template form; Determining the optimal form at least includes: Comparing the fields of the candidate template form and the suggested template form, and marking the different fields; Calculating an importance score of the difference field based on the difference field and the source document; Based on the preset threshold, the difference fields with importance scores higher than the preset threshold are filtered and added to the recommended template form to form the optimal form; The supplementation position is determined based on the semantics of the difference field to be supplemented and the semantics of the candidate template.

Images