A Text Summarization Quality Assessment Method and System Based on Text Matching

Abstract

This invention discloses a text summarization quality assessment method and system based on text matching. The method involves inputting the text summary to be evaluated and its corresponding original text into a pre-trained text summarization quality assessment model to obtain the quality assessment result. The text summarization quality assessment model includes semantic sub-models for measuring four fine-grained quality aspects: relevance, consistency, coherence, and fluency. The input to each semantic sub-model is the text summary to be evaluated and its corresponding original text, and the output is the quality assessment result for the corresponding fine-grained quality aspect. This invention can capture subtle differences in summary quality from a semantic perspective, avoiding the suppression of textual diversity caused by forced alignment with a single reference summary. It possesses advantages such as excellent model performance and good quality interpretability, achieving a comprehensive measurement of summary quality.

Description

Technical Field

[0001] This invention relates to quality assessment technology for automatic text summarization, specifically to a text summarization quality assessment method and system based on text matching. Background Technology

[0002] Automatic text summarization aims to compress long documents into fluent, concise text that preserves the salient information of the original. With the application of technologies such as pre-trained language models, automatic text summarization modeling techniques have significantly improved. However, research on summarization evaluation metrics lags behind summarization modeling techniques. Currently, ROUGE remains the mainstream evaluation metric for summarization. Research shows that the ROUGE metric is only a coarse proxy for summarization quality assessment, and the following issues remain unresolved in assessing whether the quality of the summarized text meets human standards:

[0003] First, existing summary evaluation methods fail to capture subtle semantic differences between summary texts. Mainstream evaluation metrics typically only measure the literal shape overlap similarity between the test summary and the reference summary. For example, common summary evaluation metrics like ROUGE, BLEU, and METEOR simply calculate the N-gram overlap between the candidate and the reference. Since the same meaning can have different expressions, and literally similar sentences can have opposite meanings, literal matching methods only provide a simple and general metric, but are insufficient to capture subtle semantic differences in summaries.

[0004] Secondly, some studies have shown that mainstream evaluation metrics do not correlate well with human assessment judgments and lack interpretability. Research suggests that accurate abstract evaluation methods should align with human evaluation. When assessing abstract quality, humans typically consider multiple fine-grained dimensions, such as relevance, information richness, redundancy, coherence, and structural soundness, to conduct a comprehensive quality assessment. Abstract evaluation metrics also need to address these fine-grained quality dimensions that are of interest to humans, providing interpretable scores.

[0005] Finally, existing evaluation methods for summarizing models suppress the diversity of summaries generated. This is because mainstream summarizing metrics rely on comparing the tested summary with a single reference summary, neglecting the semantics and properties of the original text. However, humans write summaries only after reading the original text. The process of comparing a summary with a reference summary and then scoring it doesn't exist in real life. Furthermore, readers with different knowledge bases and different reading purposes will write summaries that, while different, are all correct. It's unreasonable to assume that a summary that "doesn't look like" a single reference summary is unqualified. This evaluation method significantly suppresses the diversity of summaries generated by summarizing models.

[0006] A good evaluation metric should help filter out: (1) those summaries that have good word overlap with the original text or reference summary and are semantically correct; (2) those summaries that contain sufficient relevant information and are semantically correct, even if their wording differs from the reference summary. Although mainstream heuristic-based evaluation metrics are not perfect, they still dominate in practice, which is reasonable. This is because their design principles are very reasonable, and they can be transferred to new tasks without training when there is a lack of labeled data for summarization quality scoring. However, research on other language tasks has confirmed that the performance of related models can be greatly improved by using training-based evaluation metrics that have learned to match human metrics. Therefore, it is very necessary to develop training-based automatic evaluation metrics for text summarization systems. This has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a text summarization quality assessment method and system based on text matching, which addresses the above-mentioned problems in the prior art. The present invention can capture subtle differences in summary quality from a semantic perspective, avoids the suppression of the diversity of summary text caused by forced alignment with a single reference summary, and has the advantages of excellent model performance and good quality interpretability, so as to achieve the purpose of comprehensively measuring the quality of summary.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A text summarization quality assessment method based on text matching is characterized by including inputting the text summary to be assessed and its corresponding original text into a pre-trained text summarization quality assessment model to obtain the quality assessment result corresponding to the text summary to be assessed. The text summarization quality assessment model includes semantic sub-models for measuring four fine-grained qualities: relevance, consistency, coherence, and fluency. The input of the semantic sub-model is the text summary to be assessed and its corresponding original text, and the output is the quality assessment result of the corresponding fine-grained quality.

[0010] Optionally, the four fine-grained quality semantic sub-models have the same structure, all including a pre-trained cross-encoder and a regression model based on a multilayer perceptron; the input of the cross-encoder is the text summary to be evaluated and its corresponding original text, and the first token of the input is the token character [CLS], the text summary to be evaluated and its corresponding original text are separated by the delimiter character [SEP], and the final hidden state V corresponding to the token character [CLS] is... [CLS]As the final extracted feature, the multilayer perceptron-based regression model is used to map the final feature to a predicted score for a specified interval.

[0011] Optionally, in the semantic sub-model used to measure relevance and consistency, the final hidden state V corresponding to the token character [CLS] [CLS] The expression for the computation function is:

[0012] V [CLS] =Cross-Encoder([CLS],O,[SEP],S)

[0013] In the above formula, Cross-Encoder represents the interactive encoder Cross-Encoder, [CLS] is the token character, O is the original text, [SEP] is the delimiter character, and S is the text digest to be evaluated;

[0014] In the semantic sub-model used to measure coherence and fluency, the final hidden state V corresponding to the token character [CLS] is... [CLS] The expression for the computation function is:

[0015] V [CLS] =Cross-Encoder([CLS],[SEP],S)

[0016] In the above formula, Cross-Encoder represents the interactive encoder Cross-Encoder, [CLS] is the token character, [SEP] is the separator character, and S is the text digest to be evaluated.

[0017] Optionally, the function expression for mapping the final features to predicted scores for a specified interval in the multilayer perceptron-based regression model is as follows:

[0018] y = WV [CLS] +b

[0019] In the above formula, y is the predicted score, and the weights W and biases b are trainable network parameters.

[0020] Optionally, the loss function used during training of the four fine-grained quality semantic sub-models is:

[0021]

[0022] In the above formula, L is the loss function, N is the number of training samples, y is the prediction score of the fine-grained quality semantic sub-model output, and y′ is the fitting label score of the training samples.

[0023] Optionally, before inputting the text summary to be evaluated and its corresponding original text into the pre-trained text summarization quality evaluation model, the method further includes the step of training a semantic sub-model for each fine-grained quality level:

[0024] S1. For the initial sub-model M0 of this fine-grained semantic sub-model, conduct the first round of supervised training using a labeled dataset to obtain the optimal sub-model M0. best ;

[0025] S2. Regarding the obtained optimal sub-model M0 best Multiple rounds of semi-supervised training were performed using an unlabeled training dataset to improve the performance of the semantic sub-model by leveraging the unlabeled training data, resulting in the optimal fine-grained quality semantic sub-model M. best .

[0026] Optionally, step S1 includes:

[0027] S1.1, the optimal sub-model M0 best Initialize the initial sub-model M0 to the semantic sub-model of this fine-grained quality;

[0028] S1.2, using the labeled training dataset D train For the optimal submodel M0 best Perform a specified number of supervised training iterations (ep), and for any i-th training iteration, the resulting sub-model M... i+1 The labeled validation dataset D eval For each sub-model M i+1 The current optimal sub-model M0 best To verify, if sub-model M i+1 The validation results are better than the current best sub-model M0. best The verification results will then be used to transform the sub-model M. i+1 As the new current optimal submodel M0 best Otherwise, maintain the current optimal sub-model M0. best The process remains unchanged; ultimately, after completing a specified number of supervised training iterations (ep), the final optimal sub-model M0 is determined. best The final optimal sub-model M0 best .

[0029] Optionally, step S2 includes:

[0030] S2.1, the labeled training dataset D train Add the training dataset D, and then use the optimal fine-grained quality semantic sub-model M. best Initialize as the optimal submodel M0 best The unlabeled training dataset is divided into k parts to obtain unlabeled training datasets D1 to D2.k Initialize the loop variable t to 1 to begin iterating through the unlabeled training dataset D1 to D2. k ;

[0031] S2.2, using the optimal sub-model M from the previous round t-1. t-1 best Let D be the t-th unlabeled training dataset. t Label the data to obtain the corresponding pseudo-label dataset. and pseudo-label dataset Add to training dataset D;

[0032] S2.3, the optimal sub-model M of this round t. t best Initialize the initial sub-model M0 to the semantic sub-model of this fine-grained quality;

[0033] S2.4, using the training dataset D to find the optimal sub-model M for this round t. t best Perform a specified number of supervised training iterations (ep), and for any i-th training iteration, the resulting sub-model M... i+1 The labeled validation dataset D eval For each sub-model M i+1 The optimal sub-model M in this round t t best To verify, if sub-model M i+1 The validation results are better than the optimal sub-model M in this round t. t best The verification results will then be used to transform the sub-model M. i+1 As the optimal sub-model M in the new current round t t best Otherwise, maintain the current optimal sub-model M for round t. t best The process remains unchanged; ultimately, after completing a specified number of supervised training iterations (ep), the optimal sub-model M from the current round (t) is determined. t best The optimal sub-model M obtained in this round t is... t best ;

[0034] S2.5, the labeled validation dataset D eval For the optimal sub-model M in this round t respectively t best The optimal fine-grained quality semantic sub-model M best Verification is performed to determine if the optimal sub-model M in this round t is... t best The validation results outperform the best fine-grained quality semantic submodel M. bestThe verification results will then determine the optimal sub-model M for this round t. t best M, as the new optimal fine-grained quality semantic sub-model best Otherwise, maintain the optimal fine-grained quality semantic submodel M. best constant;

[0035] S2.6, determine whether the loop variable t is greater than k. If it is true, output the final optimal fine-grained quality semantic sub-model M. best If the loop terminates and exits, then increment the loop variable t by 1 and jump to step S2.2.

[0036] Furthermore, the present invention also provides a text summarization quality assessment system based on text matching, including a microprocessor and a memory interconnected thereto, the microprocessor being programmed or configured to perform the steps of the text summarization quality assessment method based on text matching.

[0037] Furthermore, the present invention provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to perform the steps of the text summarization quality assessment method based on text matching.

[0038] Compared with the prior art, the present invention has the following main advantages:

[0039] 1. This invention does not use literal similarity-based measurement methods such as ROUGE and BLEU, but instead uses a semantic sub-model. The input of the semantic sub-model is the text summary to be evaluated and its corresponding original text, and the output is the quality evaluation result with corresponding fine-grained quality, thereby capturing the subtle differences in summary quality from a semantic perspective.

[0040] 2. The semantic sub-model of this invention takes as input the text summary to be evaluated and its corresponding original text, and outputs the quality assessment result with corresponding fine-grained quality. It adopts the input mode of "original text-summary to be evaluated". By comparing the semantics of the summary to be evaluated with the original text to score it, it avoids the suppression of the diversity of summary text caused by forced alignment with a single reference summary.

[0041] 3. The text summary quality assessment model of this invention includes semantic sub-models for measuring four fine-grained qualities: relevance, consistency, coherence, and fluency. It is consistent with the human assessment process and the scores are interpretable. It measures the four fine-grained qualities of relevance, consistency, coherence, and fluency respectively, thereby achieving the goal of comprehensively measuring the quality of the summary.

[0042] In summary, this invention, inspired by the text matching principle of retrieval, has the ability to efficiently identify the quality of summary texts. It can capture subtle differences in summary quality from a semantic perspective, starting from fine-grained quality dimensions that humans care about. It has excellent human relevance and strong interpretability, avoids the suppression of summary text diversity caused by forced alignment with a single reference summary, helps to improve the diversity of summary texts generated by the summary system, and has the advantages of excellent model performance and good quality interpretability, thus achieving the goal of comprehensively measuring summary quality. Attached Figure Description

[0043] Figure 1 This is a schematic diagram illustrating the basic principle of the method in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the semantic sub-model in an embodiment of the present invention.

[0045] Figure 3 This is the ablation experiment result showing the impact of pseudo-annotation data volume on Fluency scoring in an embodiment of the present invention.

[0046] Figure 4 This is a scatter plot showing the scoring effect of input pattern [O||S] on the coherence dimension in this embodiment of the invention.

[0047] Figure 5 This is a scatter plot showing the scoring effect of input pattern [S] on the coherence dimension in this embodiment of the invention.

[0048] Figure 6 This is the ablation experiment result showing the impact of input methods [S||ref] and [O||S] on the diversity of the abstract in this embodiment of the invention.

[0049] Figure 7 This is a typical case analysis diagram in the embodiments of the present invention. Detailed Implementation

[0050] like Figure 1As shown, this embodiment of the text summarization quality assessment method based on text matching includes inputting the text summary to be evaluated and its corresponding original text into a pre-trained text summarization quality assessment model to obtain the quality assessment result corresponding to the text summary to be evaluated. The text summarization quality assessment model includes semantic sub-models for measuring four fine-grained qualities: relevance, consistency, coherence, and fluency. The input of the semantic sub-model is the text summary to be evaluated and its corresponding original text, and the output is the quality assessment result for the corresponding fine-grained quality. Relevance refers to the summary containing only important information from the source document, penalizing summaries with redundant information; Consistency refers to the summary containing only the facts and topics of the original text, which should be consistent and avoid illusory facts; Coherence refers to the summary being a coherent set of information about a topic, with good logical organization between sentences; Fluency refers to the quality of the language and the presence of grammatical errors that affect readability. By dividing the quality of the summary into multiple fine-grained dimensions and then scoring them on specific dimensions, the goal of comprehensively measuring the quality of the summary is achieved.

[0051] In this embodiment, the semantic sub-model structures for the four fine-grained qualities are identical. For example... Figure 2 As shown, both include a pre-trained Cross-Encoder and a regression model based on a Multilayer Perceptron (MLP). The input to the Cross-Encoder is the text summary to be evaluated and its corresponding original text. The first token in the input is a token character [CLS]. The text summary to be evaluated and its corresponding original text are separated by a delimiter character [SEP]. The final hidden state V corresponding to the token character [CLS] is... [CLS]As the final extracted feature, the regression model based on the multilayer perceptron is used to map the final feature to a predicted score within a specified interval (e.g., a predicted score between 0 and 5 in this embodiment), which can be represented as score(o,s), where o represents the original text and s represents the text summary. The Cross-Encoder (Hugging Face Sentence Transformers) is an existing neural network model widely used in information retrieval due to its outstanding performance. In question-answering retrieval, when ranking candidate answers, it is usually necessary to calculate the similarity between the question and each candidate answer. The higher the similarity score between the answer and the question, the more accurate the answer is considered. Specifically, this can be achieved by concatenating the question text and the answer text using the [esp] symbol and inputting it into the Cross-Encoder model for training. Similarly, a summary can be seen as semantically similar text obtained after compressing the original text. A heuristic idea is that the higher the semantic similarity of the summary to the original text, the higher the quality of the summary. Inspired by the principles of question-answering retrieval, we also consider the scoring of summary quality as a process of calculating the similarity between the original text and the summary text.

[0052] In this embodiment, in the semantic sub-model used to measure relevance and consistency, the final hidden state V corresponding to the token character [CLS] is... [CLS] The expression for the computation function is:

[0053] V [CLS] =Cross-Encoder([CLS],O,[SEP],S)

[0054] In the above formula, Cross-Encoder represents the interactive encoder, [CLS] is the token character, O is the original text, [SEP] is the delimiter character, and S is the text digest to be evaluated. See also Figure 2 The original text O represents n words, token1. o ~token n o The word sequence constituted; the text summary S to be evaluated is a sequence of m words, token1. s ~token m s The sequence of words formed.

[0055] In this embodiment, the four fine-grained quality semantic sub-models have the same structure and are all trained and optimized based on the Cross-Encoder described above. The four sub-models have the same structure, but for coherence and fluency, the input mode no longer uses sentence pairs. This is because fluency assesses the language quality of the summary itself. Experts can evaluate fluency without referring to the original text. For coherence scoring, experts only focus on whether the summary text itself has a clear theme and rigorous sentence logic. Therefore, including original text information in the input is not consistent with real-world scenarios. In contrast, experts need to repeatedly compare the summary and the original text when scoring consistency and relevance. Therefore, for the coherence and fluency sub-models, we removed original text information from the input. In the semantic sub-models used to measure coherence and fluency, the final hidden state V corresponding to the token character [CLS] is... [CLS] The expression for the computation function is:

[0056] V [CLS] =Cross-Encoder([CLS],[SEP],S)

[0057] In the above formula, Cross-Encoder represents the interactive encoder Cross-Encoder, [CLS] is the token character, [SEP] is the separator character, and S is the text digest to be evaluated.

[0058] In this embodiment, the function expression for mapping the final features to predicted scores within a specified interval using the regression model based on the multilayer perceptron is as follows:

[0059] y = WV [CLS] +b

[0060] In the above formula, y is the predicted score, and the weights W and biases b are trainable network parameters.

[0061] In this embodiment, the loss function used during the training of the four fine-grained quality semantic sub-models is:

[0062]

[0063] In the above formula, L is the loss function, N is the number of training samples, y is the prediction score of the fine-grained quality semantic sub-model output, and y′ is the fitting label score of the training samples. Although the model is simple, it has excellent performance.

[0064] In this embodiment, the quality assessment result corresponding to the text summary to be evaluated includes two coefficients: Pearson coefficient and Spearman correlation coefficient. Therefore, after obtaining the quality assessment result corresponding to the text summary to be evaluated, a scoring function f() can be used to compare the correlation between the scores predicted by the sub-model and the scores of the human annotations on the test data, which judges the quality of the scoring model. The scoring function f() can be selected from max(Pearson), max(Spearman), and max(Pearson*Spearman), that is, the largest Pearson coefficient, the largest Spearman correlation coefficient, or the maximum value of the product of the Pearson coefficient and the Spearman correlation coefficient.

[0065] Considering the limited availability of labeled data resources, in order to improve the accuracy of the text summarization quality assessment model, this embodiment includes a step of training a semantic sub-model for each fine-grained quality level before inputting the text summary to be evaluated and its corresponding original text into the pre-trained text summarization quality assessment model:

[0066] S1. For the initial sub-model M0 of this fine-grained semantic sub-model, conduct the first round of supervised training using a labeled dataset to obtain the optimal sub-model M0. best ;

[0067] S2. Regarding the obtained optimal sub-model M0 best Multiple rounds of semi-supervised training were performed using an unlabeled training dataset to improve the performance of the semantic sub-model by leveraging the unlabeled training data, resulting in the optimal fine-grained quality semantic sub-model M. best .

[0068] Since multi-round semi-supervised training uses unlabeled training datasets, it does not require labeled datasets. This can effectively utilize unlabeled training data to improve the performance of semantic sub-models, enhance the accuracy of text summarization quality assessment models, and increase the relevance to human scoring.

[0069] In this embodiment, step S1 includes:

[0070] S1.1, the optimal sub-model M0 best Initialize the initial sub-model M0 to the semantic sub-model of this fine-grained quality;

[0071] S1.2, using the labeled training dataset D train For the optimal submodel M0 best Perform a specified number of supervised training iterations (ep), and for any i-th training iteration, the resulting sub-model M... i+1 The labeled validation dataset D evalFor each sub-model M i+1 The current optimal sub-model M0 best To verify, if sub-model M i+1 The validation results are better than the current best sub-model M0. best The verification results will then be used to transform the sub-model M. i+1 As the new current optimal submodel M0 best Otherwise, maintain the current optimal sub-model M0. best The process remains unchanged; ultimately, after completing a specified number of supervised training iterations (ep), the final optimal sub-model M0 is determined. best The final optimal sub-model M0 best .

[0072] In this embodiment, step S2 includes:

[0073] S2.1, the labeled training dataset D train Add the training dataset D, and then use the optimal fine-grained quality semantic sub-model M. best Initialize as the optimal submodel M0 best The unlabeled training dataset is divided into k parts to obtain unlabeled training datasets D1 to D2. k Initialize the loop variable t to 1 to begin iterating through the unlabeled training dataset D1 to D2. k ;

[0074] S2.2, using the optimal sub-model M from the previous round t-1. t-1 best Let D be the t-th unlabeled training dataset. t Label the data to obtain the corresponding pseudo-label dataset. and pseudo-label dataset Add to training dataset D;

[0075] S2.3, the optimal sub-model M of this round t. t best Initialize the initial sub-model M0 to the semantic sub-model of this fine-grained quality;

[0076] S2.4, using the training dataset D to find the optimal sub-model M for this round t. t best Perform a specified number of supervised training iterations (ep), and for any i-th training iteration, the resulting sub-model M... i+1 The labeled validation dataset D eval For each sub-model M i+1 The optimal sub-model M in this round t t best To verify, if sub-model M i+1 The validation results are better than the optimal sub-model M in this round t.t best The verification results will then be used to transform the sub-model M. i+1 As the optimal sub-model M in the new current round t t best Otherwise, maintain the current optimal sub-model M for round t. t best The process remains unchanged; ultimately, after completing a specified number of supervised training iterations (ep), the optimal sub-model M from the current round (t) is determined. t best The optimal sub-model M obtained in this round t is... t best ;

[0077] S2.5, the labeled validation dataset D eval For the optimal sub-model M in this round t respectively t best The optimal fine-grained quality semantic sub-model M best Verification is performed to determine if the optimal sub-model M in this round t is... t best The validation results outperform the best fine-grained quality semantic submodel M. best The verification results will then determine the optimal sub-model M for this round t. t best M, as the new optimal fine-grained quality semantic sub-model best Otherwise, maintain the optimal fine-grained quality semantic submodel M. best constant;

[0078] S2.6, determine whether the loop variable t is greater than k. If it is true, output the final optimal fine-grained quality semantic sub-model M. best If the loop terminates and exits, then increment the loop variable t by 1 and jump to step S2.2.

[0079] To validate the text summarization quality assessment model of the text summarization quality assessment method based on text matching in this embodiment, the cross-encoder with a large number of parameters, such as RoBERTa-large (24 layers), was abandoned. Only the publicly available DistilRoBERTa-base (6 layers) and RoBERTa-base (12 layers) were selected for tuning and training. In the experiment, the number of tuning ep was set to 6, and the batch size was set to 4. The scale of the newly expanded pseudo-annotation data in each round of semi-supervised training was 1000 data points. When the amount of newly expanded pseudo-annotation data reached 10000 (approximately ten times that of the annotation data), the model could achieve better performance. At the beginning of each new round of semi-supervised training, the model first performed linear warm-start training. The number of steps in the warm-start training was 1 / 10 of the total number of steps in a single round of semi-supervised training, and was proportional to the amount of pseudo-annotation data. The model uses Adam as the optimizer, with the learning rate (a known parameter) set to 2e-5 and the weight decay (a known parameter) set to 0.01. Training uses gradient clipping with a maximum gradient norm of 1, and no regularization is applied. Consistent with previous experiments on metrics, the evaluation metrics used are Pearson's coefficient and Spearman's rank correlation coefficient. These correlation coefficients are used to assess the correlation between the metrics and human ratings, and to determine the quality of the metric model. Several existing methods were also used for comparison, and the final results are shown in Table 1. The evaluation metrics Pearson and Spearman will be referred to as P and S in subsequent experiments.

[0080] Table 1: Comparative experimental results of the method in this embodiment and existing methods.

[0081]

[0082] In the table above, the training-free methods (using pre-trained models) for comparison are as follows: BLEU1 to BLEU4 represent implementations of the evaluation metric BLEU based on 1-gram to 4-gram, respectively; TF-IDF is a weighted technical metric commonly used in information retrieval and data mining; ROUGE-1 to ROUGE-3 and ROUGE-L represent implementations of the evaluation metric ROUGE based on 1-gram to 3-gram and the longest common subsequence principle, respectively; BERTscore-r, BERTscore-p, and BERTscore-f represent implementations of the evaluation metric BERTscore based on recall, precision, and harmonic mean principles, respectively; SUPERT is a high-performance evaluation metric that does not rely on reference summaries. The training-based methods for comparison (requiring pre-training) include: BLEUT, an evaluation metric with task-specific innovative pre-training for BERT; BERT for MTE, an evaluation metric obtained after fine-tuning BERT training on labeled data; and BERT for MTE. Base This indicates the version of BERT forMTE obtained after a warm start based on the RoBERTaBase model. (SummScore) Distil This indicates that the semantic sub-model of the method in this embodiment adopts DistilRoBERTa-base (6 layers), and SummScore. Base This example illustrates that the semantic sub-model of the method in this embodiment is implemented using RoBERTa-base (12 layers).

[0083] Referring to Table 1, compared to methods that do not require training (using pre-trained models), the semantic sub-model of this embodiment significantly outperforms them. Furthermore, these metrics, except for SUPERT, exhibit low relevance to human annotations across various fine-grained quality levels. However, we found that they (such as BLEU, ROUGE, and SUPERT) often show better relevance to human judgment than coherence and fluency in the relevance and consistency dimensions. We believe this is because these metrics require comparison with the literal N-grams or semantic information of the reference summary when scoring. Since the reference summary is a compressed text that captures the central idea of ​​the original text, these metrics can achieve the initial purpose of measuring the relevance and consistency of the generated summary with respect to the central semantics of the original text. Unfortunately, because they were designed without considering the quality requirements of measuring the coherence and fluency of the summary text itself, these metrics often perform poorly in the coherence and fluency dimensions. For the comparative experimental group based on training-based metrics, our model also outperformed any of them. However, we found that these metrics also exhibited excellent performance after multiple rounds of semi-supervised training on manually annotated data. To eliminate the influence of the pre-trained language model, we replaced the BERT for MTE pre-trained model with the same DistilRoBERTaBase model trained on STS data. We found that the BERT for MTE model demonstrated stronger performance competitiveness. This, to some extent, proves the rationality of the design approach of this embodiment based on the principle of text similarity matching in information retrieval.

[0084] We believe that multi-round semi-supervised training is a crucial factor in improving the performance of the semantic sub-model in this embodiment. This training method rapidly expands the pseudo-annotation data, alleviating the problem of insufficient data volume. To explore the impact of the amount of pseudo-annotation data on model performance, we conducted corresponding ablation experiments. Here, we only introduce the SummScore used in this embodiment. Base The ablation experiments on the Fluency quality dimension yielded similar conclusions for other quality dimensions. In the ablation experiments, the span of the pseudo-annotation data expanded in each round was 2000 entries. The results of the ablation experiments are as follows: Figure 3As shown, it is clear that the model's performance significantly improves in the early stages as the amount of pseudo-annotation data increases. This indicates that the scale of the data is a major limiting factor for model performance during this period. When the expanded pseudo-annotation data reaches approximately 10,000 entries (10 times the amount of annotation data), the fluency correlation reaches its peak. This indicates that at this point, the model has maximized the benefits from the increased data volume. Subsequently, even with further increases in data, the model's performance no longer improves and may even begin to decline. The ablation experiment results demonstrate that, in situations where labeled data is scarce, reasonable multi-round semi-supervised training can effectively improve the performance of the semantic sub-model of the method in this embodiment. This also provides a new training approach for researchers in similar experimental scenarios in the future to overcome the limitations of small data volumes.

[0085] For the coherence and fluency quality dimensions, we believe the model should draw inspiration from the human process of evaluating summaries. In this embodiment, the semantic sub-model scoring only needs to focus on the quality of the text itself. Therefore, we explore ablation experiments on the influence of sentence-based input patterns on pattern [O||S] and single-summary form [S] in coherence and fluency. In this embodiment, the semantic sub-model SummScore... Base The correlation between coherence and fluency and human assessment is shown in Table 2.

[0086] Table 2: The degree of correlation between coherence and fluency and human assessment.

[0087]

[0088] As shown in Table 2, the [S] form significantly outnumbers the sentence pair pattern [O||S]. To visually represent the difference in scoring effectiveness, we randomly sampled a subset of samples along the quality dimension of coherence and plotted a quality scoring scatter plot. The results are as follows: Figure 4 and Figure 5 As shown. Combined with Figure 4 and Figure 5 We can also intuitively see that the distribution of [S] is more correlated with human scoring. In fact, this result is expected, because for the two quality dimensions of coherence and fluency, the semantic sub-model of this embodiment focuses on the original text information, which may distract from the summary itself.

[0089] To explore the differences between the original text-to-text summarization input pattern [O||S] and the more common input pattern [S||ref] (commonly used in models such as BLEURT and BERT for MTE), we also conducted relevant analytical experiments. Table 3 shows the results of the semantic sub-model SummScore in this embodiment. Base The two input methods are correlated with human assessments in terms of Consistency and Relevance, respectively.

[0090] Table 3: Semantic Sub-model SummScore Base A comparison of the correlation between the two input methods.

[0091]

[0092] Referring to Table 3, we can see that [O||S] achieves better results compared to [S||ref]. Initially, we were concerned that the longer original text information in [O||S] might be more difficult for the semantic sub-model to learn in this embodiment, compared to the concise reference summary in [S||ref]. Analyzing the reasons for the superior performance of [O||S], we believe it's because the [O||S] format is more consistent with the human scoring process in terms of consistency and relevance. This is because, generally, humans begin writing a summary after reading the original text. In reality, reference summaries are rarely repeatedly consulted to achieve the goal of writing a new summary. During the scoring process, experts often only read the original text before scoring. The input pattern of [O||S] also aligns with the expert scoring process.

[0093] In-depth experiments revealed that the original text-summary format [O||S] can support the diversity of textual expression in the summary. This is because the ROUGE score depends on the reference summary. The more similar the summary is to the reference summary in N-grams, the higher the ROUGE score. Conversely, a lower ROUGE score indicates a greater difference between the current summary and the reference summary. However, summaries that differ from the reference summary in expression but are semantically correct are also acceptable. A good summary metric should be able to identify summaries that are diverse in expression but of acceptable quality. From the Summeval annotation data, for each original text, we extracted data with low ROUGE scores but high human scores. We plotted scatter plots of human scores for these two input modes and, in this embodiment, scores for the two modes in the semantic sub-model. Only experiments on the relevance dimension are listed here, and the results are as follows... Figure 6As shown, we can clearly see that [O||S] more closely approximates the distribution of human scoring. However, [S||ref] is more likely to assign low scores to summaries that would otherwise receive high human scores. Therefore, this suggests that [O||S] can identify summaries with different literal expressions but acceptable quality. This also indirectly indicates that the [O||S] pattern helps improve the diversity of summaries generated. We were also curious about the interrelationships among these four fine-grained quality levels. Therefore, we measured the correlation coefficients among the four fine-grained quality levels. The correlation results are shown in Table 4.

[0094] Table 4: Correlation coefficients among the four fine-grained qualities.

[0095]

[0096] As shown in Table 4, there is a strong correlation between coherence and relevance. This explains why, in abstract research, improving relevance can also improve coherence. This also suggests that focusing on optimizing a fine-grained quality can sometimes yield benefits in other quality areas.

[0097] As a specific case, such as Figure 7As shown, this embodiment tested a text: River Plate is pursuing and signing Manchester United striker Radame Falcao, followed by some information about Radame Falcao. The summary obtained using the semantic sub-model in this embodiment also exhibited illusion errors, receiving low scores in consistency (as rated by experts). Analysis of the sentence structure revealed a severe lack of semantic logic in the summary. The summary contradicted itself by stating that Radame Falcao was in good form one moment and struggling and out of form the next at Manchester United. This lack of logical flow made reading difficult, resulting in low scores from both the semantic sub-model and experts in the coherence dimension. However, BERT for MTE scored nearly 5 points. In terms of fine-grained quality, the semantic sub-model in this embodiment demonstrates better scoring ability than BERT for MTE, and its scoring is closer to human scoring. Because the test abstract and the reference abstract have good N-gram overlap, ROUGE-1 gave this incomplete abstract a high score of 0.436. This demonstrates that ROUGE is indeed a crude metric that lacks interpretability. ROUGE cannot tell people the specific quality of the abstract, such as identifying whether there are factual errors or grammatical errors. In conclusion, extensive experiments show that the semantic sub-model in this embodiment significantly outperforms existing evaluation metrics in terms of human relevance and helps improve the diversity of generated abstracts.

[0098] Furthermore, this embodiment also provides a text summarization quality assessment system based on text matching, including a microprocessor and a memory interconnected thereto, the microprocessor being programmed or configured to execute the steps of the text summarization quality assessment method based on text matching.

[0099] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to perform the steps of the text summarization quality assessment method based on text matching.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A text summarization quality assessment method based on text matching, characterized in that, The method involves inputting the text summary to be evaluated and its corresponding original text into a pre-trained text summary quality assessment model to obtain the quality assessment result corresponding to the text summary to be evaluated. The text summary quality assessment model includes a semantic sub-model for measuring four fine-grained qualities: relevance, consistency, coherence, and fluency. The input of the semantic sub-model is the text summary to be evaluated and its corresponding original text, and the output is the quality assessment result of the corresponding fine-grained quality. The four fine-grained quality semantic sub-models have the same structure, all including a pre-trained cross-encoder and a regression model based on a multilayer perceptron. The input of the cross-encoder is the text summary to be evaluated and its corresponding original text, with the first token being a token character [CLS]. The text summary to be evaluated and its corresponding original text are separated by a delimiter character [SEP]. The final hidden state V corresponding to the token character [CLS] is... [CLS] As the final extracted feature, the multilayer perceptron-based regression model is used to map the final feature to a predicted score for a specified interval. In the semantic sub-model used to measure relevance and consistency, the final hidden state V corresponding to the token character [CLS] is... [CLS] The expression for the computation function is: V [CLS] = Cross-Encoder([CLS],O,[SEP],S) In the above formula, Cross-Encoder represents the interactive encoder Cross-Encoder, [CLS] is the token character, O is the original text, [SEP] is the delimiter character, and S is the text digest to be evaluated; In the semantic sub-model used to measure coherence and fluency, the final hidden state V corresponding to the token character [CLS] is... [CLS] The expression for the computation function is: V [CLS] = Cross-Encoder([CLS], [SEP],S) In the above formula, Cross-Encoder represents the interactive encoder Cross-Encoder, [CLS] is the token character, [SEP] is the separator character, and S is the text digest to be evaluated.

2. The text summarization quality assessment method based on text matching according to claim 1, characterized in that, The function expression for mapping the final features to predicted scores for a specified interval in a regression model based on a multilayer perceptron is as follows: y=WV [CLS] +b In the above formula, y is the predicted score, and the weights W and biases b are trainable network parameters.

3. The text summarization quality assessment method based on text matching according to claim 1, characterized in that, The loss function used during training of the four fine-grained quality semantic sub-models is: In the above formula, L For loss function, N The number of training samples. y The predicted score output by the fine-grained quality semantic sub-model, y The fitting labeled score for the training samples.

4. The text summarization quality assessment method based on text matching according to claim 1, characterized in that, Before inputting the text summary to be evaluated and its corresponding original text into the pre-trained text summarization quality evaluation model, the process also includes the step of training a semantic sub-model for each fine-grained quality level: S1, The initial sub-model for this fine-grained quality semantic sub-model. M 0. Use a labeled dataset for the first round of supervised training to obtain the optimal sub-model. M 0 best ; S2. Regarding the obtained optimal sub-model M 0 best Multiple rounds of semi-supervised training were performed using an unlabeled training dataset to improve the performance of the semantic sub-model by leveraging the unlabeled training data, resulting in an optimal fine-grained quality semantic sub-model. M best .

5. The text summarization quality assessment method based on text matching according to claim 4, characterized in that, Step S1 includes: S1.1, the optimal sub-model M 0 best Initialized as the initial sub-model of this fine-grained quality semantic sub-model M 0; S1.2, using a labeled training dataset D train For the optimal sub-model M 0 best Perform a total of specified quantities ep Supervised training, and for any one of them i The sub-model obtained from the second training M i+1 Labeled validation dataset D eval For each sub-model M i+1 The current optimal sub-model M 0 best To verify, if the sub-model M i+1 The validation results are better than the current best sub-model. M 0 best The verification results will then be used to transform the sub-model. M i+1 As the new current optimal sub-model M 0 best Otherwise, maintain the current optimal sub-model. M 0 best Unchanged; ultimately, upon completion of the specified quantity. ep The supervised tuning and training process will yield the final optimal sub-model. M 0 best As the final optimal sub-model M 0 best .

6. The text summarization quality assessment method based on text matching according to claim 5, characterized in that, Step S2 includes: S2.1, the labeled training dataset D train Add to training dataset D The optimal fine-grained quality semantic sub-model M best Initialize as the optimal sub-model M 0 best The unlabeled training dataset is divided into... k This yields an unlabeled training dataset. D 1~ D k ; Set the loop variable t Initialize to 1 to begin iterating through the unlabeled training dataset. D 1~ D k ; S2.2, using the previous round t -1 optimal submodel M t-1 best For the first t Unlabeled training dataset D t Label the data to obtain the corresponding pseudo-label dataset. D t pseudo and the pseudo-label dataset D t pseudo Add to training dataset D ; S2.3, will be the current round t Optimal sub-model M t best Initialized as the initial sub-model of this fine-grained quality semantic sub-model M 0; S2.4, using the training dataset D In response to this round t Optimal sub-model M t best Perform a total of specified quantities ep Supervised training, and for any one of them i The sub-model obtained from the second training M i+1 Labeled validation dataset D eval For each sub-model M i+1 This round t Optimal sub-model M t best To verify, if the sub-model M i+1 The verification results are better than those of this round. t Optimal sub-model M t best The verification results will then be used to transform the sub-model. M i+1 As a new round t Optimal sub-model M t best Otherwise, maintain the current round. t Optimal sub-model M t best Unchanged; ultimately, upon completion of the specified quantity. ep This supervised optimization training will be the final round of this training. t Optimal sub-model M t best As the final result of this round t Optimal sub-model M t best ; S2.5, the labeled validation dataset D eval For this round respectively t Optimal sub-model M t best Optimal fine-grained quality semantic sub-model M best Verification will be conducted if this round t Optimal sub-model M t best The validation results outperform the best fine-grained quality semantic sub-model. M best The verification results will then determine the outcome of this round. t Optimal sub-model M t best As a new optimal fine-grained quality semantic sub-model M best Otherwise, maintain the optimal fine-grained quality semantic sub-model. M best constant; S2.6, Check the loop variable t Greater than k If true, output the final optimal fine-grained quality semantic sub-model. M best If the loop terminates and exits, then the loop variable will be terminated; otherwise, the loop variable will be terminated. t Add 1, then proceed to step S2.

2.

7. A text summarization quality assessment system based on text matching, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to perform the steps of the text summarization quality assessment method based on text matching as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to perform the steps of the text summarization quality assessment method based on text matching as described in any one of claims 1 to 6.

Images