A block-wise part-of-speech prompting method for factual error correction

Abstract

Factual error correction aims to identify and correct factual errors in a given statement. Most existing methods rely on a language model masker to identify factual errors by masking tokens in a sentence. However, this approach often fails to ensure the compositional integrity of the masked chunks, resulting in corrections that deviate from the original sentence. To address this issue, we propose a chunking-based approach and part-of-speech hints for FEC, which aim to identify and correct factual errors that are consistent with the statement content. Specifically, we first describe a chunking-based masking module that chunks a sentence into multiple spans while preserving the compositional integrity of each span. We then label the part-of-speech information of each span and combine it with the evidence span predicted by the span boundary prediction module to form a hint that is used to generate a correction for each span-based masked sentence using a T5-based corrector. Finally, we use the correction score to select the final correction.

Description

Technical Field

[0001] The present invention belongs to the technical field of text generation, and more particularly, relates to a text error correction method based on block-based part-of-speech prompts. Background Art

[0002] The goal of factual error correction (FEC) is to identify and correct factual errors and inaccuracies in specific statements, thereby improving the accuracy and reliability of factual statements. With the increase in real-world statements, the demand for correcting factual errors or misinformation (e.g., hallucinations) generated by large language models (LLMs) and providing corresponding correction evidence is also increasing. To improve the efficiency of this task, automatic FEC methods have been proposed to automatically identify and correct evidence of factual errors.

[0003] Most existing FEC methods use language model maskers to identify possible factual errors in statements and then correct them using given evidence or documents. However, these methods mainly focus on using token-level masking strategies for correction, which often leads to correction results that are inconsistent with the original statement. For example, "US Airways Flight 1549 [mask]." is a statement masked by a token-level language model masker, which often leads to a correction that is inconsistent with the original statement, even if the correction is correct, such as "US Airways Flight 1549 crashed.". Therefore, in order to maintain consistency between the correction and the original statement, it is necessary to ensure that the correction does not deviate from the constituent elements of the original statement.

[0004] Most false claims are generated by replacing text spans with specific components in the original claim. For example, the false claim "Boeing 787 Dreamliner is a boat." is generated by replacing "boat" with "plane." Existing mask-based FEC methods mostly generate corrections based on evidence without considering the part-of-speech information of the mask span, such as "Boeing 787 Dreamliner is a twin-engine jet airliner." This can easily introduce unnecessary information or new errors. Therefore, in order to improve the effectiveness and accuracy of error correction, the part-of-speech information of the mask span must be considered, that is, "plane" is selected from the evidence, which is more consistent with the part-of-speech information of the mask span, instead of "twin-engine jet airliner." Summary of the Invention

[0005] The present invention aims to overcome the shortcomings of the prior art and provide a Chunking-based FEC with PoS Prompt (CPP) to extract text spans consistent with the composition and PoS of factual errors from correction evidence. Specifically, i) to avoid component deviation in correction, the present invention first introduces a Chunking-based masking module to identify potential false spans in the statement while preserving the component integrity of each span. As shown in the figure, the present invention chunks the input statement into multiple spans, each of which contains a specific component (such as subject, predicate, object, prepositional phrase, etc.). Then, the present invention applies a T5-based error corrector to correct each masked statement (such as: "US Airways Flight 1549 had [mask] onboard.") to achieve component alignment between the correction and the original statement. ii) to reduce the introduction of unnecessary information or new errors, the present invention first uses a pre-trained SpanBERT model to extract text spans related to the masked statement from the given evidence, and then combines the PoS of the masked span to extract the text spans related to the masked statement from the given evidence. Tagging is used as a prompt condition to enable the T5-based Corrector to generate corrections that are more consistent with the original claim. Finally, the Scoring module is used to select the final correction by considering the entailment score between the evidence and the candidate corrections and the consistency score between the original claim and the correction.

[0006] The present invention has the following beneficial effects:

[0007] 1) This paper introduces a Chunking-based masking module to effectively align false spans in the original claim, ensuring that the correction is consistent with the claim;

[0008] 2) This invention introduces a PoS-based prompt to adjust the corrector to reduce unnecessary information or new errors in the correction;

[0009] 3) The present invention adopts a new scoring method to select the final revision by combining the "implicity" and "consistency" scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flowchart of a specific implementation method of applying part-of-speech prompts based on language chunks to factual error correction in the present invention;

[0011] Figure 2 This is the overall framework diagram of the application of chunk-based part-of-speech prompts to factual error correction in the present invention; DETAILED DESCRIPTION

[0012] The specific embodiments of the present invention are described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0013] Example

[0014] Figure 1 This is a flowchart of a specific implementation method of the present invention based on the part-of-speech prompt applied to the fact error correction method; Figure 1 As shown, the specific steps of applying the part-of-speech prompt based on the chunk to fact correction in the present invention include:

[0015] S101: Constructing a block-based mask module:

[0016] To avoid the correction from deviating from the grammatical structure of the original statement, the present invention proposes to block the statement into multiple text spans according to its components and treat them as potential error spans; specifically, the present invention first uses a block-based masking module to block the statement c into multiple text spans S = {s1, s2, ...., s k}; The calculation formula is as follows:

[0017] S=Chunker(c)

[0018] Wherein, k represents the number of components in the claim;

[0019] Next, the present invention uses a masked language model (MLM) to mask consecutive spans according to the extracted spans, thereby generating a candidate mask statement m, such as "US Airways Flight 1549 had [mask] onboard.".

[0020] m i =MLM(c,s i ),i∈{1,2,...,k}

[0021] The present invention uses M={m1,m2,...,m k} to indicate a candidate mask declaration.

[0022] S102: Span boundary prediction module:

[0023] After obtaining the candidate masking claims, the next step of the present invention is to find the most relevant evidence span for each masking claim from the given evidence; specifically, the present invention first concatenates the masking claim and the given evidence E into an input sequence, and then inputs it into the cross-boundary objective (SBO) to predict the masking span; the calculation formula is as follows.

[0024] a i =SBO(mi ,E)

[0025] Among them, a i is the evidence span of the statement and can be used to generate correction prompts.

[0026] S103: Part-of-speech tagging module:

[0027] To address the inconsistency between the original statement and the correction, this paper proposes to construct a part-of-speech sequence (PoS) hint to help the correction model generate corrections that are more consistent with the original statement. Specifically, this paper first uses PoSTagging in the NLTK tool to obtain the PoS tag P of the original statement. The calculation formula is as follows:

[0028] P=PoSTagging(c)

[0029] Where P={p1,p2,...,p n}, p i represents the PoS tag of the i-th tag in c, and PoSTagging represents the PoS analysis of nltk; with the help of nltk, the present invention can obtain a lexical syntax tree; then, the present invention converts the lexical syntax tree into a one-dimensional tree; secondly, after obtaining P = {p1, p2, ..., p n}After that, the present invention connects them to form a PoS token.

[0030] S104: combining the output results of the block-based mask module, the span boundary prediction module, and the part-of-speech tagging module as a prompt.

[0031] After completing the above three steps, the present invention obtains the covered claim Highlight evidence And PoS mark P s =p1p2...p n ; In order to combine the structure of the highlighted evidence and claim c, the present invention combines the mask claim, the highlighted evidence and the PoS tag to create a candidate hint CP; the candidate hint of the i-th mask claim can be expressed as follows:

[0032]

[0033] Among them, [SEP] represents the separator between the shielded claim and the highlighted evidence, and the separator between the highlighted evidence and the PoS marker; here, cp i Represents the i-th candidate hint corresponding to the i-th blocked claim.

[0034] S105: Factual Error Correction Module:

[0035] After obtaining candidate hints CP=cp1,cp2,...,cp k Afterwards, the present invention inputs it into the T5 corrector after distant supervision fine-tuning to correct the masked statement to obtain candidate corrections.

[0036] c i ′=FEC(cp i )

[0037] Among them, it means that i-th is masked. Candidate revisions.

[0038] S201: Scoring module: After putting the candidate hint into the corrector, the present invention obtains the masked statement The corresponding candidate corrections c1′,c′2,...,c′ k .

[0039] The present invention takes into account the consistency of the candidate revision with the original claim and the consistency of the candidate revision with the evidence, and designs the following scoring function S, which is calculated as follows:

[0040]

[0041] Among them, NLI represents DocNLI, which is used to calculate the entailment relationship between documents and sentences; R1 represents ROUGH-1, which is used to calculate the similarity between the candidate revision and claim c.

[0042] Finally, by using the scoring function S to score all candidate revisions, the present invention selects the revision with the highest score as the final revision c″:

[0043]

[0044] To better illustrate the technical effects of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, two commonly used datasets are used: FEVER and SCIFACT. FEVER is a fact-checking dataset consisting of 185,000 claims supported or refuted by evidence collected from Wikipedia. SCIFACT is expert-annotated fact-checking data containing 1.4,000 scientific claims and paired evidence. To reuse these two datasets for FEC tasks, a redesigned method is used to form faithful claims and infidelity claims that need to be corrected.

[0045] To comprehensively evaluate the performance of the proposed method, we compared it with other baseline methods using three evaluation metrics: 1) We used BARTScore to measure the semantic overlap between claims and evidence, calculated as the log probability of generating evidence given a claim; 2) We used FACTCC, an entailment-based metric, to evaluate the fidelity between claims and evidence; 3) We used the vocabulary-based metric SARI to measure the improvement of the correction compared to the original claim. A higher SARI score indicates that the correction is more similar to the original claim.

[0046] This paper considers six baseline error correction models: T5-FULL is a powerful pre-trained baseline model trained in a fully supervised manner for error correction; T5-MASKCORRECT automatic error correction model is trained on supporting data samples and learns to correct errors by restoring masked claims using corresponding evidence; T5-DISTANT evidence model extends the masking method of T5-MASKCORRECT automatic error correction model by introducing a multi-mask method; REVISEREF, selectively rewrites unsupported claims by training and contrastive learning on synthetic datasets; COMPEDIT, focuses on entity errors and implements error correction; ZEROFEC, simulates the human ability to identify and correct factual errors, and designs an error correction framework.

[0047] The present invention is implemented by PyTorch and trained on an NVIDIA RTX 3090 GPU. The present invention uses a self-training t-5 model.

[0048] Table 1 is a statistical table of error correction results of the present invention and the comparative method under different data sets in this embodiment.

[0049]

[0050] As shown in Table 1, it can be seen from the results in Table 1 that the present invention achieves the best results in both FEVER and SCIFACT datasets, thereby verifying the effectiveness of the present invention.

[0051] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached statements, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A block part-of-speech prompting method for factual error correction, characterized in that: The following steps are involved: S101: Constructing a block-based mask module: According to the components of the statement, it is divided into multiple text spans and treated as potential error spans. The statement c is divided into multiple text spans S = {s1, s2, ...., s k }, the calculation formula is as follows: S=Chunker(c) Where k represents the number of components of the claim. Next, the masked language model MLM is used to mask the consecutive spans according to the extracted spans to generate a candidate masked claim m, so: m i =MLM(c,s i ),i∈{1,2,...,k} Where S101 uses M={m1,m2,...,m k } to indicate a candidate mask declaration; S102: Span boundary prediction module: After obtaining the candidate mask claims, the most relevant evidence span is found for each mask claim from the given evidence. The mask claim and the given evidence E are then concatenated into an input sequence, which is finally input into the cross-boundary target SBO to predict the mask span. The calculation formula is as follows: a i =SBO(m i ,E) Among them, a i is the evidence span of the statement and can be used to generate correction prompts; S103: Part-of-speech tagging module: Construct a part-of-speech sequence PoS prompt and use PoSTagging in the nltk tool to obtain the PoS tag P of the original statement. The calculation formula is as follows: P=PoSTagging(c) Where P = {p1, p2, ..., p n }, p i Indicates the PoS tag of the i-th tag in c, PoSTagging represents the PoS analysis of nltk; then, the lexical syntax tree is converted into a one-dimensional tree; after obtaining P = {p1, p2, ..., p n }After that, they are concatenated to form the PoS token.

2. A block part-of-speech prompting method for fact error correction according to claim 1, characterized in that: The step S104 is also included: The output of the combined block-based mask module, span boundary prediction module, and part-of-speech tagging module is the prompt; Got a covered statement Highlight evidence and PoS mark P s =p 1， p2...,p n ; Combine the masked claim, highlighted evidence, and PoS tag to create a candidate hint CP. The candidate hint for the i-th masked claim can be expressed as follows: Among them, [SEP] represents the separator between the shielded claim and the highlighted evidence, and the separator between the highlighted evidence and the PoS marker, cp i Represents the i-th candidate hint corresponding to the i-th blocked claim.

3. The method for block-wise part-of-speech prompting for factual error correction according to claim 2 further comprises step S105: Factual Error Correction Module: After obtaining candidate hints CP=cp1,cp2,...,cp k After that, it is fed into the distantly supervised fine-tuned T5 corrector to correct the masked statement to obtain candidate corrections: c i ′=FEC(cp i ) in, c i ′ represents the candidate revision of the i-th masked statement; Also includes scoring module: After feeding the candidate hint into the corrector, we obtain candidate corrections c1′,c′2,...,c′ corresponding to the masked statement. k Taking into account the consistency between the candidate revision and the original claim and the consistency between the candidate revision and the evidence, the following scoring function S is designed, and the calculation formula is as follows: Wherein, DocNLI represents DocNLI, which is used to calculate the entailment relationship between documents and sentences; R1 represents ROUGH-1, which is used to calculate the similarity between candidate revisions and claims; finally, all candidate revisions are scored using the scoring function S, and S302 selects the revision with the highest score as the final revision c″: .

Images