Block part-of-speech prompting method applied to fact error correction

Abstract

The fact error correction aims to identify and correct fact errors in a given statement. Most existing methods depend on a language model masker, and fact errors are identified through tokens in masked statements. However, this method often cannot ensure the component integrity of masked word blocks, resulting in correction deviating from the original statement. In order to solve the problem, the invention provides a Chunking-based method and a part-of-speech information prompt for FEC (Forward Error Correction), and aims to identify and correct fact errors consistent with declaration contents. Specifically, firstly, a Chunking-based mask module is described, the module can divide a statement into a plurality of spans, meanwhile, the component integrity of each span is reserved, then part-of-speech information of each span is marked, the part-of-speech information is combined with an evidence span predicted by a span boundary prediction module, and a prompt is formed. A correction is generated for each span-based mask statement using the cue to a T5-based corrector, and finally a final correction is selected using the correction score.

Description

Technical Field

[0001] The invention belongs to the technical field of text generation, and more specifically, relates to a text error correction method based on block-based part-of-speech prompts. Background Art

[0002] The goal of factual error correction (FEC) is to identify and correct factual errors and inaccuracies in specific claims, thereby improving the accuracy and reliability of factual claims. With the increase in real-world claims, the need to correct factual errors or misinformation (e.g., hallucinations) generated by large language models (LLMs) and provide corresponding correction evidence is also increasing. To improve the efficiency of this task, automatic FEC methods have been proposed to automatically identify and correct evidence of factual errors.

[0003] Most existing FEC methods use language model maskers to identify possible factual errors in a statement and then correct them with given evidence or documents; however, these methods mainly focus on using token-level masking strategies for correction, which often leads to correction results that are inconsistent with the original statement; for example, "US Airways Flight 1549 [mask]." is a statement masked by a token-level language model masker, which often leads to corrections that are inconsistent with the original statement even if the correction is correct, such as "US Airways Flight 1549 crashed."; therefore, in order to maintain consistency between the correction and the original statement, it is necessary to ensure that the correction does not deviate from the constituent elements of the original statement.

[0004] Most false claims are generated by replacing text spans with specific components in the original claim, for example, the false claim “Boeing 787 Dreamliner is a boat.” is generated by replacing “boat” with “plane”; most existing mask-based FEC methods generate corrections based on evidence without considering the part-of-speech information of the mask span, such as “Boeing 787 Dreamliner is a twin-engine jet airliner.”, which can easily introduce unnecessary information or new errors; therefore, in order to improve the effectiveness and accuracy of error correction, the part-of-speech information of the mask span must be considered, that is, “plane”, which is more consistent with the part-of-speech information of the mask span, is selected from the evidence instead of “twin-engine jet airliner”. Summary of the invention

[0005] The present invention aims to overcome the deficiencies of the prior art and provide a Chunking-based FEC with PoS Prompt (CPP), which aims to extract text spans consistent with the composition and PoS of factual errors from correction evidence; specifically, i) in order to avoid component deviation in correction, the present invention first introduces a Chunking-based masking module to identify potential false spans in the statement while retaining the component integrity of each span; as shown in the figure, the present invention chunks the input statement into multiple spans, each span containing a specific component (such as subject, predicate, object, prepositional phrase, etc.); then, the present invention applies a T5-based corrector to correct each masked statement (such as: "US Airways Flight 1549 had [mask] onboard.") to achieve component alignment between the correction and the original statement; ii) in order to reduce the introduction of unnecessary information or new errors, the present invention first uses a pre-trained SpanBERT model to extract text spans related to the masked statement from the given evidence, and then combines the PoS of the masked span to extract the text spans related to the masked statement from the given evidence. Tagging is used as a prompt condition to enable the T5-based Corrector to generate corrections that are more consistent with the original claim. Finally, the Scoring module is used to select the final correction by considering the entailment score between the evidence and the candidate corrections and the consistency score between the original claim and the correction.

[0006] The present invention has the following beneficial effects:

[0007] 1) This invention introduces a Chunking-based mask module to effectively align the false spans in the original claim and ensure that the correction is consistent with the claim;

[0008] 2) The present invention introduces a PoS-based prompt to adjust the corrector to reduce unnecessary information or new errors in the correction;

[0009] 3) The present invention adopts a new scoring method to select the final revision by combining the "implicity" and "consistency" scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of a specific implementation method of applying the part-of-speech prompt based on the language block to factual error correction of the present invention;

[0011] Figure 2 It is the overall framework diagram of the application of the chunk-based part-of-speech prompt in the present invention to factual error correction; DETAILED DESCRIPTION

[0012] The specific implementation modes of the present invention are described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be ignored here.

[0013] Example

[0014] Figure 1 1 is a flowchart of a specific implementation method of the present invention based on the part-of-speech prompt based on the language block and applied to the factual error correction method; Figure 1 As shown, the specific steps of applying the part-of-speech prompt based on the language block to factual error correction in the present invention include:

[0015] S101: Constructing a block-based mask module:

[0016] In order to avoid the correction from deviating from the grammatical structure of the original statement, the present invention proposes to block the statement into multiple text spans according to its components and take them as potential erroneous spans; specifically, the present invention first uses a block-based masking module to block the statement c into multiple text spans S = {s1, s2, ...., s k}; The calculation formula is as follows:

[0017] S=Chunker(c)

[0018] Where k represents the number of components in the claim;

[0019] Next, the present invention uses a masked language model (MLM) to mask consecutive spans according to the extracted spans, thereby generating a candidate mask statement m, such as "US Airways Flight 1549 had [mask] onboard.".

[0020] m i =MLM(c,s i ),i∈{1,2,...,k}

[0021] The present invention uses M = {m1, m2, ..., m k} to indicate a candidate mask declaration.

[0022] S102: Span boundary prediction module:

[0023] After obtaining the candidate masking claims, the next step of the present invention is to find the most relevant evidence span for each masking claim from the given evidence; specifically, the present invention first concatenates the masking claim and the given evidence E into an input sequence, and then inputs it into the cross-boundary objective (SBO) to predict the masking span; the calculation formula is as follows.

[0024] a i =SBO(mi ,E)

[0025] Among them, a i is the evidence span of the statement and can be used to generate correction prompts.

[0026] S103: Part-of-speech tagging module:

[0027] In order to solve the problem of inconsistency between the original statement and the correction, the present invention proposes to construct a part-of-speech sequence (PoS) prompt to help the correction model generate corrections that are more consistent with the original statement; specifically, the present invention first uses PoSTagging in the NLTK tool to obtain the PoS tag P of the original statement; the calculation formula is as follows:

[0028] P=PoSTagging(c)

[0029] Where P = {p1, p2, ..., p n}, p i represents the PoS tag of the i-th tag in c, and PoSTagging represents the PoS analysis of nltk; with the help of nltk, the present invention can obtain a lexical syntax tree; then, the present invention converts the lexical syntax tree into a one-dimensional tree; secondly, after obtaining P = {p1, p2, ..., p n}Afterwards, the present invention connects them to form a PoS tag.

[0030] S104: combining the output results of the block-based mask module, the span boundary prediction module and the part-of-speech tagging module as a prompt.

[0031] After completing the above three steps, the present invention obtains the covered claim Highlight the evidence And PoS mark P s =p1p2...p n ; In order to combine the structure of the highlighted evidence and the claim c, the present invention combines the mask claim, the highlighted evidence and the PoS tag to create a candidate hint CP; the candidate hint of the i-th mask claim can be expressed as follows:

[0032]

[0033] Where [SEP] represents the separator between the shielded claim and the highlighted evidence, and the separator between the highlighted evidence and the PoS marker; here, cp i Denotes the i-th candidate suggestion corresponding to the i-th blocked claim.

[0034] S105: Factual Error Correction Module:

[0035] After obtaining the candidate hint CP=cp1,cp2,...,cp k Afterwards, the present invention inputs it into the T5 corrector after distant supervision fine-tuning to correct the masked statement to obtain candidate corrections.

[0036] c i ′=FEC(cp i )

[0037] Among them, it means that the i-th is masked. Candidate corrections.

[0038] S201: Scoring module: After putting the candidate hint into the corrector, the present invention obtains the masked statement The corresponding candidate corrections c1′,c′2,...,c′ k .

[0039] The present invention takes into account the consistency between the candidate revision and the original claim and the consistency between the candidate revision and the evidence, and designs the following scoring function S, which is calculated as follows:

[0040]

[0041] Among them, NLI stands for DocNLI, which is used to calculate the entailment relationship between documents and sentences; R1 stands for ROUGH-1, which is used to calculate the similarity between the candidate revision and claim c.

[0042] Finally, by using the scoring function S to score all candidate revisions, the present invention selects the revision with the highest score as the final revision c″:

[0043]

[0044] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, two commonly used datasets are used: FEVER and SCIFACT. FEVER is a fact-checking dataset consisting of 185k claims supported or refuted by evidence collected from Wikipedia. SCIFACT is expert-annotated fact-checking data containing 1.4K scientific claims and paired evidence. In order to reuse these two datasets for FEC tasks, the method is redesigned to form faithful claims and unfaithful claims that need to be corrected.

[0045] In order to comprehensively evaluate the performance of the method proposed in the present invention, the present invention compares the method of the present invention with other baseline methods under these three evaluation indicators: 1) The present invention uses BARTScore to measure the semantic overlap between the claim and the evidence. The calculation method is the logarithmic probability of generating evidence given the claim; 2) The present invention uses FACTCC to evaluate the fidelity between the claim and the evidence. FACTCC is an implication-based indicator; 3) The present invention uses the vocabulary-based indicator SARI to measure the improvement of the correction compared with the original claim. The higher the SARI score, the more similar the correction is to the original claim.

[0046] Six baseline error correction models are considered in this paper: T5-FULL is a powerful pre-trained baseline model trained in a fully supervised manner for error correction; T5-MASKCORRECT automatic error correction model is trained on supporting data samples and learns to correct errors by restoring masked claims using corresponding evidence; T5-DISTANT evidence model extends the masking method of T5-MASKCORRECT automatic error correction model by introducing a multi-mask method; REVISEREF, selectively rewrites unsupported claims by training and contrastive learning on synthetic datasets; COMPEDIT, focuses on entity errors and realizes error correction; ZEROFEC, simulates the human ability to identify and correct factual errors and designs an error correction framework.

[0047] The present invention is implemented by PyTorch and trained on NVIDIA RTX3090 GPU. The present invention uses a self-training t-5 model.

[0048] Table 1 is a statistical table of error correction results of the present invention and the comparative method under different data sets in this embodiment.

[0049]

[0050] As shown in Table 1, it can be seen from the results in Table 1 that the present invention has achieved the best results in both the FEVER and SCIFACT datasets, thereby verifying the effectiveness of the present invention.

[0051] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A block part-of-speech prompting method for factual error correction, characterized in that: The following steps are involved: S101: Constructing a block-based mask module: To avoid the correction from deviating from the grammatical structure of the original statement, S101 divides the statement into multiple text spans according to its components and regards them as potential erroneous spans. S101 first uses a block-based masking module to divide the statement c into multiple text spans S = {s1, s2, ..., s k }, the calculation formula is as follows: S=Chunker(c) Where k represents the number of components of the claim. Next, S101 uses a masked language model (MLM) to mask consecutive spans according to the extracted spans to generate a candidate masked claim m, such as "US Airways Flight 1549had[mask]onboard.", so: m i =MLM(c,s i ),i∈{1,2,...,k} Where S101 uses M = {m1,m2,...,m k } to indicate a candidate mask declaration; S102: Span boundary prediction module: After obtaining the candidate mask claims, S102 finds the most relevant evidence span for each mask claim from the given evidence, then concatenates the mask claim and the given evidence E into an input sequence, and finally inputs it into the cross-boundary objective (SBO) to predict the mask span, which is calculated as follows: a i =SBO(m i ,E) Among them, a i is the evidence span of the statement, which can be used to generate correction prompts; S103: Part-of-speech tagging module: S103 constructs a part-of-speech sequence (PoS) hint to help the correction model generate corrections that are more consistent with the original statement. S103 first uses PoSTagging in the NLTK tool to obtain the PoS tag P of the original statement. The calculation formula is as follows: P=PoSTagging(c) Where P = {p1, p2, ..., p n }, p i represents the PoS tag of the i-th tag in c, PoSTagging represents the PoS analysis of nltk. With the help of nltk, S103 can obtain the lexical syntax tree; then, the lexical syntax tree is converted into a one-dimensional tree; secondly, after obtaining P = {p1, p2, ..., p n }After that, S103 connects them to form a PoS token.

2. A block part-of-speech prompting method for fact error correction according to claim 1, characterized in that: The final input prompt is constructed as follows: S201: combining the output results of the block-based mask module, the span boundary prediction module and the part-of-speech tagging module as a prompt; S201 first completes the combination of the output results of the block-based mask module, the span boundary prediction module and the part-of-speech tagging module to obtain the masked statement Highlight the evidence And PoS mark P s =p1p2...p n ; S201 combines the masked claims, highlighted evidence, and PoS tags to create candidate hints CP. The candidate hints for the i-th masked claim can be expressed as follows: Among them, [SEP] represents the separator between the shielded claim and the highlighted evidence, and the separator between the highlighted evidence and the PoS marker, cp i Denotes the i-th candidate suggestion corresponding to the i-th blocked claim.

3. The block part-of-speech prompt method for fact error correction according to claim 2 is characterized in that: Correction of factual errors is carried out in the following manner: S301: Factual Error Correction Module: After obtaining the candidate hint CP=cp1,cp2,...,cp k After that, S301 inputs it into the T5 corrector after distant supervision fine-tuning to correct the masked statement to obtain candidate corrections: c i ′=FEC(cp i ) Among them, c i ' indicates that the i-th is masked declaration Candidate amendments for S302: Scoring module: After placing the candidate hint into the corrector, S302 obtains the statement with the masked The corresponding candidate corrections c1′,c′2,...,c′ k ; S302 Considering the consistency between the candidate revision and the original claim and the consistency between the candidate revision and the evidence, the following scoring function S is designed, and the calculation formula is as follows: Wherein, DocNLI represents DocNLI, which is used to calculate the implication relationship between documents and sentences; R1 represents ROUGH-1, which is used to calculate the similarity between the candidate revision and the claim c; finally, all candidate revisions are scored by using the scoring function S, and S302 selects the revision with the highest score as the final revision c″:

Images