Two-stage few-sample automatic fact checking method, electronic equipment and storage medium

Abstract

The invention discloses a two-stage few-sample automatic fact checking method, electronic equipment and a storage medium, and belongs to the technical field of information processing. In order to improve the accuracy and efficiency of information verification, the method comprises the following steps: selecting a public fact verification data set to construct a test set; constructing an evidence retrieval model, and retrieving the documents of the test set by adopting the evidence retrieval model to obtain candidate documents; constructing a sentence selection model, inputting each sentence in the candidate document and the statement into a prompt template to be combined into the input of the sentence selection model, and outputting the correlation score of the sentences as an evidence sentence for statement verification; constructing a pre-training language model and finely adjusting the pre-training language model; and inputting an evidence sentence for statement verification and a statement to be verified into a prompt template to be combined into input of a pre-training language model, inputting the input into the pre-training language model after fine tuning to obtain natural language output, then mapping the natural language output to a classification label, and calculating a prediction score according to the generation probability of an output sequence to obtain a final prediction result.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and specifically relates to a two-stage few-sample automatic fact-checking method, an electronic device and a storage medium. Background Art

[0002] In the current era of rapid development of the Internet, people are surrounded by a large amount of information every day, which comes from various sources, including social media, news websites, blogs and other platforms. Whether as an ordinary person or a journalist, it is becoming increasingly difficult to judge the authenticity of information in the massive amount of information. Therefore, in this era of "information explosion", how to quickly and accurately detect false information has become an urgent problem to be solved.

[0003] With the popularity of social media and the rapid growth of digital content, fake news and misleading information have also increased accordingly. This type of information may mislead the public, intensify social divisions, and have a negative impact on public trust. In addition, although many news organizations and independent groups are committed to fact-checking, manual verification is a time-consuming and resource-intensive process that is difficult to keep up with the rapid flow of information in real time, especially in the Internet age where new topics and hot spots are constantly emerging.

[0004] Therefore, it is necessary to develop automatic fact-checking technology. First, automatic fact-checking technology can quickly process large amounts of information, provide preliminary authenticity assessments, and assist human verifiers in conducting verification more efficiently. Through automated tools, misleading information can be identified and corrected more quickly, thereby protecting the public from being misled by fake news and enhancing trust in the media and other sources of information. Secondly, accurate and timely information is crucial for various decision makers, such as policymakers, business leaders, and the general public. Automatic fact-checking can ensure that decisions are based on true and accurate information. In addition, automatic fact-checking is a cutting-edge research topic in the field of natural language processing, and research on it will further promote the development of related technologies. Summary of the invention

[0005] The problem to be solved by the present invention is to improve the accuracy and efficiency of information verification, and a two-stage small-sample automatic fact-verification method, an electronic device and a storage medium are proposed.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A two-stage few-shot automatic fact-checking method, comprising the following steps:

[0008] S1. Select a public fact verification dataset to build a training set and a test set, which contains the claims to be verified and the knowledge base;

[0009] S2. Use BM25 to build an index for the documents in the knowledge base as a document retrieval model, use the statement of the test set in step 1 as a query, and retrieve the first several documents that are most relevant to the statement in step S1 as candidate documents;

[0010] S3. Build a sentence selection model, input each sentence in the candidate document obtained in step S2 and the statement in step S1 into the prompt template prompt as the input of the sentence selection model, input it into the sentence selection model, and output the relevance score of the sentence;

[0011] S4. Based on the relevance scores of the sentences obtained in step S3, select the top several sentences as evidence sentences for claim verification;

[0012] S5. Build a pre-trained language model and fine-tune the pre-trained language model using a few-sample training set extracted from the training set;

[0013] S6. The evidence sentence for statement verification obtained in step S4 and the statement to be verified are input into the prompt template prompt as the input of the pre-trained language model, and then input into the fine-tuned pre-trained language model to obtain natural language output, which is then mapped to the classification label, and the prediction score is calculated based on the generation probability of the output sequence to obtain the final prediction result.

[0014] Furthermore, the public fact verification dataset described in step S1 includes one or a combination of the FEVER dataset, the SciFact dataset and the VitaminC dataset, wherein the FEVER dataset makes artificially generated claims by modifying the content in Wikipedia and provides standard evidence for supporting and refuting the labeled claims; the SciFact dataset is composed of scientific claims formed by experts rewriting sentences in biomedical literature, and the VitaminC dataset is created by using a revised version of Wikipedia to construct a test set.

[0015] Furthermore, the specific implementation method of step S2 includes the following steps:

[0016] S2.1. Construct the evidence retrieval model as the probability retrieval model BM25, calculate the frequency of query words in the document and the vocabulary distribution in the document through statistical methods, and generate the query word q i The relevance score score(q,d) in document d is expressed as:

[0017]

[0018] Among them, q represents the query word, d represents the document, and IDF(q i ) represents the query word q i The inverted document frequency, f(qi ,d) represents the query word q i The frequency of occurrence in document d, |d| and avgdl represent the document length and the average length of all documents respectively, k1 and b are the first adjustable parameter and the second adjustable parameter respectively;

[0019] S2.2. Use Elastic Search to build BM25 indexes for SciFact and FEVER respectively. Use the evidence retrieval model built in step S2.1 to search the test set obtained in step S1 based on the statement, and evaluate the retrieval results. The evaluation criteria used are ndcg@n and recall@n. The top 100 documents most relevant to the statement in step S1 are obtained as candidate documents.

[0020] Furthermore, the specific implementation method of step S3 includes the following steps:

[0021] S3.1. Build a sentence selection model and use the checkpoints released by the MonoT5 model for sentence selection tasks. MonoT5 is a retrieval model fine-tuned using the MS MARCO paragraph dataset. Use the checkpoints released by the model for sentence selection tasks.

[0022] The sentence selection model generates a target token of 'True' or 'False' based on whether the document is relevant to the statement. In the ranking task, the output results of the sentence selection model are aggregated, and the probability of assigning the token 'True' is calculated as the relevance score to generate the relevance score of the sentence. The expression is:

[0023]

[0024] Among them, relevant = 1 means that the statement is relevant to the document, logit('True') represents the probability of correct assignment of the target token, and logit('False') represents the probability of incorrect assignment of the target token;

[0025] S3.2. Split the candidate document obtained in step S2 into sentences, and then combine each sentence in the candidate document obtained in step S2 with the statement in step S1 into the prompt template prompt. The expression of the prompt template prompt is:

[0026] Query:[Q]Document:[D]Relevant:

[0027] Among them, Q is the statement in the dataset, and D is the document sentence to be evaluated;

[0028] S3.3. Input the combination obtained in step S3.2 into the sentence selection model for reordering and output the relevance score of the sentence.

[0029] Furthermore, in step S4, the top five sentences are selected as evidence sentences for claim verification.

[0030] Furthermore, the specific implementation method of step S5 includes the following steps:

[0031] S5.1. Select Flan-T5 as the base model of the pre-trained language model;

[0032] S5.2. Fine-tune the base model of the pre-trained language model using the LoRA fine-tuning method. The rank of the matrix in training is specified as 4. LoRA is applied to the (q|k|v|o|w) layers of the SelfAttention, EncDecAttention, and DenseReluDense modules of the original model to obtain the pre-trained language model.

[0033] Furthermore, the specific implementation method of step S6 includes the following steps:

[0034] S6.1. Set the input of the fact-checking task to be verified by the statement C i and the evidence sentence e obtained in step S4 for the statement verification i Composition, will (C i ,e i ) into the prompt template, the output x i Mapped to label y i , the prompt template example is:

[0035] Suppose i}, canwe inferthat{C i}?;

[0036] S6.2. Based on the model input x i , generate a target sequence y i The probability p θ (y i ,x i ):

[0037]

[0038] Among them, p θ (t j |x i ,t <j ) is the model in the autoregressive process given the input sequence x i and the previously generated sequence t <j Under the condition of generating t j The probability of

[0039] Due to the correspondence between the output sequence and the classification label, the model is i The prediction score of is defined as the log-length normalized probability of generating the target sequence, which is:

[0040]

[0041] All categories are ranked using the prediction scores, and the top-ranked category is taken as the final category prediction.

[0042] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a two-stage few-sample automatic fact-checking method when executing the computer program.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the two-stage few-sample automatic fact-checking method.

[0044] Beneficial effects of the present invention:

[0045] The present invention discloses a two-stage few-sample automatic fact-checking method, including few-sample evidence retrieval and few-sample claim verification. In claim verification, using the evidence sentences (perfect evidence) given by the data set, the accuracy reaches the SOTA (current highest level) of the fully supervised model under the condition of no more than 50 sets of training data. In the task of precise retrieval of few-sample evidence sentences, a retrieval method combining BM25 and Seq2Seq language models is used, which achieves results slightly inferior to those reported by the fully supervised model.

[0046] The two-stage few-sample automatic fact-checking method described in the present invention adopts a few-sample setting in both stages of the fact-checking task. This setting method enables the entire fact-checking link to achieve good results without too much training data. This is of great significance in practical applications and can greatly reduce the cost and workload of data collection and processing.

[0047] The two-stage few-sample automatic fact-checking method described in the present invention uses a smaller number of model parameters. A smaller number of parameters means lower training costs and computing resource requirements. Compared with other models of the same size, the method of the present invention shows the best effect on the FEVER dataset. This fully proves the effectiveness and superiority of the method and provides an efficient and low-cost solution for the field of fact-checking. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a two-stage few-sample automatic fact-checking method according to the present invention;

[0049] Figure 2It is a structural block diagram of a two-stage few-sample automatic fact-checking method according to the present invention;

[0050] Figure 3 The figures are comparison diagrams of the effects of the model of the present invention and ProToCo-3B, wherein (a) is a comparison of the SciFact data set, (b) is a comparison of the FEVER data set, and (c) is a comparison of the VitaminC data set. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0052] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0053] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 3 The detailed instructions are as follows:

[0054] Embodiment 1:

[0055] A two-stage few-shot automatic fact-checking method, comprising the following steps:

[0056] S1. Select a public fact verification dataset to build a training set and a test set, which contains the claims to be verified and the knowledge base;

[0057] Further, the public fact verification dataset described in step S1 includes one or a combination of the FEVER dataset, the SciFact dataset, and the VitaminC dataset, wherein the FEVER dataset produces artificially generated claims by modifying the content in Wikipedia and provides standard evidence for supporting and refuting the labeled claims; the SciFact dataset is composed of scientific claims formed by experts rewriting sentences in biomedical literature, and the VitaminC dataset is created by using a revised version of Wikipedia to construct a test set;

[0058] The relevant statistical information of the data set is shown in Table 1:

[0059] Table 1 Statistics of the collected datasets

[0060]

[0061] Because the dataset only provides perfect evidence for claims with certain labels, some sentences are randomly selected as evidence for claims with NEI labels.

[0062] S2. Use BM25 to build an index for the documents in the knowledge base as a document retrieval model, use the statement of the test set in step 1 as a query, and retrieve the first several documents that are most relevant to the statement in step S1 as candidate documents;

[0063] Evidence retrieval is an essential part of the fact-checking pipeline, which aims to find evidence related to the statement in massive documents. Evidence often needs to be relatively fine-grained fragments such as sentences. After investigation, FEVER and Scifact were both included in a retrieval benchmark called BEIR (BenchMarking Information Retrieval) to evaluate retrieval systems, and both provide sentence-level annotations of relevant evidence. Therefore, this implementation is conducted on these two datasets and compared with other methods.

[0064] After investigation, it was found that the evidence retrieval in existing work is usually divided into two stages: document retrieval and sentence selection. Document retrieval selects documents related to the claim from a large number of documents, and sentence selection selects more accurate evidence sentences from these documents. However, the current open source evidence retrieval methods basically do not meet the few-sample and zero-sample conditions, that is, all training data are used to train the retrieval model. Therefore, this implementation method studies how to retrieve possible evidence without using a large amount of training data to form a few-sample fact-checking pipeline.

[0065] Furthermore, the specific implementation method of step S2 includes the following steps:

[0066] S2.1. Construct the evidence retrieval model as the probability retrieval model BM25, calculate the frequency of query words in the document and the vocabulary distribution in the document through statistical methods, and generate the query word q i The relevance score score(q,d) in document d is expressed as:

[0067]

[0068] Among them, q represents the query word, d represents the document, and IDF(q i ) represents the query word q iThe inverted document frequency, f(q i ,d) represents the query word q i The frequency of occurrence in document d, |d| and avgdl represent the document length and the average length of all documents respectively, k1 and b are the first adjustable parameter and the second adjustable parameter respectively;

[0069] S2.2. Use Elastic Search to build BM25 indexes for SciFact and FEVER respectively. Use the evidence retrieval model built in step S2.1 to search the test set obtained in step S1 based on the statement, and evaluate the retrieval results. The evaluation criteria used are ndcg@n and recall@n. The top 100 documents most relevant to the statement in step S1 are obtained as candidate documents.

[0070] The results are shown in Table 2:

[0071] Table 2 Document retrieval results using BM25

[0072]

[0073] It can be seen that among the first 100 documents in the retrieval results, the recall rate reaches a relatively high level and can be used as the input document set for the next sentence selection.

[0074] S3. Build a sentence selection model, input each sentence in the candidate document obtained in step S2 and the statement in step S1 into the prompt template prompt as the input of the sentence selection model, input it into the sentence selection model, and output the relevance score of the sentence;

[0075] The current Transformer model pre-trained with language modeling objectives has been proven to be very effective in various classification and sequence labeling tasks in NLP. Using a complex retrieval model to reorder the previous retrieval results has also become the standard architecture of all current Transformer-based document retrieval methods. This implementation method believes that sentence selection can also be regarded as a reordering task, and is therefore also applicable to these reordering retrieval models. Split the candidate document described in step 2 into sentences, input the sentence selection model of this step for reordering, and select several pieces of evidence that best meet the needs.

[0076] Furthermore, the specific implementation method of step S3 includes the following steps:

[0077] S3.1. Build a sentence selection model and use the checkpoints released by the MonoT5 model for sentence selection tasks. MonoT5 is a retrieval model fine-tuned using the MS MARCO paragraph dataset. Use the checkpoints released by the model for sentence selection tasks.

[0078] The sentence selection model generates a target token of 'True' or 'False' based on whether the document is relevant to the statement. In the ranking task, the output results of the sentence selection model are aggregated, and the probability of assigning the token 'True' is calculated as the relevance score to generate the relevance score of the sentence. The expression is:

[0079]

[0080] Among them, relevant = 1 means that the statement is relevant to the document, logit('True') represents the probability of correct assignment of the target token, and logit('False') represents the probability of incorrect assignment of the target token;

[0081] Furthermore, MonoT5 is a retrieval model based on the T5 model (base, large and 3B) and fine-tuned using the MS MARCO paragraph dataset; the method of using the above-mentioned Seq2Seq model for sentence sorting tasks is similar to claim verification, both of which require the input document and claim to be constructed as input through prompt templates, and then the output content is used to obtain a similarity score.

[0082] S3.2. Split the candidate document obtained in step S2 into sentences, and then combine each sentence in the candidate document obtained in step S2 with the statement in step S1 into the prompt template prompt. The expression of the prompt template prompt is:

[0083] Query:[Q]Document:[D]Relevant:

[0084] Among them, Q is the statement in the dataset, and D is the document sentence to be evaluated;

[0085] S3.3. Input the combination obtained in step S3.2 into the sentence selection model for reordering and output the relevance score of the sentence.

[0086] S4. Based on the relevance scores of the sentences obtained in step S3, select the top several sentences as evidence sentences for claim verification;

[0087] Further, in step S4, the top five sentences are selected as evidence sentences for claim verification;

[0088] S5. Build a pre-trained language model and fine-tune the pre-trained language model using a few-sample training set extracted from the training set;

[0089] Furthermore, the specific implementation method of step S5 includes the following steps:

[0090] S5.1. Select Flan-T5 as the base model of the pre-trained language model;

[0091] Furthermore, we investigated the models of the T5 (Text-to-Text Transfer Transformer) architecture and selected one of them as the base model. T5 is an encoder-decoder Transformer model that is pre-trained on a large corpus of unlabeled text data by predicting masked target words. It treats all NLP tasks as text-to-text conversion problems and can be applied to a variety of downstream tasks. At the same time, models of the T5 architecture often have versions of various sizes to adapt to different computing resources and task requirements, and you can choose the version that suits the application scenario.

[0092] T0 is created by fine-tuning T5 on a multi-task mixed dataset to achieve zero-shot generalization, i.e., the ability to perform tasks without any additional gradient-based training. The examples in the dataset used to train T0 are prompted by applying a prompt template from the Public Pool of Prompts, which converts each example in each dataset into a text-to-text format of a prompt.

[0093] Flan-T5 is a series of model checkpoints released by Google, which use various instruction template types to fine-tune instructions on a data source set (called Flan fine-tuning), and also combine several thought chaining datasets. These checkpoints have strong few-shot, zero-shot learning capabilities and thought chaining reasoning capabilities, which are better than the previous T5 series checkpoints.

[0094] S5.2. Fine-tune the base model of the pre-trained language model using the LoRA fine-tuning method. The rank of the matrix in training is specified as 4. LoRA is applied to the (q|k|v|o|w) layers of the SelfAttention, EncDecAttention, and DenseReluDense modules of the original model to obtain the pre-trained language model.

[0095] Furthermore, the core idea of ​​the method is to use the product of two low-rank matrices to approximate the update of the original weight matrix. By setting the rank of the matrix, the size of the parameters to be updated can be controlled. Therefore, LoRA can significantly reduce the number of parameters to be trained, reduce the demand for computing resources, and improve the stability of training on a small amount of data. At the same time, it is also possible to control the specific parts of LoRA applied to the model, such as the query (Q), key (K), and value (V) matrices in the self-attention mechanism.

[0096] During training, the rank of the matrix is ​​specified as 4, and LoRA is applied on the (q|k|v|o|w) layers of the SelfAttention, EncDecAttention, and DenseReluDense modules of the original model.

[0097] In terms of the use of loss functions, the loss function used is cross entropy loss. Using the cross entropy loss function will make the model move towards increasing the correct target sequence y i Probability direction training;

[0098]

[0099] Among them, p θ (t j |x i ,t <j ) is the model in the autoregressive process given the input sequence x i and the previously generated sequence t <j Under the condition of generating t j The probability of y i is the target sequence;

[0100] Because it is a classification task, a prediction score can be calculated for each category, so that the label y can be calculated i Maximizing the softmax probability of , also helps the model increase the probability of predicting the correct category:

[0101]

[0102] L cls = -log q θ (y i |x i )

[0103] Among them, γ is the set of all target sequences, p θ (y i ,x i ) indicates that the model is given an input sequence x in the autoregressive process i and the previously generated sequence t <j Generate the target sequence y under the condition i The probability of exp represents the natural exponential function, q θ (y i |x i ) represents y i The softmax probability of

[0104] Maximizing the probability of the correct category is equivalent to minimizing the probability of the wrong category. Therefore, this paper introduces the impossibility loss so that the model assigns a lower probability to incorrect category sequences:

[0105]

[0106] Among them, || represents the length;

[0107] In terms of training data selection, since fact checking is a three-classification problem, in order to balance the categories, this paper calls a training data containing a triplet of {SUPPORT, RRFUTE, NEI} a shot, and randomly selects [1, 2, 4, 8, 16] shots of training data from the dataset as the training set.

[0108] S6. The evidence sentence for statement verification obtained in step S4 and the statement to be verified are input into the prompt template prompt as the input of the pre-trained language model, and then input into the fine-tuned pre-trained language model to obtain natural language output, which is then mapped to the classification label, and the prediction score is calculated based on the generation probability of the output sequence to obtain the final prediction result.

[0109] Furthermore, the specific implementation method of step S6 includes the following steps:

[0110] S6.1. Set the input of the fact-checking task to be verified by the statement C i and the evidence sentence e obtained in step S4 for the statement verification i Composition, will (C i ,e i ) into the prompt template, the output x i Mapped to label y i , the expression of the prompt template is:

[0111] Suppose i}, can we infer that{C i}?;

[0112] Furthermore, label y i There are three categories: support, refutation, and insufficient information (SUPPORT, RRFUTE, NEI);

[0113] S6.2. Based on the model input x i , generate a target sequence y i The probability p θ (y i ,x i ):

[0114]

[0115] Among them, p θ (t j |x i ,t <j ) is the model in the autoregressive process given the input sequence x i and the previously generated sequence t <j Under the condition of generating t j The probability of

[0116] Due to the correspondence between the output sequence and the classification label, the model is i The prediction score of is defined as the log-length normalized probability of generating the target sequence, which is:

[0117]

[0118] All categories are ranked using the prediction scores, and the top-ranked category is taken as the final category prediction.

[0119] The experimental results of this implementation are as follows:

[0120] Using the training set of [1, 2, 4, 8, 16] randomly selected shot training data from FEVER, SciFact, and VitaminC, LoRA fine-tuning was applied on the (q|k|v|o|w) layer of the SelfAttention, EncDecAttention, and DenseReluDense modules of Flan-T5-large, and tested on the validation set of each dataset, using the macro F1 value as the classification evaluation indicator. The experimental results are shown in Table 3.

[0121] Table 3 Macro-average F1 score of models fine-tuned using different sizes of training data on the validation set

[0122]

[0123] Furthermore, the results are compared with existing few-sample fact verification methods. The baseline model involved in the comparison is the 3B checkpoint of ProToCo, which is also a fact verification model based on the T5 architecture. It uses T0 as the backbone model and uses some consistency constraints to expand the training data to perform few-sample fine-tuning on the relevant data set. The model size of this embodiment is 780M. The comparison results are shown in Figure 2. Figure 3 As shown, the model of this embodiment uses a smaller number of parameters and surpasses the baseline method on various data sets, especially under the condition of very few samples.

[0124] In addition, on the FEVER dataset, many predecessors used all training data to train fully supervised models for fact verification, and compared the label accuracy of these models using perfect evidence. The comparison results are shown in Table 4. The model fine-tuned using 4 shots, or 12 training data, has exceeded the accuracy of using all training data.

[0125] Table 4 Comparison of the accuracy of the fully supervised model and the proposed model using FEVER perfect evidence

[0126]

[0127] Based on the results of BM25, we take the first 100 documents for each query, and use the base-sized monoT5 to score and sort the sentences in them. The retrieval results are shown in Table 5:

[0128] Table 5 Sentence retrieval results using monoT5

[0129]

[0130] Since there are few results for zero-shot evidence sentence retrieval, the results that can be compared are also limited. On the FEVER dataset, the zero-shot evidence sentence retrieval result reported by LisT5 is recall@5=0.8539, but the T5 model size it uses is 3B. The computing resources of this embodiment cannot run a model of this size, but the candidate set size can be expanded to make up for the gap.

[0131] Based on the results of the few-shot evidence retrieval, the top 5 evidence sentences with the highest scores are selected for each statement as evidence. The models trained with the few-shot training sets of different sizes are used as fact verification models. The obtained fact verification results are shown in Table 6. Compared with the excellent results presented by using perfect evidence, the results of using retrieved evidence are indeed less effective. The main reason is that the recall rate of the retrieved evidence on the first five pieces of evidence is only about 75%. This means that during the retrieval process, a certain proportion of relevant evidence has not been recalled in a timely and accurate manner, thus affecting the final result performance. At the same time, some noise is mixed into the retrieved evidence, which may come from various inaccurate matches or interference from irrelevant information, further reducing the quality and effectiveness of the retrieved evidence. Moreover, it should be noted that there are usually only 1 to 2 perfect evidences annotated in the dataset. Due to the small number, these perfect evidences are often carefully selected and annotated, with high accuracy and reliability. In addition, the input length of perfect evidence is shorter, which makes it more concise and clear, and also means that the information density is larger, which can provide strong support for decision-making more efficiently. In contrast, retrieved evidence has certain deficiencies in these aspects, resulting in its results being less effective than perfect evidence.

[0132] Table 6 Few-sample fact-checking results based on few-sample evidence retrieval

[0133]

[0134] Embodiment 2:

[0135] An electronic device, characterized in that it includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of a two-stage few-sample automatic fact-checking method described in Example 1 are implemented.

[0136] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Moreover, the processor is used to implement the steps of the above-mentioned two-stage few-sample automatic fact-checking method when executing the computer program stored in the memory.

[0137] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0138] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0139] Embodiment 3:

[0140] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the two-stage few-sample automatic fact-checking method described in Example 1 is implemented.

[0141] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned two-stage few-sample automatic fact-verification method can be implemented.

[0142] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0143] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0144] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A two-stage few-shot automatic fact-checking method, characterized in that: The steps include: S1. Select a public fact verification dataset to build a training set and a test set, which contains the claims to be verified and the knowledge base; S2. Use BM25 to build an index for the documents in the knowledge base as a document retrieval model, use the statement of the test set in step 1 as a query, and retrieve the first several documents that are most relevant to the statement in step S1 as candidate documents; S3. Build a sentence selection model, input each sentence in the candidate document obtained in step S2 and the statement in step S1 into the prompt template prompt as the input of the sentence selection model, input it into the sentence selection model, and output the relevance score of the sentence; S4. Based on the relevance scores of the sentences obtained in step S3, select the top several sentences as evidence sentences for claim verification; S5. Build a pre-trained language model and fine-tune the pre-trained language model using a few-sample training set extracted from the training set; S6. The evidence sentence for statement verification obtained in step S4 and the statement to be verified are input into the prompt template prompt as the input of the pre-trained language model, and then input into the fine-tuned pre-trained language model to obtain natural language output, which is then mapped to the classification label, and the prediction score is calculated based on the generation probability of the output sequence to obtain the final prediction result.

2. A two-stage few-sample automatic fact-checking method according to claim 1, characterized in that: The public fact verification dataset described in step S1 includes one or a combination of the FEVER dataset, the SciFact dataset and the VitaminC dataset, wherein the FEVER dataset makes artificially generated claims by modifying the content in Wikipedia and provides standard evidence for supporting and refuting the labeled claims; the SciFact dataset is composed of scientific claims formed by experts rewriting sentences in biomedical literature, and the VitaminC dataset is created by using a revised version of Wikipedia to construct a test set.

3. A two-stage few-sample automatic fact-checking method according to claim 2, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Construct the evidence retrieval model as the probability retrieval model BM25, calculate the frequency of query words in the document and the vocabulary distribution in the document through statistical methods, and generate the query word q i The relevance score score(q,d) in document d is expressed as: Among them, q represents the query word, d represents the document, and IDF(q i ) represents the query word q i The inverted document frequency, f(q i ,d) represents the query word q i The frequency of occurrence in document d, |d| and avgdl represent the document length and the average length of all documents respectively, k1 and b are the first adjustable parameter and the second adjustable parameter respectively; S2.

2. Use Elastic Search to build BM25 indexes for SciFact and FEVER respectively. Use the evidence retrieval model built in step S2.1 to search the test set obtained in step S1 based on the statement, and evaluate the retrieval results. The evaluation criteria used are ndcg@n and recall@n. The top 100 documents most relevant to the statement in step S1 are obtained as candidate documents.

4. A two-stage few-sample automatic fact-checking method according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Build a sentence selection model and use the checkpoints released by the MonoT5 model for sentence selection tasks. MonoT5 is a retrieval model fine-tuned using the MS MARCO paragraph dataset. Use the checkpoints released by the model for sentence selection tasks. The sentence selection model generates a target token of 'True' or 'False' based on whether the document is relevant to the statement. In the ranking task, the output results of the sentence selection model are aggregated, and the probability of assigning the token 'True' is calculated as the relevance score to generate the relevance score of the sentence. The expression is: Among them, relevant = 1 means that the statement is relevant to the document, logit('True') represents the probability of correct assignment of the target token, and logit('False') represents the probability of incorrect assignment of the target token; S3.

2. Split the candidate document obtained in step S2 into sentences, and then combine each sentence in the candidate document obtained in step S2 with the statement in step S1 into the prompt template prompt. The expression of the prompt template prompt is: Query:[Q]Document:[D]Relevant: Among them, Q is the statement in the dataset, and D is the document sentence to be evaluated; S3.

3. Input the combination obtained in step S3.2 into the sentence selection model for reordering and output the relevance score of the sentence.

5. A two-stage few-sample automatic fact-checking method according to claim 4, characterized in that: In step S4, the top five sentences are selected as evidence sentences for claim verification.

6. A two-stage few-sample automatic fact-checking method according to claim 5, characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. Select Flan-T5 as the base model of the pre-trained language model; S5.

2. Fine-tune the base model of the pre-trained language model using the LoRA fine-tuning method. The rank of the matrix in training is specified as 4. LoRA is applied to the (q|k|v|o|w) layers of the SelfAttention, EncDecAttention, and DenseReluDense modules of the original model to obtain the pre-trained language model.

7. A two-stage few-sample automatic fact-checking method according to claim 6, characterized in that: The specific implementation method of step S6 includes the following steps: S6.

1. Set the input of the fact-checking task to be verified by the statement C i and the evidence sentence e obtained in step S4 for the statement verification i Composition, will (C i ,e i ) into the prompt template, the output x i Mapped to label y i , the prompt template example is: Suppose{e i },can we infer that{C i }?; S6.

2. Based on the model input x i , generate a target sequence y i The probability p θ (y i ,x i ): Among them, p θ (t j |x i ,t <j ) is the model in the autoregressive process given the input sequence x i and the previously generated sequence t <j Under the condition of generating t j probability; Due to the correspondence between the output sequence and the classification label, the model is i The prediction score of is defined as the log-length normalized probability of generating the target sequence, which is: All categories are ranked using the prediction scores, and the top-ranked category is taken as the final category prediction.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a two-stage few-sample automatic fact-checking method as described in any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the two-stage few-sample automatic fact-checking method according to any one of claims 1 to 7 is implemented.

Images