Automatic identification of fact-checking factors

Abstract

Methods, systems, and apparatuses, including computer programs encoded on a computer storage medium, facilitate the automatic identification of a set of fact-checking factors from a digital document. Digital documents from multiple sources can be identified. For each digital document, a trained sequence labeling model is used to identify the set of fact-checking factors. Based on the sequence labeling model, a confidence value representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors for that digital document is determined. The set of fact-checking factors is stored in association with the digital document. A request for fact-checking factors for a specific digital document is received from a fact-checking entity. In response, the set of fact-checking factors identified from that specific digital document is provided to the fact-checking entity.

Description

Technical Field

[0001] This specification relates to the automatic identification of a set of fact-checking factors from digital documents. Background Technology

[0002] Misinformation, in the form of fabricated stories and articles that distort facts (often referred to as "fake news"), is prevalent on the World Wide Web. To combat such misinformation, journalists write articles that determine the accuracy and correctness of factual claims by conducting research and linking evidence. These articles are often called fact-checks, and the individuals or organizations that publish such articles are called fact-checkers (also referred to as fact-checking entities in this specification).

[0003] A typical fact-checking article (also referred to as a fact-checking article in this specification) may include at least the following information (also referred to as fact-checking factors in this specification): the claim to be checked, the claimant making the claim, and a judgment describing the veracity of the claim, as determined by the fact-checker. For example, a fact-checking article may (1) identify the claim that “Barack Obama was born in Kenya,” (2) identify the claimant, XYZ News Organization, and (3) specify the veracity of the claim, which in this case is false. Fact-checking article documents may also provide additional information, such as, for example, the context in which the claim is made, evidence supporting or refuting the claim, etc. Summary of the Invention

[0004] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method that may include the following operations: identifying digital documents from multiple publishers; for each digital document: identifying a set of fact-checking factors from the digital document and using a trained sequence labeling model, the set of fact-checking factors specifying the claim to be fact-checked, the claimant making the claim, and the veracity of the claim; and determining a confidence value based on the trained sequence labeling model, the confidence value representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors for the digital document; storing the set of fact-checking factors in association with the digital document in a storage device; receiving a request from a fact-checking entity for fact-checking factors of a specific digital document; and, in response to receiving the request for fact-checking factors of a specific digital document, providing the fact-checking entity with the set of fact-checking factors identified from the specific digital document. Other embodiments of this aspect include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the method. The computer program (e.g., instructions) may be encoded on a computer storage device.

[0005] These and other embodiments may each optionally include one or more of the following features.

[0006] In some implementations, the method may further include the following operations: for each digital document, determining whether the confidence value meets a confidence threshold; and wherein the set of stored fact-checking factors associated with the digital document includes a set of stored fact-checking factors associated with the digital document only if the confidence value meets the confidence threshold.

[0007] In some implementations, the method may also include the following operations: for each digital document: using a search engine and based on a set of fact-checking factors, identifying a set of resources that cite the set of fact-checking factors in addition to the digital document; and adjusting the confidence value based on whether the number of identified resources meets a specific threshold.

[0008] In some implementations, the method may further include the following operations: training a sequence labeling model, the training of which includes: identifying a set of known fact-checking articles and a corresponding set of known fact-checking factors; and for each known fact-checking article: generating a word sequence of equal length using words present in the known fact-checking article; and for each word sequence, generating a label sequence for each word sequence by matching words of known fact-checking factors in the known fact-checking article with words in the word sequence, wherein each label represents one of the fact-checking factors or represents a word that does not belong to any fact-checking factor; and training the sequence labeling model using the word sequence in the known fact-checking article and the generated corresponding label sequence to generate a label sequence for the input word sequence.

[0009] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, the innovations described in this specification automatically identify fact-checking factors from fact-checking articles, regardless of the entity that drafted the fact-checking article, and even when the article does not contain specific markup language with annotations specifically designed to identify fact-checking factors. Some conventional techniques identify fact-checking factors based on markup language present within the fact-checking article (e.g., ClaimReview, which is the schema.org open standard annotation system or "tags" used by search engines and other major digital platforms to find and highlight fact-checking articles) with annotations for identifying fact-checking factors. However, most fact-checking articles, typically prepared by different entities / publishers, generally do not contain such markup, and therefore, these conventional techniques cannot identify fact-checking factors from such fact-checking articles. Other conventional techniques may utilize a set of rules to identify fact-checking factors based on certain language patterns in the fact-checking article. For example, an assertion may typically follow certain verbs, such as "(someone) asserts..." or "(someone) says...". However, such pattern-based rules fail to account for the various possible differences in writing style and format among different publishers, and therefore cannot consistently and accurately identify fact-checking factors from fact-checking articles. Unlike these conventional techniques, the techniques described in this specification (as described throughout) can more accurately identify fact-checking factors from fact-checking articles without any specific markers in the article, or even when fact-checking factors are included in these articles in different formats and styles (i.e., without following any specific pattern). In this way, aspects of this disclosure solve the technical problems associated with data identification, particularly when the format of the data to be identified is unknown.

[0010] The innovations described in this specification can also enhance network security by first determining whether a particular digital document is actually a fact-checking article before further processing it within a potentially sensitive environment of the fact-checking entity (or any other entity). In some implementations, the innovations described in this specification can be implemented, for example, as a service separate from the fact-checking entity (or another entity). In such implementations, the techniques described for identifying fact-checking factors can determine whether a particular digital document is a fact-checking article (e.g., by determining whether any fact-checking factors can be identified from the content of the digital document). By making this determination, the techniques described in this specification can prevent any malicious files or resources (e.g., spam, malware, viruses, etc.) from being uploaded to the fact-checking entity's system and spreading throughout the fact-checking entity's system. In this way, aspects of this disclosure also address technical problems related to computer security.

[0011] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a block diagram of an example environment for identifying fact-checking factors from a collection of digital documents.

[0013] Figure 2 This is a flowchart of an example process for identifying fact-checking factors from a collection of digital documents.

[0014] Figure 3 This is a block diagram of an example computer system.

[0015] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0016] This specification generally relates to fact-checking factor identifyers, which identify fact-checking factors from digital documents regardless of the document's format, style (including any markup that may be present in the document), or publisher / author.

[0017] Figure 1 This is a block diagram of an example environment 100 for identifying fact-checking factors from a collection of digital documents.

[0018] Example environment 100 includes network 102, such as a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof. Network 102 connects fact-checking entity 106, publisher 104, search engine 120, and fact-checking factor recognizer 108. Example environment 100 may include many different fact-checking entities 106, publishers 104, and search engines 120.

[0019] As used in this specification, publisher 104 is an entity that publishes or makes available digital documents via network 102. In some implementations, publisher 104 may create digital documents in addition to publishing (or making available) digital documents authored by other individuals or entities. Examples of digital documents published by publisher 104 may include: web pages, word processing documents, portable document format (PDF) documents, images, videos, and search results pages. As used in this specification, fact-checking entity 106 is an entity that verifies or evaluates the accuracy of statements made in fact-checking articles or other digital documents. Fact-checking entity 106 may be separate from or part of publisher 104.

[0020] In some implementations, the fact-checking factor identifier 108 can be implemented as a service within the environment of fact-checking entity 106, publisher 104, or search engine 120. In some implementations, the fact-checking factor identifier 108 can be implemented as a web service separate from any fact-checking entity 106, publisher 104, or search engine 120, but can still be accessed by any fact-checking entity 106, publisher 104, or search engine 120 via network 102.

[0021] Fact-checking factor identifyer 108 identifies fact-checking factors included in digital documents provided / published by one or more publishers 104. Fact-checking factor identifyer 108 includes sequence generator 118, sequence labeling model 114, combiner model 116, rule-based sequence labeler 110, and known fact-checking storage device 112. (See below for further details.) Figure 2 Additional details regarding the structure and operation of the fact-checking factor identifyer 108 are described, including the structure and operation of each of these sub-components.

[0022] Figure 2 This is a flowchart of an example process 200 for identifying fact-checking factors from a collection of digital documents. The operation of process 200 is described below for illustrative purposes only. The operation of process 200 can be performed by any suitable device or system, such as any suitable data processing apparatus. The operation of process 200 can also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing apparatuses to perform the operation of process 200.

[0023] Fact checker 108 identifies digital documents from multiple sources (at 202). In some implementations, fact checker 108 may identify digital documents from multiple sources (such as publisher 104) in response to a request sent by fact checker 108 to these sources. In some implementations, fact checker 108 may receive digital documents from multiple sources (such as publisher 104), for example, periodically and without first sending a request.

[0024] For each identified digital document, operations 204, 206, and 208 can be performed, as described below.

[0025] Fact-checking factor recognizer 108 identifies from the digital document a set of fact-checking factors that at least specify the claim to be fact-checked, the claimant making the claim, and the veracity of the claim (at 204). In some implementations, fact-checking factor recognizer 108 first sends the digital document to sequence generator 118, which generates equal-length word sequences for the words in the digital document. Sequence generator 118 feeds the generated word sequences to a trained sequence labeling model 114 (additional details regarding the training of the sequence labeling model are provided below).

[0026] Sequence labeling model 114 takes a sequence of words as input and outputs a sequence of labels of equal length—that is, the label for each word in the input word sequence. In some implementations, sequence labeling model 114 can be implemented as a natural language processing model (e.g., BERT), a supervised machine learning model, or another statistical learning model. In some implementations, many different sequence labeling models other than BERT can be used.

[0027] The labels output by a sequence labeling model can include both positive and negative labels. Each positive label represents one of the fact-checking factors, while each negative label represents a word that does not belong to any fact-checking factor. For example, an implementation including three fact-checking factors (e.g., claim, claimant, and verdict) would have three corresponding positive labels: claim, claimant, and verdict. Below is an example of a sequence of words that can be input into a sequence labeling model, which in turn can output a corresponding sequence of labels (where label C represents claim, label P represents claimant, label V represents the veracity of claim, and label NF represents that the corresponding word in the sequence is not a fact-checking factor):

[0028] John X Doe Falsely said that the Earth is flat P P P V NF NF C C C C

[0029] The label sequence and corresponding word sequence generated by sequence labeling model 114 are then input into combiner model 116, which uses these input word sequences and corresponding label sequences to generate a set of fact-checking factors for the digital document. In some implementations, for each label corresponding to a specific fact-checking factor, combiner model 116 obtains the words in the word sequence corresponding to that label to generate that fact-checking factor.

[0030] In some implementations, combiner model 116 is a rule-based engine that can combine words that are close to each other. For example, consider a scenario where a portion of an assertion (i.e., a fact-checking factor) is found in two regions of a sentence consisting of nine words: a first region consisting of the first three words (words at positions 1 to 3), and a second region consisting of the last three words (words at positions 6 to 9). In this example, combiner model 116 can determine that the two regions are close to each other (e.g., based on a proximity threshold that considers regions separated by, for example, ten or fewer words as close to each other), and therefore, combine all the words spanning the two regions (i.e., words 1 to 9) to obtain the assertion. Alternatively, combiner model 116 can combine only the words that make up the two regions, in which case the assertion would consist of the words at positions 1 to 3 followed by the words at positions 6 to 9. Other rules can be used alone or in conjunction with the rules described above to combine the identified portions / regions of the fact-checking factor. For example, combiner model 116 may have rules that remove redundant words or phrases that may be found in different identified regions and then combine only the unique words in the identified regions.

[0031] In some implementations, the combiner model 116 can be implemented as a learning model (e.g., a supervised machine learning model, a natural language processing model, or another suitable statistical model) trained using known labeled sequences and corresponding fact-checking factors, rather than a rule-based engine. Such a model can be trained to take into account various features in the labeled sequences and corresponding digital documents (e.g., distances between identified regions, redundancy across different regions, semantics of combining different regions, the location of identified regions within the digital document, etc.). Therefore, when such a trained model receives a labeled sequence of a digital document, it can determine the sequence of combinations that constitute the fact-checking factors.

[0032] Fact-checking factor recognizer 108 determines a confidence value (at 206) representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors for the digital document. In some implementations, for each label sequence output by sequence labeling model 114, model 114 also outputs an initial label confidence value, representing the probability that the identified label (corresponding to a specific fact-checking factor) actually identifies a fact-checking factor. Subsequently, when combiner model 116 generates fact-checking factors using the label sequence and word sequence, combiner model 116 uses the initial label confidence value to determine the confidence value (also called the overall confidence value) representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors for the digital document.

[0033] In some implementations, the overall confidence value determination is performed as a log-likelihood of generating a sequence of labels on words. In such implementations, the overall confidence value determination may be biased towards what the model has observed during training. In some implementations, this bias can be avoided or reduced by using a separate model (e.g., a machine learning model or another suitable statistical model) to determine the overall confidence value by considering one or more of the following example factors: the identified fact-checking factors, the initial confidence value, the length of the region within the digital document from which the fact-checking factors are identified, the number of such regions, and the number of times the identified claims are found in other resources (as further described with reference to operation 208). Such a model can be trained using known fact-checking factors and data about the identified factors that can be identified from the corresponding fact-checking article (from which the known fact-checking factors are identified). Once trained, such a model can accept data about one or more of the above factors and one or more of the above example factors (or other example factors) as input and use them to estimate the overall confidence value.

[0034] Fact checker 108 identifies a set of resources (at 208) that reference a set of fact check factors, in addition to digital documents. In some implementations, the fact checker issues a search query to one or more search engines 120, including the identified set of fact check factors in the search query. In response to the search query, one or more search engines 120 may return a set of search results. For a threshold number of search results that do not include a specific set of digital documents (e.g., top 10 search results, top 50 search results, etc.), fact checker 108 may identify a set of resources (e.g., web pages) corresponding to the search results. A resource is any data that can be provided by publisher 104 via network 102 and associated with a resource address. Resources include HTML pages, word processing documents and portable document format (PDF) documents, images, videos, and feed sources, to name a few. Resources may include content such as words, phrases, images, etc., and may include embedded information (such as metadata and hyperlinks) and / or embedded instructions (such as scripts).

[0035] For each resource, the fact-checking factor recognizer 108 determines whether the words of the fact-checking factor match all or part of the words of the resource. For resources where the amount of matched words meets (e.g., reaches or exceeds) a specific threshold (e.g., 80%, 90%, etc.), the fact-checking factor recognizer 108 identifies such resources as a set of resources that reference the set of fact-checking factors.

[0036] Fact-checking factor identifier 108 adjusts the confidence value (at 210) based on the number of identified resources (such as those identified in operation 208) that meet a specific threshold. In some implementations, fact-checking factor identifier 108 may increase the confidence value by a certain amount if the number of identified resources (such as those described in operation 208) meets (e.g., reaches or exceeds) a specific threshold (e.g., five, ten, twenty, etc.). In some implementations, fact-checking factor identifier 108 may increase the confidence value by a fixed amount, regardless of the number of identified resources exceeding the specific threshold. Alternatively, fact-checking factor identifier 108 may increase the confidence value by an amount proportional to the number of identified resources exceeding the specific threshold (e.g., increasing the confidence by 5% for every five additional resources exceeding the specific threshold). In some implementations, if the number of identified resources (as described in operation 208) does not meet (e.g., is less than) a specific threshold (e.g., five, ten, twenty, etc.), the fact-checking factor identifyr 108 may reduce the confidence value by a certain amount. In some implementations, the fact-checking factor identifyr 108 may reduce the confidence value by a fixed amount or by an amount proportional to the number of identified resources less than the specific threshold (e.g., reducing the confidence value by 5% for every five additional resources less than the specific threshold).

[0037] Fact check factor recognizer 108 stores fact check factors in a storage device (at 212). In some implementations, fact check factor recognizer 108 stores all fact check factors generated by the combiner model, regardless of their corresponding confidence values ​​(as described with reference to operations 206-210). Alternatively, fact check factor recognizer 108 stores only those fact check factors whose confidence values ​​satisfy (e.g., reach or exceed) a specific threshold (e.g., 60%, 70%, etc.). In some implementations, fact check factor recognizer 108 stores, for example, a set of identifiers of digital documents and their corresponding fact check factors in a table. In some implementations, the table may also include confidence values ​​output by the combiner model, and in some cases, confidence values ​​adjusted by the fact check factor recognizer (as described with reference to operations 206 and 208).

[0038] Fact-checking factor identifyr 108 receives a request for fact-checking factors for a specific digital document (at 214). In some implementations, fact-checking entity 106 (or another entity) may submit a request to fact-checking factor identifyr 108 for fact-checking factors for a specific digital document. This request may include the actual digital document or a webpage link to the digital document. The specific digital document may be one of the digital documents identified at operation 202, or it may be another digital document not included in the documents identified at operation 202.

[0039] Fact check factor recognizer 108 provides a set of fact check factors (at 216) identified for a specific digital document. If the specific digital document is one of the documents identified at operation 202, the fact check factor recognizer can use the digital document's identifier (e.g., filename, file hash, etc.) to look up the corresponding fact check factors stored in the storage device for that specific digital document. The fact check factor recognizer 108 provides the set of fact check factors retrieved from the storage device to the entity requesting the fact check factors, such as fact check entity 106. If the specific digital document is not among the documents identified at operation 202, the fact check factor recognizer 108 can generate fact check factors for that specific document as needed using operations 204-206 (and in some implementations, also operations 208 and 210).

[0040] In some implementations, a sequence labeling model 114 is trained using a set of known fact-checking articles and a corresponding set of known fact-checking factors stored in known fact-checking storage device 112. As part of this training, for example, a sequence generator 118 is used to segment the words of each known fact-checking article into equal-length word sequences. For each such word sequence, a rule-based sequence labeler 110 generates a corresponding label sequence by matching the words of the fact-checking factors in the fact-checking articles with the words in the word sequence. Each label can be a positive label representing one of the fact-checking factors or a negative label representing a word that does not belong to any fact-checking factor. The sequence labeling model 114 is trained to generate a label sequence for the input word sequence using each label sequence generated by the rule-based sequence labeler and the corresponding word sequence in the known fact-checking articles.

[0041] At least two types of rule-based sequence labelers 110 may exist: (1) concise labelers and (2) fluent labelers. In the implementation of a concise labeler, if a word in a word sequence matches a word of a fact-checking factor, the labeler assigns the corresponding label of that fact-checking factor to the position in the label sequence corresponding to the word in the word sequence. This is illustrated using the example table below, which identifies (1) the known set of fact-checking factors, including claims, assertors, and judgments (as shown in the first row), (2) the word sequence (as shown in the second row), and (3) the corresponding label sequence that will be generated by the concise labeler (as shown in the third row). As used in the table below, label C represents a claim, label P represents an assertor, label V represents the veracity of a claim, and label NF represents that the corresponding word in the sequence is not a fact-checking factor.

[0042]

[0043] In the implementation using the fluent annotator, if a word in the word sequence matches a word of a fact-checking factor, the annotator (1) assigns the label of that fact-checking factor to the position in the label sequence corresponding to the position of that word in the word sequence, and (2) also assigns the label to all other words in the word range that end in book end with the same label (or fact-checking factor). This is illustrated using the example table below, which identifies (1) the known set of fact-checking factors, including claims and assertors and judgments (as shown in the first row), (2) the word sequence (as shown in the second row), and (3) the corresponding label sequence that will be generated by the fluent annotator (as shown in the third row). As used in the table below, label C represents a claim, label P represents an assertor, label V represents the veracity of a claim, and label NF represents that the corresponding word in the sequence is not a fact-checking factor.

[0044]

[0045] As shown in the table above, the Fluent Annotator identifies the claimant as John X Doe, even though John Doe is known to be the claimant. Similarly, the Fluent Annotator identifies the claim as "Africa is actually a country," even though the claim is known to be "Africa is a country." In this respect, the Fluent Annotator focuses more on the readability of the generated tags, even if it includes unnecessary details in the sequence. Conversely, the Concise Annotator focuses more on conciseness, therefore it only selects the necessary words for the factors, but the results may be less readable if the matched factors omit a few words.

[0046] In this way, a sequence labeling model 114 is trained using a set of known fact-checking articles and a corresponding set of known fact-checking factors stored in a known fact-checking storage device 112 to generate a sequence of labels for the input word sequence.

[0047] Figure 3 This is a block diagram of an example computer system 300 that can be used to perform the operations described above. System 300 includes a processor 310, memory 320, storage device 330, and input / output device 340. Each of components 310, 320, 330, and 340 may be interconnected, for example, using a system bus 350. Processor 310 is capable of processing instructions for execution within system 300. In some implementations, processor 310 is a single-threaded processor. In another implementation, processor 310 is a multi-threaded processor. Processor 310 is capable of processing instructions stored in memory 320 or on storage device 330.

[0048] Memory 320 stores information within system 300. In one implementation, memory 320 is a computer-readable medium. In some implementations, memory 320 is a volatile memory cell. In another implementation, memory 320 is a non-volatile memory cell.

[0049] Storage device 330 provides high-capacity storage for system 300. In some implementations, storage device 330 is a computer-readable medium. In various implementations, storage device 330 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other high-capacity storage device.

[0050] Input / output device 340 provides input / output operations for system 300. In some implementations, input / output device 340 may include one or more of the following: a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another implementation, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, printer, and display device 360. However, other implementations, such as mobile computing devices, mobile communication devices, set-top box TV client devices, etc., may also be used.

[0051] Although already Figure 3 An example processing system is described herein, but the subject matter and functional operations described herein may be implemented in other types of digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed herein and their structural equivalents), or in a combination of one or more of them.

[0052] Embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium (or multiple media), for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these, or included therein. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices) or included therein.

[0053] The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0054] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination of the foregoing. Apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. Apparatus and execution environments can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0055] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. A computer program may, but is not required to, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing portions of one or more modules, subroutines, or code). Computer programs can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected through a communication network.

[0056] The processes and logic flows described in this specification can be executed by one or more programmable processors executing one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0057] Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled thereto to receive data from or transfer data to, or both. However, a computer does not necessarily have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM discs. The processor and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0058] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to the webpage in response to a request received from a webpage on the user's client device.

[0059] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which its users can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0060] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from it). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0061] Although this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from the combination, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0062] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments; rather, it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0063] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for the automatic identification of fact-checking factors, comprising: Identify digital documents from multiple sources; For each of the digital documents: A trained sequence labeling model is implemented to identify a set of fact-checking factors from the digital document, the set of fact-checking factors specifying the claim to be fact-checked, the claimant making the claim, and the veracity of the claim, wherein the trained sequence labeling model is configured to: Obtain the word sequence from the digital document as input, and Output a label for each word in the word sequence from the digital document, wherein each label corresponds to one of the fact-checking factors or indicates a word that does not belong to any fact-checking factor; The labels for each word in the word sequence from the digital document are processed to generate the set of fact-checking factors; A confidence value is determined based on the trained sequence labeling model, the confidence value representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors of the digital document; and The set of fact-checking factors is stored in a storage device in association with the digital document; Receive requests from the fact-checking entity for fact-checking factors of a specific digital document within the digital document; and In response to receiving a request for fact-checking factors for the specific digital document, the fact-checking entity is provided with a set of fact-checking factors identified from the specific digital document.

2. The method of claim 1, further comprising: For each digital document, determine whether the confidence value meets the confidence threshold; as well as The set of fact-checking factors stored in association with the digital document includes a set of fact-checking factors that are stored in association with the digital document only when the confidence value meets the confidence threshold.

3. The method of claim 2, further comprising: For each digital document: Using a search engine, identify a set of resources, in addition to the digital documents, that reference the set of fact-checking factors, based on the set of fact-checking factors. as well as The confidence value is adjusted based on whether the number of identified resources meets a specific threshold.

4. The method of claim 1, further comprising: Training the sequence labeling model, the training includes: Obtain the set of known fact-checking articles and the corresponding set of known fact-checking factors; and For each known fact-checking article: Generate word sequences of equal length using words present in the known fact-checking article; and For each word sequence, a tag sequence is generated by matching the words of the known fact-checking factors in the known fact-checking article with the words in the word sequence, wherein each tag represents one of the fact-checking factors or represents a word that does not belong to any fact-checking factor; and The sequence labeling model is trained using the known facts to examine the word sequences in the article and the generated corresponding label sequences to generate a label sequence for the input word sequence.

5. The method of claim 1, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Based on the distance between a first region and a second region of the word sequence, one or more words from the first region associated with a label corresponding to one of the fact-checking factors are combined with one or more words from the second region also associated with the label corresponding to one of the fact-checking factors.

6. The method of claim 1, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Determine whether one or more words from a first region of the word sequence associated with a label corresponding to one of the fact-checking factors are redundant with one or more words from a second region of the word sequence also associated with a label corresponding to one of the fact-checking factors; and In response to determining that one or more words from the first region are redundant with one or more words from the second region, one or more words from the first region are removed.

7. A system for automatically identifying fact-checking factors, comprising: One or more processing devices; and One or more non-transitory machine-readable storage devices storing instructions that can be executed by the one or more processing devices to cause operations to be performed, the operations including: Identify digital documents from multiple publishers; For each of the digital documents: A trained sequence labeling model is implemented to identify a set of fact-checking factors from the digital document, the set of fact-checking factors specifying the claim to be fact-checked, the claimant making the claim, and the veracity of the claim, wherein the trained sequence labeling model is configured to: Obtain the word sequence from the digital document as input, and Output a label for each word in the word sequence from the digital document, wherein each label corresponds to one of the fact-checking factors or indicates a word that does not belong to any fact-checking factor; The labels for each word in the word sequence from the digital document are processed to generate the set of fact-checking factors; A confidence value is determined based on the trained sequence labeling model, the confidence value representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors of the digital document; and The set of fact-checking factors is stored in a storage device in association with the digital document; Receive requests from the fact-checking entity for fact-checking factors of a specific digital document within the digital document; and In response to receiving a request for fact-checking factors for the specific digital document, the fact-checking entity is provided with a set of fact-checking factors identified from the specific digital document.

8. The system of claim 7, wherein, The operation also includes: For each digital document, determine whether the confidence value meets the confidence threshold; and The set of fact-checking factors stored in association with the digital document includes a set of fact-checking factors that are stored in association with the digital document only when the confidence value meets the confidence threshold.

9. The system of claim 8, wherein, The operation also includes: For each digital document: Using a search engine, identify a set of resources, other than the digital documents, that cite the set of fact-checking factors, based on the set of fact-checking factors; and The confidence value is adjusted based on whether the number of identified resources meets a specific threshold.

10. The system of claim 7, wherein, The operation also includes: Training the sequence labeling model, the training includes: Obtain the set of known fact-checking articles and the corresponding set of known fact-checking factors; and For each known fact-checking article: Generate word sequences of equal length using words present in the known fact-checking article; and For each word sequence, a tag sequence is generated by matching the words of the known fact-checking factors in the known fact-checking article with the words in the word sequence, wherein each tag represents one of the fact-checking factors or represents a word that does not belong to any fact-checking factor; and The sequence labeling model is trained using the known facts to examine the word sequences in the article and the generated corresponding label sequences to generate a label sequence for the input word sequence.

11. The system of claim 7, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Based on the distance between a first region and a second region of the word sequence, one or more words from the first region associated with a label corresponding to one of the fact-checking factors are combined with one or more words from the second region also associated with the label corresponding to one of the fact-checking factors.

12. The system of claim 7, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Determine whether one or more words from a first region of the word sequence associated with a label corresponding to one of the fact-checking factors are redundant with one or more words from a second region of the word sequence also associated with a label corresponding to one of the fact-checking factors; and In response to determining that one or more words from the first region are redundant with one or more words from the second region, one or more words from the first region are removed.

13. A non-transitory computer-readable medium storing instructions executable by one or more processing devices to cause the execution of operations, said operations including: Identify digital documents from multiple publishers; For each of the digital documents: A trained sequence labeling model is implemented to identify a set of fact-checking factors from the digital document, the set of fact-checking factors specifying the claim to be fact-checked, the claimant making the claim, and the veracity of the claim, wherein the trained sequence labeling model is configured to: Obtain the word sequence from the digital document as input, and Output a label for each word in the word sequence from the digital document, wherein each label corresponds to one of the fact-checking factors or indicates a word that does not belong to any fact-checking factor; The labels for each word in the word sequence from the digital document are processed to generate the set of fact-checking factors; A confidence value is determined based on the trained sequence labeling model, the confidence value representing the probability that the set of fact-checking factors identified from the digital document is the actual set of fact-checking factors of the digital document; and The set of fact-checking factors is stored in a storage device in association with the digital document; Receive requests from the fact-checking entity for fact-checking factors of a specific digital document within the digital document; and In response to receiving a request for fact-checking factors for the specific digital document, the fact-checking entity is provided with a set of fact-checking factors identified from the specific digital document.

14. The non-transitory computer-readable medium of claim 13, wherein the operation further comprises: For each digital document, determine whether the confidence value meets the confidence threshold; as well as The set of fact-checking factors stored in association with the digital document includes a set of fact-checking factors that are stored in association with the digital document only when the confidence value meets the confidence threshold.

15. The non-transitory computer-readable medium of claim 14, wherein the operation further comprises: For each digital document: Using a search engine, identify a set of resources, in addition to the digital documents, that reference the set of fact-checking factors, based on the set of fact-checking factors. as well as The confidence value is adjusted based on whether the number of identified resources meets a specific threshold.

16. The non-transitory computer-readable medium of claim 13, wherein the operation further comprises: Training the sequence labeling model, the training includes: Obtain the set of known fact-checking articles and the corresponding set of known fact-checking factors; and For each known fact-checking article: Generate word sequences of equal length using words present in the known fact-checking article; and For each word sequence, a tag sequence is generated by matching the words of the known fact-checking factors in the known fact-checking article with the words in the word sequence, wherein each tag represents one of the fact-checking factors or represents a word that does not belong to any fact-checking factor; and The sequence labeling model is trained using the known facts to examine the word sequences in the article and the generated corresponding label sequences to generate a label sequence for the input word sequence.

17. The non-transitory computer-readable medium of claim 13, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Based on the distance between a first region and a second region of the word sequence, one or more words from the first region associated with a label corresponding to one of the fact-checking factors are combined with one or more words from the second region also associated with the label corresponding to one of the fact-checking factors.

18. The non-transitory computer-readable medium of claim 13, wherein, Processing the labels of each word in the word sequence from the digital document to generate the set of fact-checking factors includes: Determine whether one or more words from a first region of the word sequence associated with a label corresponding to one of the fact-checking factors are redundant with one or more words from a second region of the word sequence also associated with a label corresponding to one of the fact-checking factors; and In response to determining that one or more words from the first region are redundant with one or more words from the second region, one or more words from the first region are removed.

Images