Document verification assistance device, document verification assistance method, and program

Abstract

This document verification assistance device comprises: an evaluation unit configured to evaluate similarity between a certain document and a document published by a prescribed organization; and an output unit configured to output the similarity evaluation result. Thereby, the document verification assistance device provides assistance in verifying the authenticity of said certain document.

Description

Document verification support device, document verification support method, and program

[0001] The present invention relates to a document verification support device, a document verification support method, and a program.

[0002] With the advancement of natural language processing (NLP), it is expected that machine-generated information such as documents and images will become more prevalent in society. A notable example is ChatGPT, which has already begun to be applied in companies less than a year after its release.

[0003] Easily accessible documents and other information have the potential to be central to DX (Digital Transformation) and are highly convenient, but documents and other information that are easily and automatically generated are likely to contain errors. Such documents are particularly susceptible to errors caused by the following cases: - Numerical errors during generation - Errors contained in the training data - Outdated training data Furthermore, the accuracy of information varies depending on the user of the information, depending on various conditions such as country, region, position, and specialty.

[0004] "To ChatGPT, or not to ChatGPT: That is the question!" [online], Internet, <URL: https: / / arxiv.org / abs / 2304.01487> Kakimoto, Aramaki, Miyabe, "Text feature analysis for determining information reliability", 78th National Convention of Information Processing Society of Japan, pp. 1-577, 2016.

[0005] From the above, it is difficult for users without background information or specialized knowledge to determine the accuracy of automatically generated information. Document generation, such as GPT, produces documents that appear plausible at first glance, but it is difficult to reduce errors.

[0006] Research is underway into technology to detect documents automatically generated by AI (for example, Non-Patent Document 1), but as automatic generation technology advances, it is becoming more difficult every year to determine whether a document was created by a human or an AI.

[0007] This differs from technology for identifying fake news written by humans (e.g., Non-Patent Document 2). For example, humans may have variations in spelling when writing false documents, but such variations are unlikely to appear in automatically generated documents by AI. Natural document generation by AI will undoubtedly continue to develop in the future, and there are limits to determining whether a document was created by AI or a human.

[0008] Therefore, rather than judging whether the document we receive is automatically generated by AI, we will likely need to accept AI-generated documents as a piece of information and examine the reliability of the information itself from a different perspective.

[0009] Large-scale document generation engines are widely available to general users, but it is difficult to adapt the learning content of the document generation engine to the user. Fine-tuning a document generation engine requires expertise in NLP, and generally, only the organization that owns the document generation engine can perform the fine-tuning. Therefore, it is generally difficult for users to directly adjust the document generation engine. Because the learning content of the document generation engine is unknown, it is not possible to predict whether an error will be present in the output document, and even if an error does occur, it is difficult to determine that an error has been included.

[0010] In addition, because documents generated by document generation engines are natural, it is difficult to quantitatively determine the number of errors in the content. The claims of major institutions change over time, making it difficult to accurately capture these changes.

[0011] Furthermore, it is difficult for the user to understand which country or region the document learned by the document generation engine is from. The present invention has been made in consideration of the above points, and aims to support verification of the reliability of a certain document.

[0012] In order to solve the above problem, the document verification support device includes an evaluation unit configured to evaluate the similarity between a certain document and a document published by a predetermined organization, and an output unit configured to output the evaluation result of the similarity.

[0013] It can help verify the authenticity of a document.

[0014] FIG. 1 is a diagram illustrating an example of a hardware configuration of a document verification support device 10 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a functional configuration of the document verification support device 10 according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating an example of a processing procedure for selecting a reference organization. FIG. 4 is a diagram illustrating an example of a result of vectorization of important words in a target document. FIG. 5 is a diagram illustrating an example of a result of vectorization of important words in documents of each organization. FIG. 6 is a flowchart illustrating an example of a processing procedure for verification support processing of a target document. FIG. 7 is a diagram illustrating verification support processing of a target document. FIG. 8 is a diagram illustrating an example of output information related to one evaluation unit of a target document.

[0015] In this embodiment, a document verification support device 10 is disclosed that evaluates the similarity between the content (e.g., assertions) of a document from a trusted institution and that of the target document in order to verify the authenticity of the target document. A user uses the evaluation results of the document verification support device 10 as reference material to determine the authenticity (reliability) of the target document. Note that the target document refers to a document automatically generated by a computer using a machine learning model based on natural language processing (NLP) or the like. In this embodiment, a document refers to a set of one or more sentences.

[0016] An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a diagram showing an example of the hardware configuration of a document verification support device 10 according to an embodiment of the present invention. The document verification support device 10 shown in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0017] The program that realizes the processing in the document verification support device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0018] When an instruction to start the program is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the document verification support device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0019] 2 is a diagram showing an example of the functional configuration of the document verification support device 10 according to the embodiment of the present invention. In FIG. 2, the document verification support device 10 includes a selection unit 11, an evaluation unit 12, and an output unit 13. Each of these units is realized by a process executed by a processor 104 of one or more programs installed in the document verification support device 10.

[0020] The selection unit 11 selects an organization (hereinafter referred to as a "reference organization") to be used as a reference for verifying the reliability of the target document from among multiple organizations based on the content of the target document, and outputs the selection result. The selection unit 11 is particularly effective when the user does not have expertise and does not know which organization or document to refer to.

[0021] The evaluation unit 12 evaluates the similarity between the target document and documents published by a predetermined organization (an organization designated by the user or a reference organization selected by the selection unit 11).

[0022] The output unit 13 outputs information based on the evaluation result by the evaluation unit 12 .

[0023] The document verification support device 10 also uses a document DB 14. The document DB 14 can be realized using, for example, the auxiliary storage device 102 or a storage device connectable to the document verification support device 10 via a network. The document DB 14 stores data (hereinafter referred to as "document data") indicating one or more documents (for example, in the case of technical documents, highly cited papers) made public (issued) by each of a plurality of highly reliable organizations (governments, government ministries, national agencies, news organizations, companies, associations, organizations, etc.).

[0024] The following describes the processing procedure executed by the document verification support apparatus 10. Fig. 3 is a flowchart for explaining an example of the processing procedure for selecting a reference organization.

[0025] In step S101, the selection unit 11 inputs a target document. The target document may be stored in the auxiliary storage device 102 or may be received via a network.

[0026] Next, the selection unit 11 extracts important words (e.g., a group of words that appear frequently) from the target document (S102). To evaluate whether each word constituting the target document is an important word, a known method such as TF-IDF may be used, and the method is not limited to a specific one in this embodiment.

[0027] Next, the selection unit 11 vectorizes (converts into vectors) each of the important words extracted in step S102 (S103).

[0028] FIG. 4 is a diagram showing an example of the result of vectorizing important words in a target document. As shown in FIG. 4, as a result of executing step S103, a vector is obtained for each important word. Note that word vectorization refers to extracting and normalizing the features of a word and converting the word into vector information. By vectorizing words, it becomes possible to handle the words on a numerical basis. Word vectorization may be performed using natural language processing (e.g., Word2Vec).

[0029] Next, the selection unit 11 extracts important words from each document data stored for each organization in the document DB 14 (S104).

[0030] Next, the selection unit 11 vectorizes (converts into vectors) each of the important words extracted in step S104 (S105). The vectorization method may be the same as that in step S103.

[0031] 5 is a diagram showing an example of the result of vectorization of important words in documents of each organization. As shown in FIG. 5, as a result of executing step S105, vectors for each important word are obtained for each organization.

[0032] Next, the selection unit 11 evaluates the similarity between each key word in each document of each organization and each key word in the target document (S106). The evaluation of the similarity between two key words may be performed, for example, by calculating a similarity (e.g., cosine similarity) derived from the inner product of the vectors of the two key words. If the number of key words in each document of an organization is N and the number of key words in the target document is M, an N x M similarity is calculated for the organization.

[0033] Next, based on the similarity evaluation results in step S106, the selection unit 11 selects some organizations as reference organizations that have documents to be referenced in verifying the target document (S107). For example, the top K organizations may be selected as reference organizations based on the average similarity of each document for each organization. The average similarity of each document for each organization indicates the evaluation result of the similarity of the target document to information published by each reliable organization. The top K organizations based on the average similarity correspond to organizations that publish information that is relatively similar to the content of the target document.

[0034] FIG. 6 is a flowchart illustrating an example of the processing procedure for the verification support process for the target document. FIG. 7 is a diagram illustrating the verification support process for the target document. In step S201, the evaluation unit 12 vectorizes (e.g., Doc2Vec) the target document for each unit for evaluating similarity (hereinafter referred to as "evaluation unit"). For example, each page or section may be used as the evaluation unit. However, if the number of characters or sentences in the target document is less than a threshold, the evaluation unit may be the entire target document.

[0035] Next, the evaluation unit 12 vectorizes each document of the reference organization (hereinafter referred to as "reference document") for each evaluation unit (S202). Here, the reference organization is the organization selected by the selection unit 11 when the processing procedure of FIG. 3 is executed. Alternatively, a specific organization may be designated as the reference organization by the user. When the user selects a reference organization, the user may select the reference organization based on an aspect that the user values ​​in the content of the target document. For example, if the user values ​​the newsworthiness of the target document, the user may select a news organization or the like as the reference organization. Alternatively, if the target document contains content related to government policy, the user may select the government, a ministry, a national agency, or the like as the reference organization.

[0036] Furthermore, at some point before step S202 is executed, the evaluation unit 12 may accept input of temporal constraints on the reference documents (for example, information indicating the publication time (publication date)) from the user. In this case, the evaluation unit 12 sets as reference documents only documents whose publication time falls within the time period (period) specified by the user. Because government policies and academic theories change over time, by selecting the document publication time, the user can refer to only documents from a specific period without referencing older documents. The publication time of each document may be registered in advance in the document DB 14.

[0037] Similarly, the user may be allowed to specify the country or region in which the document was issued. In this case, the evaluation unit 12 only sets as reference documents documents whose issuing region is included in the region specified by the user. This allows the basis for verifying the reliability of the target document to be limited to the assertions made in the document for the desired region. The issuing region of each document may be registered in advance in the document DB 14. The evaluation unit 12 may determine whether the issuing region of each document is included in the region specified by the user based on whether a character string indicating the region matches. Alternatively, information indicating the inclusion relationship (hierarchical relationship) of regions may be prepared in advance. In this case, the evaluation unit 12 may identify a region that matches a character string indicating the issuing region based on the information, and determine whether the identified region is included in the region specified by the user based on the information.

[0038] Note that distributed representations may be used for the vectorization in steps S201 and S202. Furthermore, if the amount of data (number of words, etc.) for each evaluation unit of the target document or each evaluation unit of the reference document is large, for example, a summary generated by natural language processing may be vectorized.

[0039] For convenience, FIG. 7 shows an example in which the target document and all of the reference documents are the evaluation unit.

[0040] Next, the evaluation unit 12 evaluates the similarity between each evaluation unit of the target document and each evaluation unit of each reference document (S203). The vector of each evaluation unit is used to evaluate the similarity. As described above, the method for evaluating the similarity between vectors may be performed by calculating the similarity (cosine similarity). As a result, a set of similarities between each evaluation unit of the target document and each reference document is obtained. Hereinafter, the similarity evaluation result will be simply referred to as "similarity."

[0041] Next, for each evaluation unit of the target document, the evaluation unit 12 identifies evaluation units of reference documents whose similarity to the evaluation unit is equal to or greater than a threshold value α or equal to or less than a threshold value β (S204). Hereinafter, an evaluation unit of a reference document whose similarity is equal to or greater than the threshold value α will be referred to as a "similar evaluation unit," and an evaluation unit of a reference document whose similarity is equal to or less than the threshold value β will be referred to as a "dissimilar evaluation unit." The value of α may be set arbitrarily as long as it is a value that can determine that the target document is likely to contain content that is very similar to the evaluation unit of the target document. The value of β may be set arbitrarily as long as it is a value that can determine that the target document is likely to contain content that is opposite to that of the evaluation unit of the target document.

[0042] For example, when cosine similarity is used to evaluate similarity, the similarity ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity and values ​​closer to -1 indicating less similarity. If the criterion for extracting similar evaluation units is that the angle between two vectors is ±45° or less (vectors are similar in direction = content is similar), then cos ±45° is approximately 0.7, so evaluation units of reference documents with similarity between 0.7 and 1.0 are similar evaluation units. Therefore, in this case, α = 0.7. On the other hand, if the criterion for extracting dissimilar evaluation units is that the angle between two vectors is between 135° and 225° (vectors are nearly opposite in direction = content is opposite), then cos 135° to cos 225° is a value between approximately -0.7 and -1, so β ​​= -0.7.

[0043] Next, the output unit 13 outputs a list of information (document name, document publication date, page number or section number, organization name, similarity, etc.) regarding each similar evaluation unit or each dissimilar evaluation unit for each evaluation unit of the target document (S205). As a result, for example, information such as that shown in FIG. 8 is output for one evaluation unit of the target document. Note that the document name, document publication date, organization name, etc. may be registered for each document in the document DB 14. Furthermore, the output form of the information is not limited to a predetermined one. The information may be displayed on a display device, stored in a file, or transmitted to another computer via a network.

[0044] By referring to such output information, the user can verify the likelihood (reliability) of each evaluation unit of the target document. For example, the user can determine that an evaluation unit that is highly similar to the reference document (an evaluation unit with many similar evaluation units and few dissimilar evaluation units) is highly reliable, and that an evaluation unit that is low in similarity to the reference document (an evaluation unit with few similar evaluation units and many dissimilar evaluation units) is low in reliability.

[0045] The output unit 13 may extract important words from each similar evaluation unit or each similar evaluation unit and output the important words as information about the similar evaluation unit or the dissimilar evaluation unit. The output unit 13 may also summarize each similar evaluation unit or each similar evaluation unit and output the summary as information about the similar evaluation unit or the dissimilar evaluation unit.

[0046] Furthermore, the output information does not have to be limited to the above (similar evaluation units or dissimilar evaluation units). For example, the results of sorting all reference documents based on the similarity may be output.

[0047] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0048] REFERENCE SIGNS LIST 10 Document verification support device 11 Selection unit 12 Evaluation unit 13 Output unit 14 Document DB 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims

1. A document verification support device comprising: an evaluation unit configured to evaluate the similarity between a certain document and a document published by a specified organization; and an output unit configured to output the evaluation result of the similarity.

2. The document verification support device according to claim 1, further comprising a selection unit configured to select some organizations from among the plurality of organizations as the predetermined organizations based on the similarity between the key words in the document and the key words in documents published by each of the plurality of organizations.

3. The document verification support device according to claim 1 or 2, characterized in that the evaluation unit is configured to evaluate the similarity between the certain document and documents published at a time designated by the user among documents made public by the specified organization.

4. The document verification support device according to claim 1 or 2, characterized in that the evaluation unit is configured to evaluate the similarity between the certain document and documents published in a region designated by the user among documents made public by the specified organization.

5. A document verification support method characterized in that a computer executes an evaluation procedure configured to evaluate the similarity between a certain document and a document published by a specified organization, and an output procedure configured to output the evaluation results of the similarity.

6. A program that causes a computer to execute an evaluation procedure configured to evaluate the similarity between a certain document and a document published by a specified organization, and an output procedure configured to output the evaluation results of the similarity.

Images