Information processing system and information processing method

Abstract

This information processing device is provided with: a calculation unit that calculates the degree of similarity between a sentence or token in a document to be analyzed and a sentence or token in a reference document, and determines information related to the sentence or token in the reference document and the degree of similarity; and an output unit that outputs the document to be analyzed to which the information has been added.

Description

Information processing system and information processing method

[0001] The present invention relates to a technique for collecting documents and analyzing the collected documents.

[0002] With the recent spread of the Internet and information processing devices, it is now possible to collect large amounts of information much faster than in the days when research was done through books.

[0003] For example, in organizations such as companies, there are many cases where it is necessary to analyze collected documents (a group of sentences) and extract necessary sentences from a large amount of sentences.

[0004] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, "https: / / arxiv.org / abs / 1810.04805"Nils Reimers, Iryna Gurevych , "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," https: / / arxiv.org / abs / 1908.10084

[0005] However, it takes a lot of time and effort for a human to analyze a large amount of collected text, for example, to understand the relationship between the text in the document being analyzed and other documents.

[0006] The present invention has been made in consideration of the above points, and aims to provide a technology that makes it possible to quickly grasp the relevance of sentences in a document to be analyzed to other documents.

[0007] According to the disclosed technology, there is provided an information processing system including: an arithmetic unit that calculates the similarity between a sentence or token in a document to be analyzed and a sentence or token in a reference document, and determines information relating to the sentence or token in the reference document and the similarity; and an output unit that outputs the document to be analyzed with the information added.

[0008] The disclosed technology provides a technology that enables quick understanding of the relevance of sentences in a document to be analyzed to other documents.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 100; FIG. 2 is a flowchart illustrating a procedure of pre-processing; FIG. 3 is a flowchart illustrating a procedure of analysis processing; FIG. 4 is a diagram illustrating grouping processing; FIG. 5 is a diagram illustrating an example of tag information; FIG. 6 is a diagram illustrating an example of display; and FIG. 7 is a diagram illustrating an example of the hardware configuration of the information processing system 100.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] (Terminology) First, definitions of the main terms used in this specification are given below.

[0012] NLP (Natural Language Processing): Natural language processing LLM (Large Language Models): Large-scale language processing Sentence: A set of sentences connected by punctuation marks, or a string of several characters separated by phrases and constants Document: A set of sentences Token: A string divided into words (singular or plural) Feature extraction: Converting the features of a sentence into a feature vector using NLP Heat map: A method of representing numerical values ​​with colors (a display method used in general thermography, etc.) Below, we will explain the conventional technology and its problems in more detail, and then explain the technology related to this embodiment. Note that the analysis of the conventional technology and the explanation of the problems described below are not publicly known.

[0013] (Regarding the Prior Art) As mentioned above, the recent spread of the Internet and information processing devices has made it possible to collect large amounts of information much faster than in the days when books were used for research. For example, in organizations such as companies, there are many cases where it is necessary to analyze collected documents (groups of sentences) and extract necessary sentences from the large amount of text. However, it takes a lot of time and effort for humans to analyze a large amount of collected text.

[0014] There are various perspectives for analyzing texts, but the main perspectives from which an organization typically analyzes texts are as follows: Note that "our organization" refers to the organization that conducts the analysis.

[0015] (1) Whether the content of the text being analyzed is consistent with the business policy, opinions, or ideas of the organization.

[0016] (2) The relevance of the content of the text being analyzed to matters of interest to the organization.

[0017] (3) Whether the content of the text being analyzed matches the law, rules, etc.

[0018] (4) Whether the content of the text being analyzed matches the content of documents used as reference by the organization (e.g., government-issued white papers, policy documents, etc.).

[0019] All of the above perspectives are necessary for checking whether the documents written by one's own organization are in line with the organization's policies or government policies. Furthermore, conducting analysis from the above perspectives is also necessary for identifying or identifying organizations that hold opposing views to one's own organization. Depending on the topic being collected, it may also be necessary to understand overseas literature, in which case language issues arise.

[0020] Generally, the task of analyzing texts from the above perspectives is performed manually by humans in organizations such as companies. However, proposals have been made to automate this process. For example, there is a feature extraction technique using BERT, etc., as a technique for grasping the semantic similarity between texts.

[0021] (Challenges) Even when using the above proposed automation, humans still need to make the final judgment on the content of the text. However, it takes a lot of time and effort for humans to analyze all the text from various perspectives.

[0022] Therefore, a technology is needed that allows a person to quickly grasp the relevance of a sentence in a document to be analyzed to other documents, but such a technology has not been proposed so far.

[0023] (Outline of the embodiment) In order to solve the above problem, in this embodiment, an information processing system 100 described below displays the similarity (relevance) between sentences or tokens in a document to be analyzed and the content of a reference document so that a person can quickly understand the similarity (relevance) between the sentences or tokens in the document to be analyzed and the content of a reference document.

[0024] Specifically, as a pre-processing step, multiple reference documents are grouped by content. Each group is assigned a specific color. By referencing the information of the grouped reference documents, the information processing system 100 displays each sentence (or each token) in the document to be analyzed in a specific color, and further displays the distance (similarity) between the document and the reference document using the intensity of the color. This allows a person to quickly grasp the relationship between the sentences or tokens in the document to be analyzed and other documents. Note that color may be added to the background of the text, or the text itself may be colored. In addition to color, emphasis may also be achieved by flashing.

[0025] (Device Configuration Example) Fig. 1 shows a configuration example of an information processing system 100 that performs analysis in this embodiment. As shown in Fig. 1, the information processing system 100 has an input unit 110, a calculation unit 120, an output unit 130, and a data storage unit 140. The information processing system 100 may also be called an information processing device 100.

[0026] The pre-processing and analysis processing will be described below. Here, the basic processing contents of the pre-processing and analysis processing will be described, and then more specific processing will be described using examples.

[0027] (Pre-processing) First, the pre-processing will be described. The pre-processing may be performed by the information processing system 100 that performs the analysis, or by a device (computer) other than the information processing system 100. Furthermore, some of the pre-processing (e.g., grouping) may be performed manually.

[0028] The pre-processing will be described in accordance with the procedure of the flowchart in Fig. 2. In the following description, it is assumed that the information processing system 100 is used when a device is used in the pre-processing.

[0029] <S101 (Step 101)> In S101, a reference document group is divided into one or more groups. In this embodiment, it is assumed that documents with a unified content are used as the document group in each group. For example, documents with content on which opinions are not divided, such as government agency documents or company documents, are used.

[0030] <S102> In S102, tag information corresponding to the group to which the document belongs is assigned to each document in each group. Note that the tag information assignment may be performed after the feature vectorization described below.

[0031] <S103> Here, it is assumed that the grouped documents are stored in the data storage unit 140 of the information processing system 100.

[0032] The calculation unit 120 of the information processing system 100 reads each document from the data storage unit 140, divides the document into tokens or sentences separated by punctuation marks, and labels the positions. The calculation unit 120 converts the sentences (or tokens) into feature vectors using NLP such as BERT, and records the feature vectors in a database (a database in the data storage unit 140). Any method may be used for converting the feature vectors, but the techniques disclosed in Non-Patent Documents 1 and 2, for example, may be used.

[0033] (Analysis Processing) Next, the flow of the analysis processing executed by the information processing system 100 will be described with reference to the flowchart in Fig. 3. Here, it is assumed that a group of reference documents that have been converted into feature vectors by pre-processing are stored in a database (DB) in the data storage unit 140.

[0034] <S201> In S201, a document to be analyzed is input from the input unit 110.

[0035] <S202, S203> In S202, the calculation unit 120 divides the document to be analyzed into tokens or sentences, as in the pre-processing, and generates a feature vector for each sentence (or each token).

[0036] In S204, the calculation unit 120 searches for each feature vector of each document in the DB using the feature vector of each sentence (or each token) in the document to be analyzed as a query, and extracts those with high similarity. The method for calculating similarity is not limited to a specific method, but for example, cosine similarity using an inner product or Jaccard similarity can be used.

[0037] <S205> In S205, the calculation unit 120 converts the similarity calculated for each query into a setting value for the color intensity of the heat map. That is, the calculation unit 120 determines the background color intensity to be applied to the sentences (or tokens) in the document to be analyzed based on the similarity (closeness) between the sentences (or tokens) to be analyzed and sentences (or tokens) that have a high similarity to the sentence (or token). The calculation unit 120 determines the color of the heat map as a color corresponding to the group of reference documents to which the sentences (or tokens) with a high similarity belong. The colors selected are colors that allow for maximum discrimination between groups (e.g., red and blue).

[0038] The calculation unit 120 assigns the color and intensity of the heat map determined as described above to the position of the sentence (or token) that is the search target (query) in the document to be analyzed. Note that the color is not limited to assigning color to the background, but may also be assigned to the text. Furthermore, as a means of expressing similarity, a blinking background or text may be used instead of or in addition to the heat map.

[0039] <S206> The output unit 130 outputs the analysis target document in which similar parts to the reference document are colored. Here, "output" may mean displaying the analysis target document on a display of the information processing system 100, or may mean displaying the analysis target document on a display of a terminal connected to the information processing system 100 via a network.

[0040] The above is the flow of the analysis process.

[0041] The calculation unit 120 may embed a hyperlink at the position of the colored sentence (or token) in the document to be analyzed. This hyperlink links to the name of a reference document that has a high similarity to the colored sentence (or token) and the position of the sentence (or token) in the reference document. This allows the user to click on the hyperlink to display the name of the reference document with a high similarity and the position of the sentence (or token) in the reference document.

[0042] The processes of S204 to S206 can be performed on reference documents of multiple groups (genres). For example, by using different colors for each group of reference documents and displaying the density (heat map), it is possible to check the relevance between the document to be analyzed and the reference documents of multiple related groups all at once. As described above, in this embodiment, it is possible to visualize the relevance of the sentences (or tokens) to be analyzed to the reference documents.

[0043] The pre-processing and analytical processing will be described in more detail below as examples. An example of the pre-processing will be referred to as Example 1, and an example of the analytical processing will be referred to as Example 2.

[0044] (Example 1: Pre-processing) <Selection of Candidates for Reference Documents> In the pre-processing, first, a reference document for analyzing the similarity with the document to be analyzed is selected. As the reference document (reference document), a plurality of documents (a document group) may be selected, or a specific document (for example, a highly reliable document such as a collection of laws and regulations or a white paper) may be selected.

[0045] When multiple documents are used as criteria, a grouping process is performed. Collecting multiple documents is equivalent to collecting a group of candidate criteria documents.

[0046] The method for collecting candidate reference documents that are assumed to belong to a certain group is not limited to a specific method. For example, a user may collect them manually using an apparatus (e.g., information processing system 100), or the apparatus (e.g., information processing system 100) may automatically collect them. In the following description, the information processing system 100 is used as the apparatus.

[0047] For example, a user inputs a keyword or a key phrase into the information processing system 100, and the information processing system 100 extracts a group of documents corresponding to the keyword or key phrase from an information source on the Internet (for example, a database such as a digital library) as a group of reference document candidates. As an example, when a keyword is used, the information processing system 100 extracts documents that are close to words in a top group of frequently occurring words in the information source.

[0048] In this embodiment, the information sources from which candidate reference documents are collected are limited to those from reliable organizations (e.g., government ministries and agencies) or those from organizations that are thought to have ideas similar to those of the organization itself, so as to maintain the reliability of the reference documents. For example, documents from government ministries and agencies are limited.

[0049] However, the selection of information sources is not limited to the selection based on the reliability as described above. For example, when examining the correspondence between the document to be analyzed and public opinion, it is possible to use information sources such as social media that are considered to contain direct opinions but are not necessarily reliable.

[0050] <Grouping into Primary Groups> Next, grouping will be described with reference to Fig. 4. Here, for example, it is assumed that a group of reference document candidates is stored in the data storage unit 140 of the information processing system 100, and that a user performs grouping using the information processing system 100.

[0051] First, the user separates documents issued by the same organization or organizations of the same lineage from the candidate reference documents into a primary group. In the example of Figure 4, documents A, B, and C are shown to have been separated into one group (primary group).

[0052] <Confirmation Process> Next, it is confirmed whether the grouping is correct. Specifically, the following process is carried out.

[0053] First, the information processing system 100 summarizes each document in the primary group using, for example, LLM. Next, the information processing system 100 converts the document summaries (e.g., sentences) into feature vectors. Any method may be used for converting the feature vectors, but for example, BERT using a Transformer is used.

[0054] Next, as shown in Fig. 4, the information processing system 100 calculates the distance between feature vectors obtained from the summaries of the documents that make up the group. The information processing system 100 checks the similarity (distance) between the documents in the group and determines whether the similarity (or distance) satisfies a predetermined condition (for example, a similarity of 0.6 or more).

[0055] If the information processing system 100 determines that the similarity does not satisfy the condition (i.e., the distance between the documents is large), it excludes the document from the group. Alternatively, a human may recheck the document.

[0056] In the example of Figure 4, the similarity between document A and document B satisfies the condition, but the similarity between document A and document C, or between document B and document C, does not satisfy the condition, so document C is excluded from the group.

[0057] The information processing system 100 manages the group of documents that have been confirmed to be grouped as a "reference document group." A group consisting of the reference documents is called a reference document group. The information processing system 100 assigns a unique ID as a reference document group ID to each document selected as a reference document.

[0058] <Creating Multiple Groups> The user / information processing system 100 performs the above-described process of selecting, grouping, and confirming candidate reference documents multiple times for organizations, fields, and categories of interest to create multiple reference document groups. Each group has a unique ID and is assigned a distinctive color. As a result, each group is associated with a color-coded display, such as a heat map.

[0059] <About Tags> Using the information processing system 100, a user assigns document information to each document belonging to a reference document group as a tag (which may also be called tag information) along with the reference document group ID (hereinafter referred to as group ID) of the reference document group. This group ID and tag information are also associated with information on feature quantities (specifically, feature vectors) described below, and are used as information during searches. Note that the group ID may also be included in the tag information.

[0060] An example of tag information assigned to a document is shown in FIG. 5. In the example of FIG. 5, each document with group ID = 1 is assigned tag information (government / agency, economic policy, finance / green bond, green, 2022, XX policy explanation) corresponding to (organization, category, theme, color, publication date, document name). Note that colors may be assigned after the feature vector is generated. In this example, colors are assigned after the feature vector is generated. The user may also select a color for a desired category or organization.

[0061] <Calculation of Document Feature Amounts> Next, the information processing system 100 divides the document into sentences or tokens at the punctuation level for each group and each document, and generates feature vectors for the sentences or tokens.

[0062] The information processing system 100 links each feature vector with the group ID of the document to which the sentence or token that is the source of the feature vector belongs, the tag information of the document, and the location information (page number, paragraph, number of characters, etc.) of the sentence or token that is the source of the feature vector.

[0063] The information processing system 100 stores the generated feature vector and the information linked to the feature vector together for each group and each document in the DB in the data storage unit 140 .

[0064] The user determines a reference color for each group ID and assigns the determined color to the group with that group ID. This color must not overlap between group IDs. The determined color is stored in the DB in the data storage unit 140 in association with the group ID.

[0065] (Example 2) Next, details of the analysis process executed by the information processing system 100 will be described as Example 2. Here, it is assumed that the information created in the pre-processing of Example 1 is stored in a DB of the data storage unit 140 of the information processing system 100. Hereinafter, this DB will be referred to as a reference document DB. Note that the reference document DB does not need to be provided within the information processing system 100, and may be provided, for example, on a network. In the following processing by the information processing system 100, input is performed by the input unit 110, output is performed by the output unit 130, and other processing is performed by the calculation unit 120.

[0066] <Extraction of Features from Document to be Analyzed> A document to be analyzed is input to the information processing system 100. The information processing system 100 divides the input document to be analyzed into sentence units or token units in the same manner as in the pre-processing.

[0067] Next, the information processing system 100 generates a feature vector for each of the divided sentences (or each of the tokens). The method for generating the feature vector is not limited to a specific method, but for example, a method using a Transformer model such as BERT can be used.

[0068] <Search> The information processing system 100 searches the reference document DB for feature vectors of each sentence (or each token) in the document to be analyzed that have a high similarity (i.e., a short distance) to the feature vector (referred to as a query).

[0069] Specifically, for example, the information processing system 100 searches for the feature vector that has the highest similarity (i.e., the closest distance) to the query feature vector among all feature vectors in the reference document DB. Note that any algorithm may be used for the search, such as a neighborhood search.

[0070] <Determining Color and Intensity> Next, for each sentence (or each token) in the analysis target document, the information processing system 100 obtains a color (a color corresponding to the group ID corresponding to the feature vector) associated with the feature vector extracted from the reference document DB by search. Furthermore, for each sentence (or each token) in the analysis target document, the information processing system 100 determines the color intensity based on the similarity to the extracted feature vector. For example, the information processing system 100 may use a darker color for higher similarity and a lighter color for lower similarity. For example, if the similarity is below a certain threshold, no color may be applied.

[0071] <Display> The information processing system 100 superimposes the background color determined by the above-described process onto the characters in each sentence (or each token) in the document to be analyzed, and displays the document to be analyzed with the background color superimposed. This makes it possible to display a heat map based on similarity superimposed on the document. An image of a document with a background color superimposed on multiple sentences is shown in FIG. 6. In FIG. 6, colors are represented by shading, etc. As shown in FIG. 6, the colors correspond to the genres, etc., of the corresponding reference documents. For example, blue corresponds to energy policy, red corresponds to telecommunications carrier policies, and green corresponds to monetary policy, etc.

[0072] As mentioned above, for user convenience, a link to a corresponding position in the reference document that is similar to a sentence (or token) in the document being analyzed may be embedded as a hyperlink in the colored sentence (or token).

[0073] (Hardware Configuration Example) Any of the information processing systems 100 (which may also be called information processing devices) described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on a cloud.

[0074] That is, the information processing system 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing system 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0075] Fig. 7 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 7 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0076] The program that realizes processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. The program may also be provided from another computer via a network. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0077] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing system 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0078] As described above, the technology according to the present embodiment can display the similarity between each sentence (or token) in a document to be analyzed and a reference document in the form of a heat map. This allows a user to check the similarity between a single or multiple genres of documents at once.

[0079] In this embodiment, the degree of similarity between each sentence (or token) in the document to be analyzed and the contents of the reference document is checked, so there may be cases where multiple reference documents have high similarity results. In such cases, it can be understood that a consensus has been reached on the content.

[0080] Furthermore, if not only the degree of similarity but also the degree of agreement of opinions is displayed as a heat map, differences in opinion can be visually judged.

[0081] The following additional notes are provided regarding the above-described embodiments.

[0082] <Additional Notes> (Additional Item 1) An information processing system comprising: a calculation unit that calculates similarities between sentences or tokens in a document to be analyzed and sentences or tokens in a reference document, and determines information relating to the sentences or tokens in the reference document and the similarity; and an output unit that outputs the document to be analyzed with the information added. (Additional Item 2) The information processing system described in Additional Item 1, wherein the information is a color corresponding to the reference document and a density corresponding to the similarity, and the output unit outputs the document to be analyzed with the background of the sentences or tokens colored using the color and the density. (Additional Item 3) The information processing system described in Additional Item 2, wherein the color corresponds to a group to which the reference document belongs. (Additional Item 4) An information processing method executed by an information processing system, comprising: a step of calculating similarities between sentences or tokens in a document to be analyzed and sentences or tokens in a reference document, and determining information relating to the sentences or tokens in the reference document and the similarity; and a step of outputting the document to be analyzed with the information added. (Supplementary Item 5) A non-transitory storage medium storing a program for causing a computer to function as the information processing system according to any one of Supplementary Items 1 to 3.

[0083] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0084] REFERENCE SIGNS LIST 100 Information processing system 110 Input unit 120 Calculation unit 130 Output unit 140 Data storage unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing system comprising: a calculation unit that calculates the similarity between sentences or tokens in a document to be analyzed and sentences or tokens in a reference document, and determines information relating to the sentences or tokens in the reference document and the similarity; and an output unit that outputs the document to be analyzed with the information added.

2. The information processing system of claim 1, wherein the information is a color corresponding to the reference document and a density corresponding to the similarity, and the output unit outputs the document to be analyzed in which the background of the sentence or token is colored using the color and density.

3. The information processing system according to claim 2, wherein the color corresponds to a group to which the reference document belongs.

4. An information processing method executed by an information processing system, comprising the steps of: calculating the similarity between a sentence or token in a document to be analyzed and a sentence or token in a reference document; determining information relating to the sentence or token in the reference document and the similarity; and outputting the document to be analyzed with the information added.

Images