Literature review assistance method, literature review assistance system, and literature review assistance program

Abstract

To solve the problem that it is difficult to exclude documents having similar background but different conclusions in screening of documents on the basis of a similarity between keywords.SOLUTION: In a document review assistance method executed by a computer system having a control section and a storage section. The storage section stores data on a plurality of documents, and the document review assistance method includes the steps of: creating, by the control section, a sentence vector on the basis of a sentence included in the plurality of documents; classifying, by the control section, the plurality of documents into a plurality of clusters on the basis of the created sentence vector; identifying, by the control section, a subgraph on a network of a word in a first document set included in at least one of clusters; and controlling, by the control section, the creation of the sentence vector on the basis of the identified subgraph.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for assisting literature review. [Background technology]

[0002] When planning a clinical trial for a drug, a systematic review is essential, thoroughly examining past literature on similar drugs and examining and evaluating high-quality research data. A systematic review involves, for example, searching a medical literature database for papers from an appropriate period, screening the results in two stages, and selecting papers that are appropriate to serve as the basis for a description of a specific clinical question. For example, the first screening involves selecting papers based on the content of their abstracts, and the second screening involves classifying the papers by their content, after which further selection is made to determine whether to accept or reject them. To prevent overlooking important papers, a large amount of literature must be examined, placing a heavy burden on personnel.

[0003] One example of a technology for assisting literature review is described in Japanese Patent Laid-Open No. 2020-140435 (Patent Document 1). Patent Document 1 states, "An information retrieval system for searching for sentences that match a search request, including a sentence set receiving unit that receives a sentence set including a plurality of sentences, a user search request receiving unit that receives a user's search request, a word vector conversion unit that converts each of the sentences and the search request into a word vector, a sentence vector conversion unit that converts the word vectors derived from each sentence and the word vectors derived from the search request into sentence vectors using a trained neural network, a similarity calculation unit that calculates the similarity between the sentence vector derived from each sentence and the sentence vector derived from the search request, and a search result presentation unit that presents the similarity between the sentence and the search request." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-140435 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 above efficiently extracts the next document to read by determining documents with similar contexts (documents with similar keywords) to the document that has been selected for adoption based on the similarity between the keyword vectors of the documents.

[0006] However, even if a document has a similar research background to the desired document, the document may be deemed unnecessary if it has a different final conclusion or a different research purpose. Even in such cases, if the similarity in the background results in similar keywords, it is not possible to exclude such documents based on the similarity between keywords, making it difficult to efficiently narrow down the search. [Means for solving the problem]

[0007] In order to solve at least one of the above problems, a representative example of the invention disclosed in the present application is a document review assistance method executed by a computer system having a control unit and a storage unit, the document review assistance method including: a first step in which the control unit creates sentence vectors based on sentences included in a plurality of acquired documents; a second step in which the control unit classifies the plurality of documents into a plurality of clusters based on the created sentence vectors; and a second step in which the control unit classifies a first set of documents included in at least one of the clusters. Obtaining binary relationship patterns specified by the two-term concepts, the type of concepts, and the relationship between the two terms that appear in the included documents a third step in which the control unit Obtained binary relation patterns and a fourth step of controlling the creation of the sentence vector based on the In the third step, the control unit classifies at least a part of the first document set into a positive set or a negative set based on the acquired binary relation pattern; and in the fourth step, the control unit controls the creation of the sentence vectors by using the positive set and the negative set as training data to learn a model for creating the sentence vectors such that sentence vectors created from documents belonging to the positive set are close to other sentence vectors created from documents belonging to the positive set, sentence vectors created from documents belonging to the negative set are close to other sentence vectors created from documents belonging to the negative set, and sentence vectors created from documents belonging to the positive set are far from sentence vectors created from documents belonging to the negative set. It is characterized by: [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to efficiently narrow down documents by controlling the vectorization of sentences contained in the documents. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the configuration of a literature review assisting device according to a first embodiment of the present invention. [Figure 2] 3 is a flowchart showing an example of processing executed by the literature review assisting device according to the first embodiment of the present invention. [Figure 3] 4 is a flowchart showing an example of a process performed by the literature review assisting device according to the first embodiment of the present invention to calculate a sentence vector. [Figure 4] 1 is a flowchart showing an example of a process in which the literature review assisting device according to the first embodiment of the present invention creates a binary relation pattern. [Figure 5] FIG. 2 is an explanatory diagram illustrating an example of binary relation data held by the literature review assisting device according to the first embodiment of the present invention. [Figure 6] FIG. 2 is an explanatory diagram showing an example of adopted / unadopted data held by the literature review assisting device according to the first embodiment of the present invention. [Figure 7] FIG. 2 is an explanatory diagram showing an example of binary relation pattern data held by the literature review assisting device according to the first embodiment of the present invention. [Figure 8] FIG. 2 is an explanatory diagram showing an example of a screen displayed by the literature review assisting device according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a block diagram showing an example of the configuration of a literature review assisting device 100 according to a first embodiment of the present invention.

[0012] The literature review assisting device 100 is a computer system having a data input / output unit 101 , a control unit 102 , a memory 103 and a storage unit 104 .

[0013] The data input / output unit 101 may include a communication interface connected to a network for communicating with an information system on the network, and an external device interface to which an external device is connected. In this embodiment, the data input / output unit 101 communicates with a literature database 105 via the network. The literature database 105 is, for example, a medical paper database such as "PubMed." In addition, an input unit 106 such as a keyboard and a mouse, and a display unit 107 such as an image display device are connected to the data input / output unit 101 as external devices.

[0014] The control unit 102 is a CPU (Central Processing Unit) that executes a program stored in the memory 103. The processing of the literature review assisting device 100 described below is actually executed by the control unit 102 in accordance with the program stored in the memory 103.

[0015] The memory 103 is a storage device such as a DRAM (Dynamic Random Access Memory), and stores programs executed by the control unit 102, data referenced by the processing of the control unit 102, data generated by the processing of the control unit 102, and the like.

[0016] The storage unit 104 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and can store the above programs, data, etc. For example, the programs, etc. may be stored in the storage unit 104, and at least a portion of them may be stored in the memory 103 as needed. Furthermore, at least a portion of the data included in the literature database 105 that is to be processed by the literature review assisting device 100 may be stored in the storage unit 104 as needed.

[0017] FIG. 2 is a flowchart showing an example of processing executed by the literature review assisting device 100 according to the first embodiment of the present invention.

[0018] First, the literature review assisting device 100 acquires literature data (step S201). For example, the literature review assisting device 100 may acquire literature data obtained by performing a primary screening such as a simple keyword search from the literature database 105. The acquired data may be stored in the storage unit 104.

[0019] Next, the literature review assisting device 100 executes a loop process from steps S202 to S205 for each of the acquired documents. Specifically, the literature review assisting device 100 extracts sentences of interest from each document (step S203). The method for extracting sentences of interest is not limited. For example, a title, a sentence including a purpose expression, a conclusion, etc., which contain information necessary from the viewpoint of screening, may be extracted. In this case, for example, a sentence including a purpose expression such as "aim," "perform," "examine," "analyze," or "estimate" may be extracted as a sentence including a purpose expression. Furthermore, for example, a sentence including "conclusion" or the last sentence of a document may be extracted as a conclusion.

[0020] Next, the literature review assisting device 100 calculates sentence vectors for the excerpted sentences (step S204). At this time, the literature review assisting device 100 uses a pooling model. When the loop from steps S202 to S205 is executed for the first time, a pooling model including pre-prepared initial parameters is used. The calculation of sentence vectors using the pooling model will be described later with reference to FIG. 3. By executing the processes of steps S203 and S204 for all the acquired documents, sentence vectors for all the acquired documents are calculated.

[0021] Next, the document review assisting device 100 performs clustering of the acquired documents based on the calculated sentence vectors (step S206). The specific clustering method is not limited, and any method, such as a publicly known method, can be adopted, so a detailed description will be omitted. Furthermore, the document review assisting device 100 performs dimensionality reduction of the sentence vectors (step S207). This can be performed using a publicly known method such as principal component analysis, so a detailed description will be omitted.

[0022] Next, the document review assisting device 100 displays the two-dimensional distribution of the sentence vectors of the clustered documents (step S208). An example of the display will be described later with reference to FIG.

[0023] Next, the literature review assisting device 100 acquires the acceptance / rejection data (step S209). For example, a user of the literature review assisting device 100 may read some of the literature belonging to one of the clusters, determine whether to accept or reject each of the literature, and input the result into the literature review assisting device 100. An example of the acceptance / rejection data acquired in this manner will be described later with reference to FIG. 6.

[0024] Next, the document review assisting device 100 calculates binary relation patterns appearing in sentences extracted as sentences of interest from the documents from which the accepted / unaccepted data has been obtained (step S210). The calculation of binary relation patterns will be described later with reference to FIG.

[0025] Next, the literature review assisting device 100 displays the binary relation patterns and accepts selections therefor (step S211). Examples of this display and selection will be described later with reference to FIG.

[0026] Next, the document review assisting device 100 creates positive / negative data by classifying each document as positive or negative according to the selection received in step S211 (step S212). The documents to be classified may be all documents acquired in step S201, or all documents belonging to the clusters generated in step S206 that were the targets of the acceptance / rejection determination in step S209. This allows documents that the user has not yet read to be classified as positive or negative and used for learning the pooling model described below, thereby improving the accuracy of document classification based on sentence vectors.

[0027] Specifically, the document review assisting device 100 classifies documents in which a binary relation pattern classified as positive appears into a positive set (i.e., the accepted documents), and classifies documents in which a binary relation pattern classified as negative appears into a negative set (i.e., the unaccepted documents). When both a binary relation pattern classified as positive and a binary relation pattern classified as negative appear in a document, the document may be classified into a positive set or a negative set based on a predetermined criterion. For example, the document review assisting device 100 may classify documents based on binary relation patterns that appear in sentences of high importance, such as classifying the document based on binary relation patterns that appear in the document's conclusion sentence. Alternatively, the document may be classified based on the number of times each binary relation pattern appears, or based on the importance (described below) of the binary relation patterns that appear.

[0028] Next, the literature review assisting device 100 uses the positive / negative data generated in step S212 to train a model for calculating sentence vectors (step S213). For example, for two sentences, sentence A and sentence B, if both are sentences from accepted documents, the literature review assisting device 100 assigns weights to the vectors of each token of each sentence (the vectorization of sentence tokens will be described later with reference to FIG. 3). At this time, since both of the above two sentences are sentences from accepted documents, the literature review assisting device 100 trains weights so that their vectors become closer to each other. The same applies when both of the above two sentences are sentences from rejected documents. On the other hand, if one of the above two sentences is a sentence from accepted documents and the other is a sentence from rejected documents, the literature review assisting device 100 trains weights so that their vectors become farther apart. The model involves weight adjustment as described above, and may involve pooling or weight adjustment in other locations. Hereinafter, the model will be referred to as a pooling model.

[0029] Thereafter, the process returns to step S204, where the literature review assisting device 100 calculates sentence vectors for each document using the learned pooling model. Furthermore, the literature review assisting device 100 executes the processes from step S206 onward. By clustering the sentence vectors calculated using the learned pooling model in step S213, documents classified as positive and documents classified as negative tend to be classified into the same cluster, and documents classified as positive and documents classified as negative tend to be classified into different clusters. This improves the accuracy of document narrowing.

[0030] Furthermore, compared to simply classifying documents using positive / negative judgment results based on binary relational patterns, by clustering sentence vectors based on the pooling model learned as described above, it becomes possible to determine that documents with similar word meanings, even if they use different syntax, are highly similar.

[0031] The literature review assisting device 100 may execute steps S206 to S213 to train a pooling model, and then use the trained pooling model to calculate sentence vectors for each document in step S203. This process may be repeated until a predetermined condition is met. The predetermined condition may be, for example, that the ratio of documents classified as negative to all documents in the positive / negative data created in step S212 is smaller than a predetermined threshold, or that the learning has converged, e.g., the ratio of documents classified as negative remains constant even after repeated learning. In this case, the process shown in FIG. 2 ends when the predetermined condition is met.

[0032] FIG. 3 is a flowchart showing an example of a process in which the literature review assisting device 100 according to the first embodiment of the present invention calculates a sentence vector.

[0033] First, the document review assisting device 100 vectorizes tokens of an input sentence (step S301). For example, the document review assisting device 100 may convert words of the input sentence into vectors using a predetermined natural language processing model (e.g., BERT: Bidirectional Encoder Representations from Transformers, etc.).

[0034] Next, the literature review assisting device 100 vectorizes the sentence using the pooling model (step S302). For example, the literature review assisting device 100 may calculate the sentence vector by multiplying the vector obtained in step S301 by the weight of the pooling model and adding the result.

[0035] When the process shown in FIG. 3 is executed in step S204 when the loop from steps S202 to S205 is executed for the first time, a predetermined initial value is used as the pooling weight, and when the process shown in FIG. 3 is executed after learning in step S213, the pooling weight learned in step S213 is used.

[0036] This completes the process of calculating the sentence vector.

[0037] FIG. 4 is a flowchart showing an example of a process in which the literature review assisting device 100 according to the first embodiment of the present invention creates a binary relation pattern.

[0038] First, the literature review assisting device 100 acquires the IDs of the literature determined to be adopted (also referred to as adopted literature) and the literature determined to be unadopted (also referred to as unadopted literature) based on the adopted / unadopted data acquired in step S209 (step S401).

[0039] Next, the literature review assisting device 100 acquires and aggregates binary relations that appear in the literature identified by the literature ID acquired in step S401 from the binary relation database. An example of the binary relation data thus aggregated will be described later with reference to Fig. 5. The binary relation database stores binary relations that indicate the relationship between two word concepts, the types of each concept, and the relationship between them as information indicating the network of words that appear in the literature, and from among these, the binary relations that appear in the literature identified by the literature ID acquired in step S401 are identified as subgraphs.

[0040] Next, the literature review assisting device 100 generates binary relation patterns from the binary relations collected in step S402, and calculates the number of occurrences of each binary relation pattern in the adopted literature and the number of occurrences in the non-adopted literature (step S403). Here, the binary relation pattern may be the binary relation itself, or may be an aggregation of multiple binary relations by setting any value for any item that identifies the binary relation (in other words, an aggregation of multiple binary relations in which any value for any item that identifies the binary relation is different). Examples of binary relation patterns will be described later with reference to FIG. 7.

[0041] Next, the literature review assisting device 100 sorts the binary relation patterns based on their tendency to appear more in accepted documents than in non-accepted documents (step S404). For example, the literature review assisting device 100 may calculate the ratio of the number of times each binary relation pattern appears in accepted documents to the number of times each binary relation pattern appears in non-accepted documents, and sort the binary relation patterns in descending order of this ratio.

[0042] This completes the process of creating a binary relation pattern.

[0043] FIG. 5 is an explanatory diagram showing an example of binary relation data held by the literature review assisting device 100 according to the first embodiment of the present invention.

[0044] The binary relation data 500 shown in FIG. 5 may be stored, for example, in the memory unit 104 of the literature review support device 100, and at least a partial copy thereof may be stored in the memory 103 as needed for processing by the control unit 102, and may be referenced, updated, etc.

[0045] The binary relation data 500 includes a relation ID 501 , a relation type 502 , a subject node 503 , a subject type 504 , an object node 505 , an object type 506 , and a document ID 507 .

[0046] In this embodiment, an example will be described in which a binary relation indicates the relationship between a subject and an object in a sentence. Relation ID 501 is information that identifies each binary relation. Relation type 502 indicates the type of each binary relation. Subject node 503 indicates the word that corresponds to the subject of each binary relation. Subject type 504 indicates the type of word that corresponds to the subject of each binary relation. Object node 505 indicates the word that corresponds to the object of each binary relation. Object type 506 indicates the type of word that corresponds to the object of each binary relation. Document ID 507 is information that identifies the document in which each binary relation appears.

[0047] For example, the first row of the binary relation data 500 in Fig. 5 stores information about a binary relation with a relation ID of "S0001." This indicates that a binary relation in which the subject "NOAC" of type "test" and the object "CVEs" of type "test" are "associated" appears in a document with a document ID of "L001."

[0048] In the example of Figure 5, one binary relation corresponds to one document, but in reality, multiple binary relations may appear in one document, and one binary relation may appear in multiple documents.

[0049] FIG. 6 is an explanatory diagram showing an example of the accepted / unaccepted data held by the literature review assisting device 100 according to the first embodiment of the present invention.

[0050] The adopted / non-adopted data 600 shown in Figure 6 may be stored, for example, in the memory unit 104 of the literature review assistance device 100, and at least a partial copy thereof may be stored in the memory 103 as needed for processing by the control unit 102, and may be referenced, updated, etc.

[0051] The accepted / unaccepted data 600 includes a document ID 601 and an accepted / unaccepted flag 602. The document ID 601 is information that identifies each document. The accepted / unaccepted flag 602 indicates the result of the user's reading of each document and their decision as to whether the document is accepted or not. In the example of FIG. 6, documents with document IDs "L001" to "L003", "L008", and "L009" have been accepted, while documents with document IDs "L004" to "L007" have been decided as not being accepted.

[0052] FIG. 7 is an explanatory diagram showing an example of binary relation pattern data held by the literature review assisting device 100 according to the first embodiment of the present invention.

[0053] The binary relation pattern data 700 shown in FIG. 7 may be stored, for example, in the memory unit 104 of the literature review assist device 100, and at least a partial copy thereof may be stored in the memory 103 as needed for processing by the control unit 102, and may be referenced, updated, etc.

[0054] The binary relation pattern data 700 includes a pattern ID 701 , a relation type 702 , a subject node 703 , a subject type 704 , an object node 705 , an object type 706 , and a ratio 707 .

[0055] Pattern ID 701 is information that identifies each binary relation pattern. Relation type 702 indicates the type of each binary relation pattern. Subject node 703 indicates the word that corresponds to the subject of each binary relation pattern. Subject type 704 indicates the type of word that corresponds to the subject of each binary relation pattern. Object node 705 indicates the word that corresponds to the object of each binary relation pattern. Object type 706 indicates the type of word that corresponds to the object of each binary relation pattern. Ratio 707 indicates the ratio of the number of times each binary relation pattern appears in adopted documents to the number of times it appears in non-adopted documents.

[0056] For example, information about a binary relation pattern with pattern ID "P001" is stored in the first row of the binary relation pattern data 700 shown in Fig. 7. In this example, a binary relation pattern in which there is "not associate" between the subject "NAFLD" of type "Disease and Symptom" and any object of type "Disease and Symptom" appears in both the non-adopted and adopted documents, and the ratio of the number of occurrences in the adopted documents to the number of occurrences in the non-adopted documents is "0.3".

[0057] In the example on the first line above, the value of the object node 705 is a wildcard (*). This indicates that if the binary relationship between the subject "NAFLD," whose type is "Disease and Symptom," and the object whose type is "Disease and Symptom" is "not associated," then the binary relationship falls under the binary relationship pattern with pattern ID "P001," regardless of the type of the object itself. In this way, by making at least one value of the item specifying the binary relationship optional, multiple binary relationships may be aggregated into a single binary relationship pattern. Alternatively, as in the example on the second line of the binary relationship pattern data 700 shown in FIG. 7, one binary relationship may directly correspond to one binary relationship pattern.

[0058] FIG. 8 is an explanatory diagram showing an example of a screen displayed by the literature review assisting device 100 according to the first embodiment of the present invention.

[0059] When the user inputs information specifying the document data to be read into the first screening data input section 801 and operates the display button 802, the specified document data after first screening is read (step S201), and steps S201 to S208 are executed for the read document data. The sentence vector display section 803 displays the two-dimensional distribution of the sentence vectors after dimension reduction, and further displays the clusters obtained as a result of clustering (step S206).

[0060] When the user selects at least one cluster by checking the cluster selection section 805 and operates the download button 806, the document data belonging to the selected cluster is downloaded. The user reads some of the data, decides whether to accept or reject them, and records the result as accepted / rejected data 600 in the document review assist device 100.

[0061] When the user inputs information specifying the generated accepted / rejected data 600 into the accepted / rejected data input section 807 and operates the upload button 808, the specified accepted / rejected data 600 is uploaded. The literature review assisting device 100 calculates a binary relation pattern for the target literature data (for example, all literature data obtained by the first screening or all literature data included in a specified cluster) based on the uploaded accepted / rejected data 600 (step S210), and displays the calculated binary relation pattern (step S211).

[0062] For example, the binary relation display unit 804 displays the calculated binary relation or binary relation pattern. While the display method is not limited, in the example of FIG. 8, a graph is displayed in which the nodes of the binary relation (e.g., subject or object) are represented by circles, and the edges corresponding to the relationships between the nodes are represented by lines connecting the two circles. For example, the number of occurrences of words corresponding to the nodes may be represented by the size of the circles, or the strength of the relationships between the words corresponding to the nodes may be represented by the distance between the circles connected by lines. The strength of the relationship may be based on the number of occurrences or the type of word indicating the relationship. Furthermore, the category of the concept corresponding to each node (e.g., subject type, object type, etc.) may be represented by the color of the circle. Although omitted in FIG. 8, the words of the nodes corresponding to the circles may be displayed on the graph. Alternatively, the binary relations may be displayed in a manner different from the above.

[0063] The negative syntax display section 809 displays a list of pairs of syntaxes specified by the values ​​of the relation type 702, subject node 703, subject type 704, object node 705, and object type 706 of the generated binary relation pattern, and the importance of each syntax. Here, the importance of a syntax corresponds to, for example, the ratio 707.

[0064] The positive syntax display section 811 may display the same list as the list displayed in the negative syntax display section 809. However, the negative syntax display section 809 may display syntaxes in descending order of importance, and the positive syntax display section 811 may display syntaxes in descending order of importance.

[0065] The user can select a negatively relevant sentence by checking the checkbox 810 corresponding to the sentence that the user has determined to be negative with reference to the negative sentence display section 809. Similarly, the user can select a positively relevant sentence by checking the checkbox 812 corresponding to the sentence that the user has determined to be positive with reference to the positive sentence display section 811 (step S211).

[0066] For example, the user may select whether each construct is positive or negative based on its importance, such as selecting a construct with a high importance as a construct that corresponds positively and selecting a construct with a low importance as a construct that corresponds negatively. Alternatively, the user may select whether a construct is positive or negative based on the content of the construct.

[0067] When the user selects a sentence corresponding to a negative sentence and a sentence corresponding to a positive sentence and operates the learning button 813, steps S212 and S213 are executed and the pooling model is learned.

[0068] The functions of the literature review assisting device 100 may be provided via an API (Application Programming Interface). For example, the screen shown in Fig. 8 may be displayed on a terminal device (not shown) connected to the literature review assisting device 100 via a network (not shown) connected to the data input / output unit 101. When a user of the terminal device operates the input unit, check boxes, buttons, etc. of Fig. 8, the results may be transmitted to the literature review assisting device 100, which may then execute the processes of Figs. 2 to 4 and transmit the results (e.g., information identifying the final clustered literature) to the terminal device.

[0069] Furthermore, the system according to the embodiment of the present invention may be configured as follows.

[0070] (1) A literature review assistance method executed by a computer system having a control unit (e.g., control unit 102) and a memory unit (e.g., memory unit 104), the literature review assistance method including a first step (e.g., S204) in which the control unit creates sentence vectors based on sentences contained in multiple acquired documents; a second step (e.g., S206) in which the control unit classifies the multiple documents into multiple clusters based on the created sentence vectors; a third step (e.g., S209 to S212) in which the control unit identifies subgraphs (e.g., binary relations) on a network of words in a first set of documents contained in at least one cluster; and a fourth step (e.g., S213) in which the control unit controls the creation of the sentence vectors based on the identified subgraphs.

[0071] This allows for efficient narrowing down of documents by controlling the vectorization of sentences contained in the documents.

[0072] (2) In (1) above, the subgraph includes information indicating word relationships (e.g., binary relationships) in the word network, and in the third step, the control unit classifies at least a portion of the first document set into a positive set or a negative set based on the word relationships included in the identified subgraph, and in the fourth step, the control unit controls the creation of sentence vectors by using the positive set and the negative set as training data to learn a model for creating sentence vectors so that sentence vectors created from documents belonging to the positive set are close to other sentence vectors created from documents belonging to the positive set, sentence vectors created from documents belonging to the negative set are close to other sentence vectors created from documents belonging to the negative set, and sentence vectors created from documents belonging to the positive set are farther apart from sentence vectors created from documents belonging to the negative set.

[0073] This allows us to refine the model that vectorizes the sentences contained in the documents, allowing us to efficiently narrow down the documents.

[0074] (3) In the fourth step of (2) above, the control unit learns a model for creating sentence vectors by adjusting the weight values ​​included in the model for creating sentence vectors based on classification into a positive set or a negative set.

[0075] This allows us to refine the model that vectorizes the sentences contained in the documents, allowing us to efficiently narrow down the documents.

[0076] (4) In (3) above, the information indicating the relationship between words includes information indicating a binary relationship pattern identified by the concept of two words, the type of concept, and the relationship between the two words (e.g., binary relationship pattern data 700), and the information indicating the relationship between words included in the identified subgraph includes information specifying at least one of the multiple binary relationship patterns as positive and at least one of the other binary relationship patterns as negative (e.g., information entered via checkboxes 810 or 812), and in the third step, the control unit classifies documents including binary relationship patterns specified as positive into a positive set and documents including binary relationship patterns specified as negative into a negative set.

[0077] This allows for efficient narrowing down of documents by controlling the vectorization of sentences contained in the documents based on the tendency of appearance of relationships between concepts contained in the documents.

[0078] (5) In the third step of (4) above, the control unit creates at least one pattern of binary relations by aggregating multiple binary relations that have different values ​​for at least one of the items of the concepts of the two words, the type of the concept, and the relationship between the two words.

[0079] This increases the number of samples that correspond to the pattern, making it easier to compare the frequency of occurrence in accepted and unaccepted documents. In addition, the general-purpose patterns make it easier to capture the characteristics of accepted and unaccepted documents.

[0080] (6) In the third step of (4) above, the control unit calculates the ratio between the number of times the binary relation pattern appears in the adopted documents and the number of times it appears in the non-adopted documents (for example, the importance in the negative syntax display section 809 and the positive syntax display section 811) based on information that designates part of the first document set as adopted documents and part of the first document set other than the adopted documents as non-adopted documents, and outputs the calculated ratio.

[0081] This facilitates classification of positive and negative documents based on the number of occurrences of binary relation patterns.

[0082] (7) In (6) above, the information indicating the relationship between the words contained in the identified subgraph includes information that designates as positive a binary relationship pattern in which the ratio of the number of occurrences in adopted documents to the number of occurrences in non-adopted documents is high, and as negative a binary relationship pattern in which the ratio of the number of occurrences in adopted documents to the number of occurrences in non-adopted documents is low.

[0083] This allows proper classification of positive and negative documents based on the number of occurrences of binary relation patterns.

[0084] (8) In the third step of (6) above, the control unit classifies documents that do not fall into either the accepted or the rejected category into either a positive set or a negative set based on the pattern of the binary relationship contained in the documents that belong to the first document set and do not fall into either the accepted or the rejected category.

[0085] This allows documents that the user has not yet read to be classified into a positive set or a negative set, which can be used to create a model and improve the accuracy of the model.

[0086] (9) In the third step of (6) above, the control unit classifies documents that do not fall into either the adopted or non-adopted categories into either a positive set or a negative set based on the binary relationship patterns contained in documents that do not fall into either the adopted or non-adopted categories among the multiple documents.

[0087] This allows documents that the user has not yet read to be classified into a positive set or a negative set, which can be used to create a model and improve the accuracy of the model.

[0088] (10) In the first step of (3) above, the control unit creates a sentence vector using a model including a predetermined weight value, and after executing the fourth step, the control unit controls the creation of the sentence vector by again executing the first step using a model including the weight value adjusted in the fourth step.

[0089] This allows us to refine the model that vectorizes the sentences contained in the documents, allowing us to efficiently narrow down the documents.

[0090] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to facilitate a better understanding of the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0091] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in storage devices such as nonvolatile semiconductor memory, hard disk drives, and solid-state drives (SSDs), or in computer-readable, non-transitory data storage media such as IC cards, SD cards, and DVDs.

[0092] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0093] 100 Literature review aid 101 Data input / output section 102 Control section 103 memory 104 Storage section 105 Literature Database 106 Input section 107 Display section

Claims

1. A literature review assistance method executed by a computer system having a control unit and a storage unit, comprising: The literature review assistance method includes: a first step in which the control unit creates a sentence vector based on sentences included in the plurality of acquired documents; a second step in which the control unit classifies the plurality of documents into a plurality of clusters based on the created sentence vectors; a third step in which the control unit acquires a pattern of a binary relation specified by two-word concepts appearing in documents included in a first document set included in at least one of the clusters, the type of the concepts, and the relationship between the two words; a fourth step in which the control unit controls creation of the sentence vector based on the acquired binary relation pattern; In the third step, the control unit classifies at least a part of the first document set into a positive set or a negative set based on the acquired binary relation pattern; In the fourth step, the control unit uses the positive set and the negative set as learning data, and controls the creation of the sentence vectors by learning a model for creating the sentence vectors so that sentence vectors created from documents belonging to the positive set are closer to other sentence vectors created from documents belonging to the positive set, sentence vectors created from documents belonging to the negative set are closer to other sentence vectors created from documents belonging to the negative set, and sentence vectors created from documents belonging to the positive set are farther apart from sentence vectors created from documents belonging to the negative set.

2. A literature review assistance method according to claim 1, comprising: In the fourth step, the control unit learns a model for creating the sentence vector by adjusting weight values ​​included in the model for creating the sentence vector based on the classification into the positive set or the negative set.

3. A literature review assistance method according to claim 2, comprising: At least one of the plurality of binary relation patterns is designated as positive and at least one of the plurality of binary relation patterns is designated as negative; In the third step, the control unit classifies documents that include the binary relation pattern designated as positive into the positive set, and documents that include the binary relation pattern designated as negative into the negative set.

4. A literature review assistance method according to claim 3, comprising: In the third step, the control unit creates at least one of the binary relationship patterns by aggregating multiple binary relationships that have different values ​​for at least one of the items of the concept of the two words, the type of the concept, and the relationship between the two words.

5. A literature review assistance method according to claim 3, comprising: In the third step, the control unit calculates a ratio between the number of times the binary relation pattern appears in the accepted documents and the number of times it appears in the non-accepted documents based on information that designates a portion of the first set of documents as accepted documents and a portion of the first set of documents other than the accepted documents as non-accepted documents, and outputs the calculated ratio.

6. A literature review assistance method according to claim 5, comprising: A literature review assistance method characterized in that a binary relationship pattern in which the ratio of the number of times it appears in the accepted literature to the number of times it appears in the non-accepted literature is high is designated as positive, and a binary relationship pattern in which the ratio of the number of times it appears in the accepted literature to the number of times it appears in the non-accepted literature is low is designated as negative.

7. A literature review assistance method according to claim 5, comprising: In the third step, the control unit classifies documents that do not fall into either the accepted documents or the rejected documents into either the positive set or the negative set based on the pattern of the binary relationship contained in the documents that belong to the first document set and do not fall into either the accepted documents or the rejected documents.

8. A literature review assistance method according to claim 5, comprising: In the third step, the control unit classifies documents that do not fall into either the accepted documents or the rejected documents into either the positive set or the negative set based on the pattern of the binary relationship contained in the documents among the plurality of documents that do not fall into either the accepted documents or the rejected documents.

9. A literature review assistance method according to claim 2, comprising: In the first step, the control unit creates a sentence vector using the model including a predetermined value of the weight; A literature review assistance method characterized in that, after executing the fourth step, the control unit controls the creation of the sentence vector by again executing the first step using the model including the weight values ​​adjusted in the fourth step.

10. A literature review assistance system having a control unit and a memory unit, The control unit a first step of creating sentence vectors based on sentences contained in the acquired documents; a second step of classifying the documents into a plurality of clusters based on the created sentence vectors; a third step of acquiring a pattern of binary relations specified by two-word concepts, the type of the concepts, and the relationship between the two words appearing in documents included in the first document set included in at least one of the clusters; a fourth step of controlling the creation of the sentence vector based on the pattern of the binary relations obtained; In the third step, the control unit classifies at least a part of the first document set into a positive set or a negative set based on the acquired binary relation pattern; In the fourth step, the control unit controls the creation of the sentence vectors by learning a model for creating the sentence vectors using the positive set and the negative set as learning data so that sentence vectors created from documents belonging to the positive set are closer to other sentence vectors created from documents belonging to the positive set, sentence vectors created from documents belonging to the negative set are closer to other sentence vectors created from documents belonging to the negative set, and sentence vectors created from documents belonging to the positive set are farther apart from sentence vectors created from documents belonging to the negative set.

11. A literature review assistance program for controlling a computer system, comprising: the computer system includes a control unit and a storage unit; The literature review assistance program a first step of creating sentence vectors based on sentences contained in the acquired documents; a second step of classifying the documents into a plurality of clusters based on the created sentence vectors; a third step of acquiring a pattern of binary relations specified by two-word concepts, the type of the concepts, and the relationship between the two words appearing in documents included in the first document set included in at least one of the clusters; a fourth step of controlling creation of the sentence vector based on the acquired binary relation pattern; the third step includes a step of classifying at least a portion of the first document set into a positive set or a negative set based on the acquired binary relation pattern; The fourth step includes a step of controlling the creation of the sentence vectors by learning a model for creating the sentence vectors using the positive set and the negative set as training data so that sentence vectors created from documents belonging to the positive set are closer to other sentence vectors created from documents belonging to the positive set, sentence vectors created from documents belonging to the negative set are closer to other sentence vectors created from documents belonging to the negative set, and sentence vectors created from documents belonging to the positive set are farther apart from sentence vectors created from documents belonging to the negative set.

Images