Training data generation program, training data generation method, and training data generation device

Abstract

To easily generate training data for a machine learning model to specify a meta-analysis object document.SOLUTION: A training data generation program specifies a plurality of meta-analysis documents citing a first document, determines respective similarities of the plurality of meta-analysis documents on the basis of respective feature information of the plurality of meta-analysis documents that are specified, and generates training data for machine learning including the first document if a similarity is equal to and more than a threshold.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a technique for generating training data. [Background technology]

[0002] In the medical field, meta-analysis is known, in which data from multiple past clinical studies are integrated and analyzed using statistical methods.

[0003] Meta-analysis is extremely important in the practice of evidence-based medicine. When conducting a meta-analysis, a great deal of effort is required to narrow down the literature to be analyzed.

[0004] The following is an example of the process of narrowing down the literature to be included in the meta-analysis using the conventional method. In the example shown below, the procedure for narrowing down the literature to be included in the meta-analysis (meta-analysis target literature) from multiple literature related to the analysis of the cancer risk of ARBs (Angiotensin Receptor Blockade) is shown.

[0005] [Step 1] Search for RCT papers on ARBs. Specifically, perform an OR search using the generic names of ARBs (losartan, valsartan, candesartan, irbesartan, eprosartan, telmisartan, and olmesartan). In addition, use the filters randomized controlled trial, human, and English. This step 1 search results in 2057 results.

[0006] [Step 2] Manually narrow down the search results based on the following conditions (i) to (v). Condition (i) Exclude experiments with a duration of less than 12 months Condition (ii) Exclude studies with fewer than 100 patients Condition (iii) Exclude articles that do not contain the words “cancer”, “carcinoma”, “malignancy”, “neoplasm”, “neoplasia”, or “tumor”. Condition (iv) excludes experiments in which all groups received ARBs Condition (v) Exclude meta-analyses After narrowing down the results in step 2, 60 results remain.

[0007] [Step 3] Researchers will then read the contents of the 60 articles narrowed down by step 2 to narrow down the list to those that report on the occurrence of cancer and those that report on deaths from cancer. This narrows down the list to five articles for the meta-analysis. However, such conventional methods of narrowing down the literature subject to meta-analysis are cumbersome and time-consuming.

[0008] Therefore, it is conceivable to approach the narrowing down of the literature subject to meta-analysis as a literature classification problem using machine learning. To build a machine learning model (classification model) for solving such literature classification problems, the literature actually cited by the meta-analysis literature is used as training data. For this reason, it is necessary to collect a set of literature that is actually the subject of meta-analysis. Note that the meta-analysis literature itself can be easily identified by searching, etc. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2017-173953 A [Patent Document 2] Special Publication No. 2009-520278 [Patent Document 3] US Patent Application Publication No. 2018 / 0373754 Summary of the Invention [Problem to be solved by the invention]

[0010] However, the literature cited in meta-analysis papers includes literature that is not the subject of the meta-analysis. For example, citations to describe experimental methods or known facts. The accuracy of the machine learning model may be reduced if literature other than the literature subject of the meta-analysis is included in the training data. Furthermore, identifying the literature subject to the meta-analysis from among the literature cited by the meta-analysis paper is complicated, as shown in steps 1 to 3 above. In one aspect, the present invention aims to easily generate training data for a machine learning model for identifying literature to be included in a meta-analysis. [Means for solving the problem]

[0011] For this reason, the training data generation program identifies a plurality of meta-analysis documents that cite the first document, and generates a training data set for each of the plurality of meta-analysis documents based on characteristic information of each of the identified plurality of meta-analysis documents. Based on the variation of the characteristic information The similarity is determined, and if the similarity is equal to or greater than a threshold, the computer is caused to execute a process of generating training data for machine learning that includes the first document. Effect of the Invention

[0012] According to one embodiment, training data for a machine learning model for identifying literature to be included in a meta-analysis can be easily generated. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram for explaining the concept of a method for identifying documents subject to meta-analysis in the information processing device according to the first and second embodiments. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device as an example of a first embodiment. [Diagram 3] 4 is a diagram illustrating document vectors calculated by a document vector calculation unit in the information processing device as an example of the first embodiment. FIG. [Figure 4] 2 is a diagram illustrating a schematic diagram of cited documents acquired by a cited document acquisition unit in the information processing device as an example of the first embodiment. FIG. [Diagram 5] 4 is a diagram illustrating an example of an average distance calculated by an average distance calculation unit in the information processing device as an example of the first embodiment; FIG. [Figure 6] FIG. 2 is a diagram for explaining a process performed by a meta-analysis target document identification unit in the information processing device as an example of the first embodiment. [Figure 7] 1 is a flowchart for explaining a process of determining documents subject to meta-analysis in an information processing device as an example of the first embodiment. [Figure 8] FIG. 11 is a diagram illustrating a functional configuration of an information processing device as an example of a second embodiment. [Figure 9] FIG. 13 is a diagram illustrating meta-analysis documents classified by a clustering processing unit in the information processing device as an example of the second embodiment. [Figure 10] 13 is a diagram illustrating a schematic diagram of cited documents acquired by a cited document acquisition unit in an information processing device as an example of a second embodiment. FIG. [Figure 11] 13 is a diagram illustrating an example of a class list calculated by a class list creation unit in an information processing device as an example of a second embodiment; FIG. [Figure 12] FIG. 13 is a diagram for explaining a process performed by a meta-analysis target document identification unit in the information processing device as an example of the second embodiment. [Figure 13] 13 is a flowchart for explaining a process of determining documents subject to meta-analysis in an information processing device as an example of the second embodiment. [Figure 14] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device as an example of the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments of the present training data generation program, training data generation method, and training data generation device will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention to exclude the application of various modified examples and techniques not explicitly stated in the embodiments. In other words, this embodiment can be implemented with various modifications (such as combining the embodiment and each modified example) without departing from the spirit of the present invention. Furthermore, each figure is not intended to include only the components shown in the figure, but can include other functions, etc.

[0015] (I) Basic principle FIG. 1 is a diagram for explaining the basic concept of a method for identifying documents subject to meta-analysis in information processing devices 1a and 1b (see FIGS. 2 and 8) according to the first and second embodiments described below.

[0016] FIG. 1 shows multiple (four in the example shown in FIG. 1) meta-analysis documents A1 to A4 and multiple (six in the example shown in FIG. 1) documents B1 to B6 cited by these meta-analysis documents A1 to A4. Documents B1 to B6 correspond to first documents cited by meta-analysis documents A1 to A4. Hereinafter, documents cited by meta-analysis documents may be referred to as cited documents. In the example shown in FIG. 1, documents B1 to B6 are each cited documents. Furthermore, documents that cite documents (cited documents) may be referred to as cited documents or cited source documents. For example, in the example shown in FIG. 1, meta-analysis document A3, which cites cited documents B2 and B6, corresponds to the cited source document of cited documents B2 and B6.

[0017] In the example shown in Figure 1, meta-analysis documents A1 and A2 are both meta-analysis documents on cancer-related topics, with meta-analysis document A1 citing references B1, B2, and B4, and meta-analysis document A2 citing references B1 to B3, respectively.

[0018] Additionally, meta-analysis reference A3 is a meta-analysis reference on cerebral hemorrhage-related topics, and cites references B2 and B6. Meta-analysis reference A4 is a meta-analysis reference on gastric ulcer-related topics, and cites references B2, B4, and B5. Generally, the literature included in a meta-analysis is cited from meta-analysis literature focusing on specific drugs or side effects.

[0019] In the example shown in Figure 1, for example, cited reference B1 is only cited by meta-analysis references A1 and A2 on cancer-related topics, so it is considered to be a meta-analysis target reference. Similarly, cited references B3, B5, and B6 are all cited by one meta-analysis reference, so they are all considered to be meta-analysis target references.

[0020] On the other hand, if the source literature covers a wide range of topics, it can be considered a citation to describe experimental methods or known facts, and not a citation for a document to be used in a meta-analysis.

[0021] In the example shown in Figure 1, cited reference B2 is cited by meta-analysis references A1 and A2 on cancer-related topics, meta-analysis reference A3 on cerebral hemorrhage-related topics, and meta-analysis reference A4 on gastric ulcer-related topics. As such, cited reference B2 is not considered to be cited as a meta-analysis subject (a non-meta-analysis subject) because the themes of the meta-analysis references from which it is cited are diverse.

[0022] In the information processing devices 1a and 1b shown in each of the following embodiments, the correlation of themes is determined based on the similarity of the cited meta-analysis documents, thereby determining whether or not a document cited from the cited meta-analysis document is a document subject to meta-analysis.

[0023] (II) Description of the First Embodiment In the information processing device 1a of the first embodiment, it is determined whether or not a document cited by a citing meta-analysis document is a target document of the meta-analysis by using the document vector.

[0024] (A) Configuration FIG. 2 is a diagram illustrating a schematic functional configuration of an information processing device 1a as an example of the first embodiment.

[0025] The information processing device 1a realizes a training data generation function for generating training data used in creating a machine learning model (classification model) for extracting documents cited in meta-analysis documents.

[0026] As shown in FIG. 2, the information processing device 1a of the first embodiment includes a document vector calculation unit 101, a cited document acquisition unit 102, an average distance calculation unit 103, a meta-analysis target document identification unit 104, a training data generation unit 105, and a training unit 106.

[0027] In the information processing device 1a, a plurality of meta-analysis documents collected in advance are used. Information on these meta-analysis documents is stored in a specific storage area such as the memory 12 or the storage device 13, which will be described later with reference to FIG.

[0028] The document vector calculation unit 101 calculates a document vector for each of a plurality of meta-analysis documents. A document vector is information generated by vectorizing a document, and is a document vector generated by converting sentences constituting the document into vectors. A document vector corresponds to feature information that represents the features of a meta-analysis document. A document vector may also be called a sentence vector.

[0029] The document vector calculation unit 101 may generate document vectors from the abstracts of documents by using a known method such as doc2vec. Alternatively, multiple keywords may be extracted from the meta-analysis document, and the document vector may be calculated by averaging the word vectors corresponding to these keywords. The document vector calculation unit 101 is not limited to these methods, and may calculate document vectors of meta-analysis documents by using various known methods. The values ​​of the document vectors calculated by the document vector calculation unit 101 are stored in a specific storage area such as the memory 12 or the storage device 13.

[0030] FIG. 3 is a diagram illustrating document vectors calculated by the document vector calculation unit 101 in the information processing device 1a as an example of the first embodiment. The example shown in FIG. 3 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0031] 3 shows document vectors calculated for each of the four meta-analysis documents A1 to A4. For example, the document vector of meta-analysis document A1 is (1.0, 1.0, 0.5).

[0032] The cited document acquisition unit 102 acquires documents cited by each meta-analysis document (cited documents). To acquire such cited documents, for example, a known document database such as PubMed (registered trademark) may be used. The cited documents acquired by the cited document acquisition unit 102 are stored in a specific storage area such as the memory 12 or the storage device 13. FIG. 4 is a diagram illustrating the cited documents acquired by the cited document acquisition unit 102 in the information processing device 1a as an example of the first embodiment.

[0033] The example shown in Fig. 4 corresponds to the meta-analysis documents A1 to A4 and documents B1 to B6 illustrated in Fig. 1. For example, cited document B1 is cited by each of meta-analysis documents A1 and A2, and cited document B2 is cited by each of meta-analysis documents A1 to A4.

[0034] The average distance calculation unit 103 calculates, for each cited document acquired by the cited document acquisition unit 102, the average distance from the center of gravity of the document vector corresponding to the meta-analysis document that cited the document.

[0035] That is, the average distance calculation unit 103 first calculates the center of gravity of the document vector corresponding to each cited meta-analysis document, calculates the distance from this center of gravity to the document vector corresponding to each meta-analysis document, and further calculates the average value (average distance) of the distances from these centers of gravity for each cited document. In addition, when there is only one meta-analysis document cited, the document vector and the center of gravity coincide, so the average distance is 0. FIG. 5 is a diagram illustrating an example of the average distance calculated by the average distance calculation unit 103 in the information processing device 1a as an example of the first embodiment. The example shown in FIG. 5 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0036] In FIG. 5, for example, with respect to cited document B1, the average distance calculation unit 103 obtains the center of gravity of each document vector of meta-analysis documents A1 and A2 (citing meta-analysis documents) that cite this cited document B1. The center of gravity G of each document vector of meta-analysis documents A1 and A2 (A1,A2) can be expressed by the following formula: Center of gravity G (A1,A2) = ((1.0+0.9) / 2, (1.0+0.8) / 2, (0.5+0.6) / 2) = (0.95, 0.9, 0.55)

[0037] Next, the average distance calculation unit 103 calculates the center of gravity G(A1,A2) Calculate the average distance from the center of gravity G to each document vector of meta-analysis documents A1 and A2. (A1,A2) The average distance from to each document vector of meta-analysis documents A1 and A2 is calculated using the following formula.

[0038]

number

[0039] Hereinafter, the average value of the distance from the center of gravity of the document vector corresponding to the meta-analysis document cited in the cited document to the document vector of each meta-analysis document may be referred to as the vector average distance. The vector average distance represents the similarity between multiple meta-analysis documents, and a smaller vector average distance represents a higher similarity, i.e., a higher degree of similarity.

[0040] In a similar manner, the average distance calculation unit 103 calculates the average distance (vector average distance) from the center of gravity of the document vectors corresponding to the citing meta-analysis documents for each of the documents B1 to B6. The vector average distance for each cited document calculated by the average distance calculation unit 103 is stored in a specific storage area such as the memory 12 or the storage device 13.

[0041] The meta-analysis target document identification unit 104 identifies (determines) whether the cited document to be identified is a document to be subjected to meta-analysis or a document not to be subjected to meta-analysis based on the vector average distance calculated by the average distance calculation unit 103 for the cited document to be identified.

[0042] As described above, a large vector average distance calculated by the average distance calculation unit 103 indicates that the document vectors of the citing meta-analysis document have a large variation with respect to the cited document. In other words, it can be determined that the themes of the citing meta-analysis document have a large variation. In this way, a large variation in the themes of the citing meta-analysis document with respect to the cited document means that the themes of the citing meta-analysis document are diverse, and therefore it is considered that the citation is not a document subject to meta-analysis (a document not subject to meta-analysis).

[0043] On the other hand, a small vector average distance calculated by the average distance calculation unit 103 indicates that the variation in document vectors of the citing meta-analysis document is small with respect to the cited document. In other words, it can be determined that the variation in themes of the citing meta-analysis document is small. In this way, the small variation in themes of the citing meta-analysis document with respect to the cited document means that the themes of the citing meta-analysis document are not diverse, and therefore it is considered to be a citation as a meta-analysis target document (meta-analysis target document).

[0044] Therefore, the meta-analysis target document identification unit 104 compares the vector average distance value (vector average value) calculated for the cited document with a threshold value, and if the vector average value is equal to or less than the threshold value, determines the cited document as a document to be subjected to meta-analysis, whereas if the vector average value is greater than the threshold value, determines the cited document as a document not to be subjected to meta-analysis. In the first embodiment, the reciprocal of the vector average value corresponds to the similarity, and the vector average value being equal to or less than the threshold value corresponds to the similarity of the meta-analysis documents being equal to or more than the threshold value.

[0045] The meta-analysis target document identification unit 104 registers information about cited documents determined to be subject to meta-analysis in a list of documents subject to meta-analysis, and registers information about cited documents determined to be not subject to meta-analysis in a list of documents not subject to meta-analysis.

[0046] The threshold value used for comparison with the vector average value may be set, for example, according to an empirical rule based on one or more specific examples, and may be modified as appropriate.

[0047] FIG. 6 is a diagram for explaining the process performed by the meta-analysis target document identification unit 104 in the information processing device 1a as an example of the first embodiment. The example shown in FIG. 6 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0048] FIG. 6 shows an example in which the meta-analysis target document identifying unit 104 identifies documents that are subject to meta-analysis and documents that are not subject to meta-analysis by comparing the vector average value calculated for the cited documents with a threshold value of 0.3.

[0049] That is, the meta-analysis target document identification unit 104 determines that the cited document B2, whose vector average value is 0.57, and the cited document B4, whose vector average value is 0.53, are not to be meta-analyzed. The meta-analysis target document identification unit 104 also determines that the cited documents B1, B3, B5, and B6, whose vector average values ​​are 0.3 or less, are to be meta-analysis target documents.

[0050] The determination result by the meta-analysis target document identifying unit 104 (list of documents to be subjected to meta-analysis / list of documents not to be subjected to meta-analysis) is stored in a specific storage area such as the memory 12 or the storage device 13.

[0051] The document vector calculation unit 101, cited document acquisition unit 102, average distance calculation unit 103, and meta-analysis target document specification unit 104 function as a meta-analysis target document determination unit that determines the meta-analysis target documents.

[0052] The training data generation unit 105 uses the determination results by the meta-analysis target document identification unit 104 to generate training data to be used in creating a machine learning model (classification model) for extracting documents cited in the meta-analysis documents.

[0053] For example, the training data generating unit 105 generates training data by combining, as correct answer data, information indicating that the cited document is a meta-analysis target document with the cited document determined to be a meta-analysis target document by the meta-analysis target document identifying unit 104. The training data generating unit 105 generates training data including documents whose vector average value is equal to or less than a threshold, i.e., whose similarity to the meta-analysis document is equal to or more than a threshold, as meta-analysis target documents. The training data generated by the training data generating unit 105 is stored in a specific storage area such as the memory 12 or the storage device 13.

[0054] The training unit 106 executes training of a machine learning model for extracting documents cited in meta-analysis documents, using the training data generated by the training data generation unit 105. The machine learning model determines whether a document is a subject of meta-analysis (a document subject to meta-analysis). Information about the machine learning model trained by the training unit 106 (for example, various parameter values) is stored in a specific storage area such as the memory 12 or the storage device 13.

[0055] (B) Operation A process for determining documents to be subjected to meta-analysis in the information processing device 1a as an example of the first embodiment configured as described above will be described with reference to the flowchart (steps S01 to S08) shown in FIG. In step S01, the document vector calculation unit 101 calculates document vectors for each of a plurality of meta-analysis documents.

[0056] In step S02, the cited document acquisition unit 102 acquires cited documents cited by each meta-analysis document. All of the acquired cited documents are treated as documents to be determined as to whether they are subject to meta-analysis.

[0057] In step S03, the average distance calculation unit 103 calculates, for each cited document acquired by the cited document acquisition unit 102, the average distance (vector average distance) from the center of gravity of the document vector corresponding to the meta-analysis document that cited the document to the document vector of each meta-analysis document. In step S04, the meta-analysis target document identifying unit 104 selects one cited document from the one or more cited documents to be identified.

[0058] In step S05, the meta-analysis target document identification unit 104 compares the value of the vector average distance corresponding to the cited document with a threshold value. If the comparison result shows that the vector average distance is equal to or less than the threshold value (see the YES route in step S05), the process proceeds to step S07.

[0059] In step S07, the meta-analysis target document identifying unit 104 determines that the cited document is a meta-analysis target document, and registers the cited document in a list of meta-analysis target documents. On the other hand, if it is determined in step S05 that the vector average distance is greater than the threshold value (see NO route in step S05), the process proceeds to step S06.

[0060] In step S06, the meta-analysis target document identifying section 104 determines that the cited document is a document not subject to meta-analysis, and registers the cited document in a list of documents not subject to meta-analysis.

[0061] Thereafter, in step S08, the meta-analysis target document identifying unit 104 checks whether the processes in steps S05 to S07 have been executed for all of the cited documents to be identified, that is, whether the documents have been identified as being subject to meta-analysis.

[0062] If the result of the check is that discrimination (determination) has not been performed for all of the cited documents to be discriminated (see the NO route in step S08), the process returns to step S04. In step S04, from among the cited documents to be discriminated, cited documents that have not yet been discriminated as to whether they are subject to meta-analysis are selected. On the other hand, if the discrimination (determination) has been performed for all the cited documents to be discriminated (see the YES route in step S08), the process ends.

[0063] Moreover, the training data generating unit 105 generates training data by combining, as correct answer data, information indicating that a cited document is a meta-analysis target document with the cited document determined to be a meta-analysis target document by the meta-analysis target document identifying unit 104. The training unit 106 trains the machine learning model using the generated training data.

[0064] (C) Effects Thus, according to the information processing device 1a as an example of the first embodiment, the document vector calculation unit 101 calculates the document vectors of each of the meta-analysis documents. In addition, the average distance calculation unit 103 obtains the center of gravity of the document vectors corresponding to the meta-analysis documents cited by each of the cited documents. Then, the average distance calculation unit 103 calculates the average of the distances from the center of gravity to the document vectors corresponding to each of the meta-analysis documents (vector average distance).

[0065] Then, the meta-analysis target document identification unit 104 compares the vector average value calculated for the cited document with a threshold value, and if the vector average value is equal to or less than the threshold value, it determines the cited document as a document to be subjected to meta-analysis, whereas if the vector average value is greater than the threshold value, it determines the cited document as a document not to be subjected to meta-analysis.

[0066] In other words, by comparing the vector average value with a threshold, it is possible to easily identify documents that are subject to meta-analysis and documents that are not subject to meta-analysis, and it is also possible to easily generate training data to be used in training (machine learning) a machine learning model (classification model) for extracting documents cited in meta-analysis documents.

[0067] In this case, a large vector average distance can be determined to mean that the document vectors of the meta-analysis document cited from the cited document vary widely, and that the themes of the meta-analysis document cited from the cited document vary widely. A large variation in the themes of the meta-analysis document cited from the cited document means that the themes of the meta-analysis document cited from the cited document are diverse, and the cited document can be determined to be a non-target document for meta-analysis.

[0068] On the other hand, a small vector average distance can be determined to mean that the document vectors of the meta-analysis document cited from the cited document have a small variation, and that the themes of the meta-analysis document cited from the cited document have a small variation. A small variation in the themes of the meta-analysis document cited from the cited document means that the themes of the meta-analysis document cited from the cited document are not diverse, and the cited document can be determined to be a meta-analysis target document.

[0069] In this manner, the information processing device 1a of the first embodiment can accurately distinguish between documents not subject to meta-analysis and documents subject to meta-analysis, and can generate high-quality training data.

[0070] (III) Description of the Second Embodiment In the information processing device 1b of the second embodiment, clustering is used to determine whether or not a document cited by a citing meta-analysis document is a target document of the meta-analysis.

[0071] (A) Configuration FIG. 8 is a diagram illustrating a schematic functional configuration of an information processing device 1b as an example of the second embodiment. Like the information processing device 1a of the first embodiment, the information processing device 1b also realizes a training data generation function that generates training data used to create a machine learning model (classification model) for extracting documents cited in meta-analysis documents.

[0072] As shown in FIG. 8, an information processing apparatus 1b according to the second embodiment includes a clustering processing unit 201, a cited document acquiring unit 202, a class list creating unit 203, a meta-analysis subject document identifying unit 204, a training data generating unit 205, and a training unit 206.

[0073] In the information processing device 1b, a plurality of meta-analysis papers collected in advance are used. Information on these meta-analysis papers is stored in a specific storage area such as the memory 12 or the storage device 13. The clustering processor 201 applies clustering to a plurality of meta-analysis documents to determine the class of each meta-analysis document.

[0074] The clustering processing unit 201 may set classes for the meta-analysis documents using, for example, the k-means method. Note that the clustering method used by the clustering processing unit 201 is not limited to the k-means method, and can be implemented in various modified forms. The classes of meta-analysis documents set by clustering the meta-analysis documents correspond to characteristic information that represents the characteristics of the meta-analysis documents.

[0075] FIG. 9 is a diagram showing meta-analysis documents classified by the clustering processing unit 201 in the information processing device 1b as an example of the second embodiment. The example shown in FIG. 9 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0076] In the example shown in Figure 9, each of the four meta-analysis documents A1 to A4 is clustered, with meta-analysis documents A1 and A2 assigned to class 1, meta-analysis document A3 assigned to class 2, and meta-analysis document A4 assigned to class 3.

[0077] The cited document acquisition unit 202 acquires documents cited by each meta-analysis document (cited documents). To acquire such cited documents, for example, a known document database such as PubMed may be used, as in the first embodiment. The cited documents acquired by the cited document acquisition unit 202 are stored in a specific storage area such as the memory 12 or the storage device 13. FIG. 10 is a diagram showing a schematic diagram of cited documents acquired by the cited document acquisition unit 202 in the information processing device 1b as an example of the second embodiment.

[0078] The example shown in Fig. 10 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 illustrated in Fig. 1. For example, cited document B1 is cited by each of meta-analysis documents A1 and A2, and cited document B2 is cited by each of meta-analysis documents A1 to A4. The class list creation unit 203 creates, for each cited document acquired by the cited document acquisition unit 102, a list of classes corresponding to the meta-analysis documents that cited the cited documents.

[0079] That is, for each cited document, the class list creation unit 203 obtains and lists each class of one or more meta-analysis documents that cite the cited document. Hereinafter, the list of classes may be referred to as a class list. The class list may be represented, for example, by listing one or more classes that apply. FIG. 11 is a diagram illustrating a class list calculated by the class list creating unit 203 in an information processing device 1b as an example of the second embodiment. The example shown in FIG. 11 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0080] 11, for example, for cited document B1, the class list creation unit 203 obtains the classes of meta-analysis documents A1 and A2 that cite this cited document B1 (source meta-analysis documents). Since these meta-analysis documents A1 and A2 are both class 1, the class list creation unit 203 creates a class list [class 1, class 1] for cited document B1.

[0081] Furthermore, for example, with respect to cited document B2, the class list creation unit 203 acquires each class of meta-analysis documents (citing meta-analysis documents) A1 to A4 that cite this cited document B2. Both meta-analysis documents A1 and A2 are in class 1, meta-analysis document A3 is in class 2, and meta-analysis document A4 is in class 3. Therefore, the class list creation unit 203 creates [class 1, class 1, class 2, class 3] as the class list for cited document B2.

[0082] Furthermore, the class list creation unit 203 calculates the variation of the class list for each cited document. For example, the class list creation unit 203 may obtain the variation by calculating the average information amount. For example, for the class list [class1, class1], when randomly selecting from the class list, Probability of being in class 1: 1 Therefore, the variation can be calculated using the following formula:

[0083]

number

[0084]

number

[0085]

number

[0086] The variance represents the similarity between the multiple meta-analysis documents (A1 to A4). The calculation of the variance is not limited to the method using the average information amount described above, and can be modified as appropriate. For example, the class list creation unit 203 can calculate the variance of the class list using variance or standard deviation, and can be modified in various ways. The class list generated by the class list generating unit 203 and the calculated values ​​of variation are stored in a specific storage area such as the memory 12 or the storage device 13.

[0087] The meta-analysis target document identifying unit 204 identifies (determines) whether the cited document to be identified is a document to be subjected to meta-analysis or a document not to be subjected to meta-analysis based on the value of variation calculated by the class list creating unit 203 for the cited document to be identified.

[0088] As described above, when the variation calculated by the class list creation unit 203 is large, it can be determined that the variation in the themes of the cited meta-analysis documents is large. In this way, when the variation in the themes of the cited meta-analysis documents is large compared to the cited documents, it means that the themes of the cited meta-analysis documents are diverse, and therefore it is considered that the citation is not a document subject to meta-analysis (a document not subject to meta-analysis).

[0089] On the other hand, when the variation calculated by the class list creation unit 203 is small, it can be determined that the variation in the themes of the cited meta-analysis documents is small. In this way, when the variation in the themes of the cited meta-analysis documents is small compared to the cited documents, it means that the themes of the cited meta-analysis documents are not diverse, and therefore the cited documents are considered to be subject documents of meta-analysis (subject documents of meta-analysis).

[0090] Therefore, the meta-analysis target document identification unit 204 compares the calculated dispersion value for the cited document with a threshold value, and if the dispersion value is equal to or less than the threshold value, it determines the cited document as a document to be subjected to the meta-analysis. On the other hand, if the dispersion value is greater than the threshold value, it determines the cited document as a document not to be subjected to the meta-analysis. In the second embodiment, the reciprocal of the dispersion value corresponds to the similarity, and the dispersion value being equal to or less than the threshold value corresponds to the similarity of the meta-analysis document being equal to or more than the threshold value.

[0091] The meta-analysis target document identification unit 204 registers information about cited documents determined to be subject to meta-analysis in a list of documents subject to meta-analysis, and registers information about cited documents determined to be not subject to meta-analysis in a list of documents not subject to meta-analysis. The threshold value used for comparison with the variation value may be set according to an empirical rule based on one or more specific examples, and may be modified as appropriate.

[0092] FIG. 12 is a diagram for explaining the process performed by the meta-analysis target document identification unit 204 in the information processing device 1b as an example of the second embodiment. The example shown in FIG. 12 corresponds to meta-analysis documents A1 to A4 and documents B1 to B6 shown in FIG.

[0093] FIG. 12 shows an example in which the meta-analysis target document identification unit 204 identifies documents that are subject to meta-analysis and documents that are not subject to meta-analysis by comparing the value of the variation in the class list calculated for the cited document with a threshold value of 0.7.

[0094] That is, the meta-analysis target document identification unit 204 determines that the cited document B2, whose variation value is 1.5, and the cited document B4, whose variation value is 1.0, are not subject to meta-analysis. The meta-analysis target document identification unit 204 also determines that the cited documents B1, B3, B5, and B6, whose variation values ​​are 0.7 or less, are subject to meta-analysis.

[0095] The determination result by the meta-analysis target document identifying unit 204 (list of documents to be subjected to meta-analysis / list of documents not to be subjected to meta-analysis) is stored in a specific storage area such as the memory 12 or the storage device 13.

[0096] The above-mentioned clustering processing unit 201, cited document acquisition unit 202, class list creation unit 203, and meta-analysis target document identification unit 204 function as a meta-analysis target document determination unit that determines the meta-analysis target documents.

[0097] The training data generation unit 205 uses the determination results by the meta-analysis target document identification unit 204 to generate training data to be used in creating a machine learning model (classification model) for extracting documents cited in the meta-analysis documents.

[0098] For example, the training data generating unit 205 generates training data by combining, as correct answer data, information indicating that the cited document is a meta-analysis target document with the cited document determined to be a meta-analysis target document by the meta-analysis target document identifying unit 204. The training data generating unit 205 generates training data including documents whose variation value is equal to or less than a threshold, i.e., whose similarity to the meta-analysis document is equal to or more than a threshold, as meta-analysis target documents. The training data generated by the training data generating unit 205 is stored in a specific storage area such as the memory 12 or the storage device 13.

[0099] The training unit 206 executes training of a machine learning model for extracting documents cited in meta-analysis documents, using the training data generated by the training data generation unit 205. The machine learning model judges whether a document is a subject of meta-analysis (a document subject to meta-analysis). Information about the machine learning model trained by the training unit 206 (for example, various parameter values) is stored in a specific storage area such as the memory 12 or the storage device 13.

[0100] (B) Operation A process for determining documents to be subjected to meta-analysis in the information processing device 1b as an example of the second embodiment configured as described above will be described with reference to the flowchart (steps S11 to S18) shown in FIG.

[0101] In step S11, the clustering processing unit 201 applies clustering to a plurality of meta-analysis documents to obtain a class for each meta-analysis document.

[0102] In step S12, the cited document acquisition unit 202 acquires cited documents cited by each meta-analysis document. All of the acquired cited documents are treated as documents to be determined as to whether they are subject to meta-analysis.

[0103] In step S13, the class list creation unit 203 creates a class list of classes corresponding to the meta-analysis documents that cite each cited document acquired by the cited document acquisition unit 202. Furthermore, the class list creation unit 203 calculates the variance of the class list for each cited document. In step S14, the meta-analysis target document identifying section 204 selects one cited document from the one or more cited documents to be identified.

[0104] In step S15, the meta-analysis target document identification unit 204 compares the variability value corresponding to the cited document with a threshold value. If the comparison result shows that the variability value is equal to or smaller than the threshold value (see YES route in step S15), the process proceeds to step S17.

[0105] In step S17, the meta-analysis target document identifying section 204 determines that the cited document is a meta-analysis target document, and registers the cited document in a list of meta-analysis target documents. On the other hand, if the result of the check in step S15 is that the value of the variation is greater than the threshold value (see NO route in step S15), the process proceeds to step S16.

[0106] In step S16, the meta-analysis target document identifying section 204 determines that the cited document is a document not subject to meta-analysis, and registers the cited document in a list of documents not subject to meta-analysis.

[0107] Thereafter, in step S18, the meta-analysis target document identification unit 204 checks whether the processes in steps S15 to S17 have been executed for all of the cited documents to be identified, that is, whether the documents have been identified as being meta-analysis target documents.

[0108] If the result of the check is that discrimination (determination) has not been performed for all of the cited documents to be discriminated (see the NO route from step S18), the process returns to step S14. In step S14, from among the cited documents to be discriminated, cited documents that have not yet been discriminated as to whether they are subject to meta-analysis are selected. On the other hand, if the discrimination (determination) has been performed for all the cited documents to be discriminated (see the YES route of step S18), the process ends.

[0109] The training data generation unit 105 generates training data by combining, as correct answer data, information indicating that a cited document is a meta-analysis target document with the cited document determined to be a meta-analysis target document by the meta-analysis target document identification unit 204. The training unit 206 trains the machine learning model using the generated training data.

[0110] (C) Effects In this way, according to the information processing device 1b as an example of the second embodiment, the clustering processing unit 201 applies clustering to each of the multiple meta-analysis documents to obtain a class for each meta-analysis document. In addition, the class list creation unit 203 creates a class list of classes corresponding to the meta-analysis documents that cited each of the documents cited by each meta-analysis document, and further calculates the variance of the cluster list.

[0111] Then, the meta-analysis target document identification unit 204 compares the calculated dispersion value for the cited document with a threshold value, and if the dispersion value is equal to or less than the threshold value, the cited document is determined to be a document to be subjected to meta-analysis, whereas if the dispersion value is greater than the threshold value, the cited document is determined to be a document not to be subjected to meta-analysis.

[0112] In other words, by comparing the variation value of the class list with a threshold, it is possible to easily identify the documents that are subject to the meta-analysis and the documents that are not subject to the meta-analysis, and it is also possible to easily generate training data used to create a machine learning model (classification model) for extracting documents cited in the meta-analysis documents.

[0113] In this case, a large variation can be judged to mean that the themes of the meta-analysis documents cited from the cited documents vary widely compared to the cited documents. A large variation in the themes of the meta-analysis documents cited from the cited documents means that the themes of the meta-analysis documents cited from the cited documents are diverse, and the cited documents can be judged to be non-target documents for meta-analysis.

[0114] On the other hand, a small variation can be judged to mean that the theme of the meta-analysis document cited from the cited document is small compared to the cited document. A small variation in the theme of the meta-analysis document cited from the cited document means that the theme of the meta-analysis document cited from the cited document is not diverse, and the cited document can be judged to be a meta-analysis target document.

[0115] In this way, the information processing device 1b of the second embodiment can also accurately distinguish between documents not subject to meta-analysis and documents subject to meta-analysis, and generate high-quality training data.

[0116] (IV) Other FIG. 14 is a diagram illustrating a hardware configuration of an information processing device 1a as an example of the first embodiment and an information processing device 1b as an example of the second embodiment. The information processing devices 1a and 1b have the same hardware configuration. The information processing devices 1a and 1b are computers, and include, for example, a processor 11, a memory 12, a storage device 13, a graphics processing device 14, an input interface 15, an optical drive device 16, a device connection interface 17, and a network interface 18 as components. These components 11 to 18 are configured to be able to communicate with each other via a bus 19.

[0117] The processor 11 is a control unit that controls the entire information processing devices 1a and 1b. The processor 11 may be a multiprocessor. The processor 11 may be, for example, any one of a CPU, a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). The processor 11 may also be a combination of two or more types of elements among the CPU, MPU, DSP, ASIC, PLD, and FPGA.

[0118] Then, when the processor 11 executes a control program (training data generation program, not shown) for the information processing device 1a, the functions of a document vector calculation unit 101, a cited document acquisition unit 102, an average distance calculation unit 103, a meta-analysis target document identification unit 104, a training data generation unit 105 and a training unit 106 illustrated in FIG. 2 are realized.

[0119] In addition, by the processor 11 executing a control program (training data generation program, not shown) for the information processing device 1b, the functions of a clustering processing unit 201, a cited document acquisition unit 202, a class list creation unit 203, a meta-analysis target document identification unit 204, a training data generation unit 205 and a training unit 206 illustrated in FIG. 8 are realized.

[0120] The information processing device 1a realizes the functions of a document vector calculation unit 101, a cited document acquisition unit 102, an average distance calculation unit 103, a meta-analysis target document identification unit 104, a training data generation unit 105 and a training unit 106, for example, by executing a program (training data generation program, OS program) recorded on a computer-readable non-transitory recording medium.

[0121] In addition, the information processing device 1b realizes the functions of a clustering processing unit 201, a cited document acquisition unit 202, a class list creation unit 203, a meta-analysis target document identification unit 204, a training data generation unit 205, and a training unit 206, for example, by executing programs (training data generation program, OS program) recorded on a computer-readable non-transitory recording medium.

[0122] The programs describing the processing contents to be executed by the information processing devices 1a and 1b can be recorded on various recording media. For example, the programs to be executed by the information processing devices 1a and 1b can be stored in the storage device 13. The processor 11 loads at least a part of the programs in the storage device 13 into the memory 12 and executes the loaded programs.

[0123] The programs to be executed by the information processing devices 1a and 1b (processor 11) may also be recorded on a non-transitory portable recording medium such as an optical disk 16a, a memory device 17a, or a memory card 17c. The programs stored on the portable recording medium are installed in the storage device 13 under the control of the processor 11, for example, and then become executable. The processor 11 may also directly read out and execute the programs from the portable recording medium.

[0124] The memory 12 is a storage memory including a ROM (Read Only Memory) and a RAM (Random Access Memory). The RAM of the memory 12 is used as a main storage device of the information processing devices 1a and 1b. The RAM temporarily stores at least a part of the program to be executed by the processor 11. The memory 12 also stores various data necessary for processing by the processor 11.

[0125] The storage device 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), etc., and stores various data. The storage device 13 is used as an auxiliary storage device for the information processing devices 1a and 1b. An OS program, a control program, and various data are stored in the storage device 13. The control program includes a training data generation program.

[0126] As the auxiliary storage device, a semiconductor storage device such as an SCM or a flash memory can also be used. Furthermore, a plurality of storage devices 13 may be used to configure a RAID (Redundant Arrays of Inexpensive Disks).

[0127] A monitor 14a is connected to the graphics processing device 14. The graphics processing device 14 displays an image on the screen of the monitor 14a in accordance with an instruction from the processor 11. Examples of the monitor 14a include a display device using a CRT (Cathode Ray Tube) and a liquid crystal display device.

[0128] A keyboard 15a and a mouse 15b are connected to the input interface 15. The input interface 15 transmits signals sent from the keyboard 15a and the mouse 15b to the processor 11. The mouse 15b is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0129] The optical drive device 16 uses a laser beam or the like to read data recorded on an optical disk 16a. The optical disk 16a is a portable, non-transient recording medium on which data is recorded so that it can be read by the reflection of light. Examples of the optical disk 16a include a DVD (Digital Versatile Disk), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable) / RW (ReWritable).

[0130] The device connection interface 17 is a communication interface for connecting peripheral devices to the information processing devices 1a and 1b. For example, a memory device 17a or a memory reader / writer 17b can be connected to the device connection interface 17. The memory device 17a is a non-transient recording medium equipped with a communication function with the device connection interface 17, such as a Universal Serial Bus (USB) memory. The memory reader / writer 17b writes data to a memory card 17c or reads data from the memory card 17c. The memory card 17c is a card-type non-transient recording medium.

[0131] The network interface 18 is connected to a network. The network interface 18 transmits and receives data via the network. Other information processing devices, communication devices, etc. may be connected to the network.

[0132] The disclosed technology is not limited to the above-described embodiment, and may be modified in various ways without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment may be selected as necessary, or may be combined as appropriate.

[0133] For example, in the second embodiment described above, the clustering processing unit 201 sets classes for each meta-analysis document using a clustering method, but this is not limited to this. For example, a known classification method may be used to classify each meta-analysis document, and the method can be implemented with appropriate modifications.

[0134] In addition, in each of the above-mentioned embodiments, an example has been shown in which documents cited in meta-analysis documents are identified as documents subject to meta-analysis / documents not subject to meta-analysis, but this is not limited to this, and the present invention may also be applied to identifying documents cited in documents other than meta-analysis documents. Furthermore, the above disclosure enables a person skilled in the art to implement and manufacture the present embodiment.

[0135] (V) Notes Regarding the above embodiment, the following supplementary notes are further disclosed. (Appendix 1) Identify multiple meta-analyses that cited the first article; determining a similarity between the plurality of meta-analysis documents based on the characteristic information of each of the identified plurality of meta-analysis documents; If the degree of similarity is equal to or greater than a threshold, generating training data for machine learning including the first document. A training data generation program that causes a computer to execute processing.

[0136] (Appendix 2) the feature information is a document vector calculated based on each of the plurality of meta-analysis documents; the similarity is calculated based on a distance between a center of gravity of a document vector calculated based on each of the plurality of meta-analysis documents and a document vector calculated based on each of the plurality of meta-analysis documents; 2. The training data generation program according to claim 1,

[0137] (Appendix 3) The feature information is a class generated by clustering the plurality of meta-analysis documents; The similarity is calculated based on the variance of the generated classes. 2. The training data generation program according to claim 1,

[0138] (Appendix 4) identifying a plurality of meta-analysis documents that cite the first document; A process of determining a similarity between each of the identified meta-analysis documents based on characteristic information of each of the identified meta-analysis documents; generating training data for machine learning including the first document if the similarity is equal to or greater than a threshold; A training data generation method characterized in that the method is executed by a computer.

[0139] (Appendix 5) the feature information is a document vector calculated based on each of the plurality of meta-analysis documents; the similarity is calculated based on a distance between a center of gravity of a document vector calculated based on each of the plurality of meta-analysis documents and a document vector calculated based on each of the plurality of meta-analysis documents; 5. The training data generation method according to claim 4,

[0140] (Appendix 6) The feature information is a class generated by clustering the plurality of meta-analysis documents; The similarity is calculated based on the variance of the generated classes. 5. The training data generation method according to claim 4,

[0141] (Appendix 7) Identify multiple meta-analyses that cited the first article; determining a similarity between the plurality of meta-analysis documents based on the characteristic information of each of the identified plurality of meta-analysis documents; If the degree of similarity is equal to or greater than a threshold, generating training data for machine learning including the first document. A training data generating device including a control unit that executes processing.

[0142] (Appendix 8) the feature information is a document vector calculated based on each of the plurality of meta-analysis documents; the similarity is calculated based on a distance between a center of gravity of a document vector calculated based on each of the plurality of meta-analysis documents and a document vector calculated based on each of the plurality of meta-analysis documents; 8. The training data generating device according to claim 7,

[0143] (Appendix 9) The feature information is a class generated by clustering the plurality of meta-analysis documents; The similarity is calculated based on the variance of the generated classes. 8. The training data generating device according to claim 7, [Explanation of symbols]

[0144] 1a, 1b Information processing device 11 Processor (control unit) 12. Memory 13 Storage device 14 Graphics Processing Unit 14a Monitor 15 Input Interface 15a Keyboard 15b Mouse 16 Optical drive device 16a Optical disc 17 Device connection interface 17a Memory Device 17b Memory Reader / Writer 17c Memory Card 18 Network Interface 18a Network 19 Bus 101 Document Vector Calculation Unit 102 Cited document acquisition department 103 Average distance calculation section 104 Part Identifying Literature Subjects for Meta-Analysis 105 Training Data Generation Unit 106 Training Department 201 Clustering Processing Unit 202 Citation acquisition department 203 Class List Creation Department 204 Part Identifying Literature Subjects for Meta-Analysis 205 Training Data Generation Unit 206 Training Department

Claims

1. Identifying multiple meta-analysis articles that cite the first article; determining a similarity between the plurality of meta-analysis documents based on the characteristic information of each of the plurality of meta-analysis documents identified, the similarity being based on the variation of the characteristic information of each of the plurality of meta-analysis documents; If the degree of similarity is equal to or greater than a threshold, generating training data for machine learning including the first document. A training data generation program that causes a computer to execute processing.

2. the feature information is a document vector calculated based on each of the plurality of meta-analysis documents; the similarity is calculated based on a distance between a center of gravity of a document vector calculated based on each of the plurality of meta-analysis documents and a document vector calculated based on each of the plurality of meta-analysis documents; 2. The training data generating program according to claim 1 .

3. The feature information is a class generated by clustering the plurality of meta-analysis documents; The similarity is calculated based on the variance of the generated classes.

2. The training data generating program according to claim 1 .

4. identifying a plurality of meta-analysis articles that cite the first article; A process of determining a similarity based on a variance in the characteristic information of each of the identified meta-analysis documents based on the characteristic information of each of the identified meta-analysis documents; generating training data for machine learning including the first document if the similarity is equal to or greater than a threshold; A training data generation method characterized in that the method is executed by a computer.

5. Identifying multiple meta-analysis articles that cite the first article; determining a similarity between the plurality of meta-analysis documents based on the characteristic information of each of the plurality of meta-analysis documents identified, the similarity being based on the variation of the characteristic information of each of the plurality of meta-analysis documents; If the degree of similarity is equal to or greater than a threshold, generating training data for machine learning including the first document. A training data generating device including a control unit that executes processing.

Images