Output program, output method, and information processing device

Abstract

To improve predictive accuracy of a machine learning model.SOLUTION: A method causes a computer to execute the processing of: obtaining first information 102 by inputting a first document 101 into a machine learning model 11a, which has been trained to output information related to an input document in response to input of the input document; identifying a second document having a similar graph structure similar to a graph structure of the first document from a storage unit 11c for storing a plurality of documents 111; and outputting information 11d related to the first document based on second information 103 related to the identified second document, and the first information.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an output program, an output method, and an information processing device. [Background technology]

[0002] There is a known AI (Artificial Intelligence) task that extracts information about text, such as specialized knowledge, from a text. For example, the task is to input a text and multiple (e.g., two) words (target pairs, word pairs) into a Neural Network (NN) and predict the relationship between the target pairs (e.g., relationship labels) based on the context information of the text.

[0003] A NN that predicts relationships (hereinafter sometimes referred to as "relationship extraction") is a machine learning model trained by supervised learning using training data that includes, for example, text containing contextual information, target pairs, and relationship labels as ground truth data.

[0004] The accuracy of relationship extraction in documents depends on the context processing method for extracting context information from documents using a NN and the machine learning processing of the NN (e.g., training data).

[0005] Known examples of context processing include the first method, which converts a document into a graph structure, extracts important words from the words contained in the document, and predicts the relationship between target pairs, and the second method, which predicts the relationship between target pairs using the context of the entire document as the target.

[0006] In the first method, a NN is used to generate a graph with named entities in a document as vertices, and then graph processing is performed. The NN removes redundant words from the graph structure, extracts important words, and predicts the relationship between target pairs.

[0007] In the second method, a machine learning model that is strong in long-term context processing, such as Transformer, is used as the NN. The NN predicts the relationship between target pairs while taking into account the words of the entire document along with the context of the entire document. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Special Publication No. 2009-528628 [Patent Document 2] US Patent Application Publication No. 2022 / 0121822 [Non-patent literature]

[0009] [Non-Patent Document 1] Wenxuan Zhou, Kevin Huang, Tengyu Ma, Jing Huang, “Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling”, AAAI2021, 9 December 2020 Summary of the Invention [Problem to be solved by the invention]

[0010] In both the first and second methods, contextual processing involves predicting the relationship between target pairs using a machine learning model trained using training data (teacher data) to which relationship labels have been added.

[0011] When the distribution of relational labels contained in training data is unbalanced, it is difficult to train a machine learning model using the training data for a relatively small number of relational labels, and the prediction accuracy of a machine learning model trained using the training data may decrease, for example.

[0012] In addition, when little contextual information is given, it may be difficult for a machine learning model trained using the above-mentioned training data to make a judgment. In this case, an incorrect judgment result may be output, in other words, the prediction accuracy may decrease.

[0013] In one aspect, an object of the present invention is to improve the predictive accuracy of a machine learning model. [Means for solving the problem]

[0014] In one aspect, the output program may cause a computer to execute the following processes. The process may obtain first information by inputting a first document into a first machine learning model trained to output information about the input document in response to input of the input document. The process may also identify a second document having a similar graph structure similar to the graph structure of the first document from a storage unit that stores a plurality of documents. The process may also output information about the first document based on the identified second information about the second document and the first information. Effect of the Invention

[0015] In one aspect, the present invention can improve the predictive accuracy of machine learning models. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an example of an AI task according to an embodiment. [Diagram 2] FIG. 13 is a diagram illustrating an example of the distribution of labels included in training data. [Diagram 3] FIG. 13 is a diagram showing an example of an erroneous prediction result. [Figure 4] FIG. 2 is a block diagram showing an example of the hardware (HW) configuration of a computer that realizes the functions of a server according to an embodiment. [Diagram 5] FIG. 2 is a block diagram showing an example of a software configuration of a server according to an embodiment. [Figure 6]FIG. 11 is a diagram for explaining an example of operation in a training phase in a server. [Figure 7] FIG. 11 is a diagram for explaining an example of an operation in the server in the inference phase. [Figure 8] FIG. 11 is a diagram for explaining an example of processing in an inference phase. [Figure 9] FIG. 2 is a diagram showing a first example of a search corpus. [Figure 10] FIG. 13 is a diagram showing a second example of a search corpus. [Figure 11] FIG. 11 is a diagram for explaining an example of a search process by a search processing unit. [Figure 12] 11 is a diagram for explaining an example of a graph structure transformation process by a graph structure transformation engine. FIG. [Figure 13] FIG. 11 is a diagram for explaining an example of a pseudo labeling process. [Figure 14] 11 is a flowchart illustrating an example of an operation of a server according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example, and is not intended to exclude various modifications or application of techniques not specified below. For example, this embodiment can be modified in various ways without departing from the spirit of the invention. In the drawings used in the following description, parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[0018] [A] Description of one embodiment 1 is a diagram for explaining an example of an AI task according to an embodiment. As shown in FIG 1, the AI ​​task is a task for extracting information about the text, for example, specialized knowledge information, from the text, and is executed by a neural network (NN) 100 that performs context processing.

[0019] The NN 100 receives as input a text 101a (denoted as "input 1" in FIG. 1) and a target pair 101b (denoted as "input 2" in FIG. 1). The NN 100 predicts (extracts) a relationship (denoted as "output" in FIG. 1) of the target pair 101b based on the context information of the text 101a, and outputs a prediction result 102. In FIG. 1, the prediction result 102 includes the target pair "BCR-ABL1" and "imatinib" and the relationship "activate."

[0020] The text 101a is an example of a document (input document, first document) and may include, for example, one or more sentences. In the following description, it is assumed that the text 101a includes multiple sentences. The target pair 101b is an example of multiple words (first multiple words) and may be referred to as a "word pair." The target pair 101b is not limited to two words, and may be three or more words. The words of the target pair 101b may exist across multiple sentences in the text 101a, as illustrated in FIG. 1.

[0021] When the words of the target pair 101b span multiple sentences, the NN 100 executes a document-level AI task of extracting the relationship of the target pair 101b across multiple sentences. In Fig. 1, an example is shown in which the NN 100 outputs the relationship of the target pair 101b related to genes, mutations, drugs, etc. as a prediction result 102 from multiple sentences included in the text 101a.

[0022] Fig. 2 is a diagram showing an example of the distribution of labels included in training data. Fig. 2 shows a case where the number of relational labels indicated by reference symbol A1 in the training data used for training the NN100 is much larger (e.g., 20 to 30 times larger) than the number of relational labels indicated by reference symbol A2.

[0023] Fig. 3 is a diagram showing an example of an error in a prediction result. Fig. 3 shows an example of an error in a prediction result that may occur, for example, when there is little context information or when there is a bias in the number of each type (value) of labels included in the training data. In symbols B1 and B2 in Fig. 3, the labels (indicated as "x", "y", "z", "X", "Y", and "Z") of samples shown as squares, triangles, and circles are respectively separated by a boundary line shown by a dashed line.

[0024] Reference symbol B1 in Fig. 3 shows an example of a prediction result by the NN 100 trained with training data with a small bias in the number of labels. Reference symbol B1a shows a label predicted for certain data (text 101a and target pair 101b) input to the NN 100. As shown by reference symbol B1a, the label of the data is within the boundary of "y", but is correctly predicted to be "z" by the NN 100.

[0025] Reference symbol B2 in Fig. 3 shows an example of a prediction result by the NN 100 trained with training data with a large bias in the number of labels (for example, the number of labels "Z" is smaller than "X" and "Y"). As shown by reference symbol B2a, when data that can be classified into any of the labels "X", "Y", and "Z" is input, the NN 100 has difficulty in correctly predicting whether the label of the data will be "X", "Y", or "Z".

[0026] In this way, when there is a bias in the number of relational labels included in the training data, the prediction accuracy of the NN 100 trained using the training data may decrease.

[0027] Furthermore, if little context information is given, the NN 100 may output an erroneous decision result, in other words, the prediction accuracy may decrease.

[0028] Therefore, in one embodiment, a method for improving the prediction (inference) accuracy of a machine learning model will be described.

[0029] [B] Hardware configuration example of one embodiment The server 10 (see FIG. 5) according to one embodiment may be a virtual server (VM; Virtual Machine) or a physical server. The functions of the server 10 may be realized by one computer or by two or more computers. Furthermore, at least a part of the functions of the server 10 may be realized using HW (Hardware) resources and NW (Network) resources provided by a cloud environment.

[0030] 4 is a block diagram showing an example of a hardware (HW) configuration of a computer 1 that realizes the functions of a server 10 according to an embodiment. When multiple computers are used as HW resources that realize the functions of the server 10, each computer may have the HW configuration shown in FIG.

[0031] As shown in FIG. 4, the computer 1 may, as a HW configuration, illustratively include a processor 1a, a graphics processing device 1b, a memory 1c, a storage device 1d, an IF (Interface) device 1e, an IO (Input / Output) device 1f, and a reading device 1g.

[0032] The processor 1a is an example of a processing unit that performs various controls and calculations. The processor 1a may be connected to each block in the computer 1 via a bus 1j so that they can communicate with each other. The processor 1a may be a multiprocessor including a plurality of processors, a multicore processor having a plurality of processor cores, or a configuration having a plurality of multicore processors.

[0033] Examples of the processor 1a include integrated circuits (ICs) such as a CPU, MPU, APU, DSP, ASIC, and FPGA. Note that a combination of two or more of these integrated circuits may be used as the processor 1a. CPU is an abbreviation for Central Processing Unit, and MPU is an abbreviation for Micro Processing Unit. APU is an abbreviation for Accelerated Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0034] The graphics processing device 1b performs screen display control for an output device such as a monitor of the IO device 1f. The graphics processing device 1b may have a configuration as an accelerator that executes machine learning processing and inference processing using a machine learning model, for example, a relationship extraction model 11a and a vector conversion model 11b shown in Fig. 5. The graphics processing device 1b may be various arithmetic processing devices, for example, integrated circuits (ICs) such as a GPU (Graphics Processing Unit), an APU, a DSP, an ASIC, or an FPGA.

[0035] The memory 1c is an example of HW that stores various data, programs, and other information. Examples of the memory 1c include one or both of a volatile memory such as a dynamic random access memory (DRAM) and a non-volatile memory such as a persistent memory (PM).

[0036] The storage device 1d is an example of HW that stores various data, programs, and other information. Examples of the storage device 1d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a non-volatile memory. Examples of the non-volatile memory include a flash memory, a storage class memory (SCM), and a read only memory (ROM).

[0037] The storage device 1d may store a program 1h that realizes all or part of various functions of the computer 1. The program 1h may include, for example, an output program that outputs information about a document (for example, the relationship between target pairs). The output program may include a machine learning program for executing machine learning processing (training) of a machine learning model, an inference program for executing inference processing using a machine learning model, and the like.

[0038] For example, the processor 1a of the server 10 can realize the functions of the server 10 (for example, the control unit 16 shown in FIG. 5) by loading a program 1h stored in the storage device 1d into the memory 1c and executing it.

[0039] The IF device 1e is an example of a communication IF that controls the connection and communication between the server 10 and other computers. For example, the IF device 1e may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support one or both of wireless and wired communication methods.

[0040] For example, the server 10 may be connected to other computers, administrator terminals, etc., via the IF device 1e and a network so as to be able to communicate with each other. The program 1h may be downloaded to the computer 1 from the network via the IF device 1e and stored in the storage device 1d.

[0041] The IO device 1f may include one or both of an input device and an output device. Examples of the input device include a keyboard, a mouse, a touch panel, etc. Examples of the output device include a display device such as a monitor, a projector, a printer, etc. The IO device 1f may also include a touch panel or the like in which the input device and the output device are integrated. The output device may be connected to the graphic processing device 1b.

[0042] The reading device 1g is an example of a reader that reads out data or program information recorded on the recording medium 1i. The reading device 1g may include a connection terminal or device to which the recording medium 1i can be connected or inserted. Examples of the reading device 1g include an adapter that complies with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 1i may store a program 1h, and the reading device 1g may read the program 1h from the recording medium 1i and store it in the storage device 1d.

[0043] Examples of the recording medium 1i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of the magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of the flash memory include semiconductor memories such as USB memories and SD cards.

[0044] The above-described HW configuration of the computer 1 is an example. Therefore, the HW in the computer 1 may be increased or decreased (for example, adding or deleting any block), divided, or integrated in any combination, or buses may be added or deleted, etc. as appropriate.

[0045] [C] Software configuration example of one embodiment 5 is a block diagram showing an example of the software configuration of the server 10 according to an embodiment of the present invention. The server 10 is an example of an information processing device or a computer that outputs information related to a first document.

[0046] 5, the server 10 may exemplarily include a memory unit 11, a training unit 12, a relationship extraction unit 13, a search unit 14, and a prediction result generation unit 15. The training unit 12, the relationship extraction unit 13, the search unit 14, and the prediction result generation unit 15 are examples of a control unit 16.

[0047] The memory unit 11 is an example of a storage area, and stores various data used by the server 10. The memory unit 11 may be realized, for example, by a storage area included in one or both of the memory 1c and the storage device 1d (see FIG. 4) of the server 10.

[0048] 5, the memory unit 11 may be capable of storing, for example, a relationship extraction model 11a, a vector conversion model 11b, a text set 11c, and a prediction result 11d. The memory unit 11 may be capable of storing various information, such as input data, prediction results, parameters, etc., that are input, generated, or output during processing by the server 10, which will be described later. The text set 11c and the prediction result 11d will be described in detail later.

[0049] The relationship extraction model 11a is an example of a first machine learning model trained to output information about an input document in response to the input of the input document, and may be, for example, the NN 100 shown in Fig. 1. The relationship extraction model 11a may be, for example, a NN that performs context processing by at least one of the first method and the second method described above.

[0050] The vector conversion model 11b is a machine learning model trained to convert graph-structured data into vectors, and is an example of a third machine learning model for obtaining a first vector obtained by vectorizing the graph structure of a first document, and a second vector obtained by vectorizing the graph structure of a second document.

[0051] In the training phase, the training unit 12 trains the relationship extraction model 11a using teacher data.

[0052] 6 is a diagram for explaining an example of the operation of the training phase in the server 10. As shown in FIG.

[0053] The training engine 120 performs training of the relationship extraction model 11a by updating parameters 123 of the relationship extraction model 11a (predictor) using training data 121 including a plurality of teacher data 122. The teacher data 122 may include text, target pairs, and relationship labels. Various known methods may be adopted as a method for training the relationship extraction model 11a by the training engine 120.

[0054] When the training engine 120 completes updating the parameters 123 (for example, when a training end condition is met), the relationship extraction model 11a to which the updated parameters 123 have been applied is used as a trained relationship extraction model 11a.

[0055] The relationship extraction unit 13, the search unit 14, and the prediction result generation unit 15 execute the operations of the inference phase (prediction phase).

[0056] Fig. 7 is a diagram for explaining an example of the operation of the inference phase in the server 10, and Fig. 8 is a diagram for explaining an example of the processing of the inference phase. As shown in Fig. 7, the relationship extraction unit 13 may include a classification engine 130.

[0057] The classification engine 130 executes a relationship extraction process in which a first document is input to the trained relationship extraction model 11a to acquire first information. For example, the classification engine 130 inputs the input data 101 to the relationship extraction model 11a to which the parameters 131, which are the updated parameters 123 (see FIG. 6), are applied, thereby acquiring (extracting) a prediction result 102 (see FIG. 8). The prediction result 102 is an example of first information.

[0058] In the example shown in FIG. 8, predicted labels such as "Activation" and "Resistance" are obtained as the prediction results 102. The predicted labels may include probabilities (0.7 and 0.1, respectively). The process illustrated in FIG. 1 may be adopted as a process for obtaining the prediction results 102 by the classification engine 130 (relationship extraction process). As shown in FIG. 8, the input data 101 may include, for example, the text 101a and the target pair 101b shown in FIG. 1.

[0059] The search unit 14 may include a search processing unit 140, as shown in Fig. 7 and Fig. 8. The search processing unit 140 extracts text having a structure similar to that of the input data 101 by searching the search corpus 110, and outputs a search result 103 including information about the text. The information about the text may include, for example, the relationship between the target pair 101b in the text.

[0060] The search corpus 110 is an example of a storage unit that stores a plurality of documents, and is an example of the text collection 11c shown in FIG.

[0061] The text set 11c may be a database that accumulates a plurality of texts. The text set 11c may include, for example, a plurality of texts (documents) related to (for example, belonging to a similar field) at least to the text 101a of the input data 101. For example, the text set 11c may include a plurality of texts including a sufficient number of relational labels to reduce bias in the types (values) of the relational labels in the training data 121 (see FIG. 6).

[0062] 9 is a diagram showing a first example of the search corpus 110. The search corpus 110A according to the first example may include a plurality of pieces of information that associate text 111, target pairs 112, and relational labels 113. A part of this information included in the search corpus 110A may be, for example, training data 122.

[0063] 10 is a diagram showing a second example of the search corpus 110. The search corpus 110B according to the second example may include a plurality of pieces of information associating the text 111 with the target pair 112. Note that in the search corpus 110B, the target pair 112 of the text 111 and the target pair 112 may be omitted, and only the text 111 may exist in a plurality of pieces.

[0064] Returning to the explanation of Fig. 7 and Fig. 8, the search results 103 are used to correct the prediction results 102 of the classification engine 130. The search results 103 may include the top N (N is an integer equal to or greater than 1) texts 111 with the highest similarity and relational labels (including probabilities) corresponding to the texts 111, as shown in Fig. 8 for example. As an example, Fig. 8 shows two search results 103: No. 1: "Similarity '0.9', Label 'Activation', Probability '0.8'" and No. 2: "Similarity '0.8', Label 'Activation', Probability '0.6'".

[0065] The search processing unit 140 may include graph structure transformation engines 141a and 141b (hereinafter, when there is no need to distinguish between them, they may be referred to as graph structure transformation engine 141) and a similarity calculation engine 142. Furthermore, when the search corpus 110 is a search corpus 110B (second example) that does not include a relation label 113, the search processing unit 140 may include a pseudo label assignment module 143.

[0066] The graph structure conversion engine 141 converts an input document into graph structure data (graph).

[0067] FIG. 11 is a diagram for explaining an example of the search processing by the search processing unit 140, and FIG. 12 is a diagram for explaining an example of the graph structure conversion processing by the graph structure conversion engine 141. As shown in FIG.

[0068] As shown in FIG. 11, the graph structure conversion engine 141a converts the text 101a included in the input data 101 into a graph (input graph) 114a having a graph structure based on proper nouns, function words, etc. (see symbol C1).

[0069] For example, the graph structure conversion engine 141a may use words of a target pair 101b associated with the text 101a (contained in the input data 101) to convert the text 101a into a graph. As an example, a character string combining the text 101a and the target pair 101b may be input to the graph structure conversion engine 141a, and a graph obtained by converting the character string into a graph structure may be output.

[0070] The graph structure conversion engine 141b converts each of the texts 111 included in the text collection 11c (search corpus 110) into a graph 114b having a graph structure based on proper nouns, function words, etc. (see symbol C2).

[0071] For example, the graph structure conversion engine 141b may use words of the target pair 112 associated with the text 111 to convert the text 111 into a graph. As an example, a character string combining the text 111 and the target pair 112 may be input to the graph structure conversion engine 141b, and a graph obtained by converting the character string into a graph structure may be output.

[0072] 12, the graph structure conversion engine 141 converts the text 101a or 111 and the target pair 101b or 112 into a graph 114. The graph 114 has a structure in which each of the multiple elements included in 101a or 111 is a node 115, and the relationships between the nodes 115 are connected by edges 116.

[0073] Node 115 may include a sentence node indicating a sentence, a mention node indicating a word included in text 101a or 111, and an entity node indicating a target word (e.g., a word in target pair 101b or 112). The sentence node is connected by edges to the mention nodes and entity nodes included in the sentence. The mention node is connected by edges to the entity node related to (for example, matching) the word. Various known methods may be adopted as the method for graph structure conversion processing by graph structure conversion engine 141.

[0074] The similarity calculation engine 142 converts the graph 114 converted by the graph structure conversion engine 141 into a vector using the vector conversion model 11b.

[0075] 11, the similarity calculation engine 142 converts the graph 114a generated by the graph structure conversion engine 141a into a vector (hereinafter, may be referred to as a "graph vector") (see symbol C3). The similarity calculation engine 142 also converts the graph 114b generated by the graph structure conversion engine 141b into a graph vector (see symbol C4). As a method of vector conversion processing by the similarity calculation engine 142, various known methods may be adopted.

[0076] Then, a graph vector similar to the graph vector of the text 101a is searched for from the graph vectors of the multiple texts 111. For example, the similarity calculation engine 142 specifies similar text 118 having a similar graph structure similar to the graph structure of the text 101a from the text set 11c (search corpus 110). The similar text 118 having a similar graph structure is text 111 having a similar context (similar context) to that of the text 101a, and is an example of a second document.

[0077] For example, the similarity calculation engine 142 calculates the inter-graph similarity between the graph vector (first vector) of the text 101a and each of the graph vectors of the multiple texts 111 included in the search corpus 110 (see reference symbol C5 in FIG. 11). As an example, the similarity calculation engine 142 may calculate the inter-graph similarity by calculating the inner product of the graph vector of the text 101a and each of the graph vectors of the multiple texts 111 included in the search corpus 110. Examples of the inter-graph similarity include vector similarity, vertex similarity, and surface similarity (surface agreement) degree.

[0078] The similarity calculation engine 142 specifies the graph 114 (before conversion) corresponding to a graph vector (second vector) whose inter-graph similarity with the graph vector of the text 101a satisfies a predetermined condition as a similar graph 117 having a similar graph structure. This allows the similarity calculation engine 142 to specify the text 111 (before conversion) corresponding to the similar graph 117 as a similar text 118 having a high similarity with the text 101a.

[0079] The predetermined condition may mean being included in the top N graphs with high inter-graph similarity (see FIG. 8). Therefore, as described above, N or less similar graphs 117 or similar texts 118 may be specified.

[0080] The similarity calculation engine 142 acquires second information related to the identified similar graph 117 or similar text 118. For example, the similarity calculation engine 142 acquires search results 103 including a relation label corresponding to the similar graph 117 or similar text 118 (see FIG. 8 ), and outputs the search results 103 to the prediction result generating unit 15. The search results 103 are an example of the second information.

[0081] The method of obtaining the search results 103 by the similarity calculation engine 142 may differ depending on whether the search corpus 110 is the first example or the second example, for example.

[0082] When the search corpus 110 is the first example search corpus 110A (see FIG. 9), the similarity calculation engine 142 may include the relation label 113 associated with the similar text 118 (text 111) in the search result 103 together with the vector similarity.

[0083] When the search corpus 110 is the second example search corpus 110B (see FIG. 10), the similarity calculation engine 142 outputs to the pseudo label assignment module 143 information indicating the similar text 118 (text 111).

[0084] Fig. 13 is a diagram for explaining an example of a pseudo labeling process. As shown in Fig. 13, the pseudo labeling module 143 may input the similar text 118 and the target pair 101b included in the input data 101 to a neural network (NN) 144 to obtain a relational label 119 (113). For example, the pseudo labeling module 143 may treat the obtained relational label 119 as a pseudo label (pseudo correct label) and include the relational label 119 in the search result 103 together with the vector similarity.

[0085] NN144 is an example of a second machine learning model that is trained to output information about an input document in response to receiving the input document.

[0086] For example, the NN 144 may be the same NN as the NN 100 (relationship extraction model 11a) shown in Fig. 1, or may be an NN having parameters similar to those of the NN 100. As an example, the NN 144 may be the NN 100 shown in Fig. 1, which is an NN that uses the first method, that is, converts the similar text 118 into a graph structure, extracts important words from among the words included in the similar text 118, and predicts the relationship label 119 of the target pair 101b.

[0087] Alternatively, the NN 144 may be a trained machine learning model that references a knowledge database (e.g., a knowledge base) to generate relation labels 119 for similar texts 118. The machine learning model may be a model trained by a technique such as Distant Supervision.

[0088] 7 and 8, the prediction result generating unit 15 generates information about the input data 101 (text 101a), such as a relational label, as a prediction result 11d based on the prediction result 102 and the search result 103, and stores it in the memory unit 11. In other words, the prediction result generating unit 15 corrects the final result of the task by the server 10 by taking into account (reflecting) the relational label of the similar text 118 extracted from the text set 11c by the search processing unit 140 into the prediction label output from the classification engine 130.

[0089] The prediction result generating unit 15 may calculate the prediction result 11d based on, for example, the prediction result 102 and one or more (N or less) search results 103 by the following method (a) or (b).

[0090] (a) Addition of Probabilities For example, the prediction result generating unit 15 may calculate the prediction result 11d by simply adding (and averaging) the probability of the predicted label of the prediction result 102 and the probability of each related label of one or more search results 103 for the same label.

[0091] (b) Weighted average sum of probabilities Alternatively, the prediction result generating unit 15 may calculate the prediction result 11d by calculating a weighted average sum of the probability of the predicted label of the prediction result 102 and the probability of each related label of one or more search results 103 for the same labels. For example, as weights, α to be assigned (multiplied) to the probability of the predicted label of the prediction result 102 and β to be assigned (multiplied) to the probability of each related label of one or more search results 103 may be used. α and β are real numbers between "0.0" and "1.0", and α+β="1.0". As an example, values ​​such as α="0.7" and β="0.3" may be used.

[0092] In addition, in the above (a) and (b), the prediction result generation unit 15 may use, as the probability of the relation label of one or more search results 103, a value obtained by multiplying the similarity (with respect to text 101a) of the similar text 118 corresponding to the relation label by the probability of the relation label.

[0093] Furthermore, the prediction result generating unit 15 may normalize the calculated probabilities in the above (a) and (b).

[0094] The method of calculating the prediction result 11d based on the prediction result 102 and the search result 103 by the prediction result generating unit 15 is not limited to the above-mentioned method, and various methods may be used.

[0095] The server 10 outputs output data including the prediction result 11d generated by the above-mentioned process. The output data may include, for example, in addition to the prediction result 11d, various intermediate data, such as at least one of the similar graph 117, the similar text 118, the prediction result 102, and the search result 103.

[0096] In "outputting" the output data, the server 10 may, for example, transmit (provide) the output data to another computer (e.g., the source of the input data 101) not shown, or may store the output data in the memory unit 11 and manage it so that it can be acquired from the server 10 or another computer. Alternatively, in "outputting" the output data, the server 10 may output information indicating the output data to a screen on an output device of the server 10 or another computer, or may output the output data in various other modes.

[0097] As described above, according to the server 10 according to an embodiment, the input data 101 is input to the relationship extraction model 11a to obtain the prediction result 102. Furthermore, similar text 118 having a similar graph 117 similar to the graph 114a of the input data 101 is specified from the text collection 11c. Then, based on the search result 103 related to the specified similar text 118 and the prediction result 102, the prediction result 11d related to the input data 101 is output.

[0098] This allows the server 10 to correct the prediction result 102 for a specific relation label output from the relation extraction model 11a using the search result 103 based on similar texts 118 included in the text set 11c. Therefore, even if the relation extraction model 11a is trained using training data 121 with a small number of the specific relation labels, the server 10 can output the prediction result 11d in which the possibility of an error in the prediction result 102 output from the relation extraction model 11a is reduced (corrected).

[0099] For example, in the server 10, similar text 118 is searched from the text set 11c by search, and the relation label 113 is taken into consideration in the prediction result 11d, so that if appropriate similar text 118 is acquired, the correct relation label 113 can be obtained. Since the search does not involve machine learning, such a method can improve the imbalance of the training data in the relation extraction model 11a.

[0100] Furthermore, the server 10 can complement the context information of the input data 101 with the context information contained in the text collection 11c, and can therefore output a prediction result 11d that corrects the determination result of the relationship extraction model 11a.

[0101] Therefore, the prediction accuracy of the relationship extraction model 11a can be improved.

[0102] In addition, in the case of a complex sentence structure, it is difficult to appropriately grasp important contexts in the case of extracting relationships in text. In response to this, the server 10 according to an embodiment acquires (searches for) similar text 118 corresponding to a similar graph structure similar to the graph structure of the input data 101. In the graph structure, the contents of the text are organized, making it easier to grasp important contexts, and therefore the context can be complemented by the appropriate similar text 118.

[0103] Note that a complex sentence structure is, for example, a structure in which the target pair 112 exists across multiple sentences. That is, in one embodiment, the same effect can be achieved not only in a simple sentence level in which the target pair 112 exists in one sentence, but also in a complex sentence level AI task in which the target pair 112 exists across multiple sentences.

[0104] In addition, the above-mentioned first or second method can be used as the relationship extraction model 11a, and an existing method can be adopted as the training method thereof. Therefore, the server 10 according to one embodiment can be easily applied to an existing server.

[0105] Furthermore, the method according to the embodiment can be applied regardless of whether the relation label 113 is associated with the text 111 in the text collection 11c.

[0106] For example, if the text 111 has a relational label 113 associated therewith (see FIG. 9), the server 10 can reliably or easily obtain the search results 103 from the similar text 118.

[0107] Furthermore, when the relational label 113 is not associated with the text 111 (see FIG. 10), the range of objects available as the text collection 11c can be expanded to include, for example, various databases and websites, which is more convenient than the example in FIG. 9. Even in this case, the relational label 119 (pseudo label) can be generated by the pseudo label assignment module 143, so that the search results 103 can be reliably obtained from the similar text 118.

[0108] Furthermore, the server 10 acquires a first vector obtained by vectorizing the graph 114a of the input data 101, and vectors of the graphs 114b of each of the multiple texts 111 in the text set 11c. Then, of the vectors, the graph 114b corresponding to a second vector whose similarity with the first vector satisfies a predetermined condition is identified as a similar graph 117.

[0109] In this way, by converting the graph 114 into a vector and calculating the similarity between texts using the vector (graph vector), it is possible to reliably or easily specify similar text 118 that is similar to the input data 101.

[0110] [D] Example of operation Next, an operation example of the server 10 according to an embodiment will be described. Fig. 14 is a flowchart illustrating an operation example of the server 10 according to an embodiment. Fig. 14 shows an operation example of the server 10 in the inference (prediction) phase.

[0111] 14, the server 10 acquires input data 101 including a text 101a and a target pair 101b as target data (step S1). The subsequent step S2 and steps S3 to S6 may be executed in succession, or at least some of the processes may be executed in parallel.

[0112] The classification engine 130 inputs the target data into the relationship extraction model 11a to obtain the prediction result 102 including the predicted label (step S2), and the process proceeds to step S7.

[0113] On the other hand, the search processor 140 converts the target data into a graph 114a by the graph structure conversion engine 141a (step S3).

[0114] The search processing unit 140 inputs the graph 114a of the target data to the vector conversion model 11b by the similarity calculation engine 142, and acquires the graph vector of the target data (step S4).

[0115] The search processing unit 140 converts each of the multiple texts 111 and the target pairs 112 included in the text set 11c into a graph 114b using the graph structure conversion engine 141b. The search processing unit 140 also inputs each of the multiple texts 111 and the graphs 114b of the target pairs 112 included in the text set 11c into the vector conversion model 11b using the similarity calculation engine 142 to convert them into graph vectors.

[0116] The search processing unit 140 searches the top N similar texts 118 corresponding to vectors similar to the graph vector of the target data by using the similarity calculation engine 142 (step S5).

[0117] The search processing unit 140 acquires the similarity of the similar text 118 obtained by the search and the relation label 113 (or 119) corresponding to the similar text 118 as the search result 103 using the similarity calculation engine 142 (step S6), and the process proceeds to step S7.

[0118] The prediction result generation unit 15 corrects the prediction result 102 obtained in step S2 using the similarity and relationship label 113 included in the search result 103 obtained in step S6, and outputs the corrected result as the prediction result 11d (step S7), and the processing ends.

[0119] [E] Other The technology according to the embodiment described above can be implemented with the following modifications and variations.

[0120] For example, the software configurations included in the server 10 shown in FIG. 5 may be combined in any desired combination, or may be divided into different parts.

[0121] 5 may be a configuration (system) in which a plurality of devices cooperate with each other via a network to realize each processing function. As an example, the memory unit 11 may be a DB (Database) server, and the training unit 12, the relationship extraction unit 13, the search unit 14, and the prediction result generation unit 15 may be a Web server or an application server. In this case, the DB server, the application server, and the Web server may cooperate with each other via a network to realize each processing function of the server 10.

[0122] In addition, in the embodiment, the text collection 11c (search corpus 110) stores data in text format, but the present invention is not limited to this.

[0123] For example, the text collection 11c may further store, in association with the text 111, one or both of a graph 114b, which is data obtained by converting the text 111 and the target pairs 112 into a graph structure, and a graph vector, which is obtained by vectorizing the graph 114b.

[0124] As a result, when the input data 101 is input to the search unit 14, the search unit 14 only needs to perform graph conversion processing and vector conversion processing of the input data 101, and refer to the graph vector from the text collection 11c for search. Therefore, it is possible to reduce the processing load on the server 10 when the input data 101 is input, and also to shorten the processing time from when the input data 101 is input to when the prediction result 11d is output.

[0125] Furthermore, the training unit 12 according to an embodiment may execute retraining of the relationship extraction model 11a using, for example, the relationship labels of the prediction result 11d output by the prediction result generating unit 15 and the input data 101. This allows retraining to be performed using retraining data that complements the teacher data 122 of the relationship labels that are few in number during training, thereby improving the accuracy of the prediction result 102 of the relationship extraction model 11a.

[0126] Note that retraining may be performed when the number of combinations between the input data 101 and the prediction results 11d reaches a predetermined number, when a predetermined time period has elapsed, etc. As a method for retraining, various known methods may be adopted.

[0127] [F] Notes Regarding the above embodiment, the following supplementary notes are further disclosed.

[0128] (Appendix 1) A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An output program that causes a computer to carry out processing.

[0129] (Appendix 2) The storage unit stores information about the document in association with each of the documents, the process of identifying the second document includes a process of extracting, from the storage unit, the second information associated with the second document having the similar graph structure; An output program as described in Appendix 1.

[0130] (Appendix 3) The process of identifying the second document includes a process of inputting the second document corresponding to the similar graph structure into a second machine learning model to obtain the second information. An output program as described in Appendix 1.

[0131] (Appendix 4) the second machine learning model is the first machine learning model; An output program as described in Appendix 3.

[0132] (Appendix 5) the second machine learning model is a trained machine learning model that generates the second information about the second document by referring to a knowledge database; An output program as described in Appendix 3.

[0133] (Appendix 6) The process of identifying the second document includes: obtaining a first vector obtained by vectorizing a graph structure of the first document; For each of the documents stored in the storage unit, a vector is obtained by vectorizing a graph structure of the document; identifying, as the similar graph structure, a graph structure corresponding to a second vector, the similarity between which satisfies a predetermined condition and the first vector, among the vectors of each of the plurality of documents; An output program according to any one of Supplementary Note 1 to Supplementary Note 5.

[0134] (Appendix 7) the first machine learning model is a machine learning model trained to output a relationship between the input words in the input document in response to input of the input document and a plurality of input words; The process of acquiring the first information includes a process of inputting the first document and a first plurality of words as the first information into the first machine learning model to acquire a first relationship; The output process includes: obtaining, as the second information, a second relationship between the first plurality of words in the second document; outputting a relationship between the first plurality of words in the first document based on the first relationship and the second relationship; An output program according to any one of Supplementary notes 1 to 6.

[0135] (Appendix 8) A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An output method in which processing is performed by a computer.

[0136] (Appendix 9) The storage unit stores information about the document in association with each of the documents, the process of identifying the second document includes a process of extracting, from the storage unit, the second information associated with the second document having the similar graph structure; Output method described in Appendix 8.

[0137] (Appendix 10) The process of identifying the second document includes a process of inputting the second document corresponding to the similar graph structure into a second machine learning model to obtain the second information. Output method described in Appendix 8.

[0138] (Appendix 11) the second machine learning model is the first machine learning model; The output method described in Appendix 10.

[0139] (Appendix 12) the second machine learning model is a trained machine learning model that generates the second information about the second document by referring to a knowledge database; The output method described in Appendix 10.

[0140] (Appendix 13) The process of identifying the second document includes: obtaining a first vector obtained by vectorizing a graph structure of the first document; For each of the documents stored in the storage unit, a vector is obtained by vectorizing a graph structure of the document; identifying, as the similar graph structure, a graph structure corresponding to a second vector, the similarity between which satisfies a predetermined condition and the first vector, among the vectors of each of the plurality of documents; An output method according to any one of Supplementary Notes 8 to 12.

[0141] (Appendix 14) the first machine learning model is a machine learning model trained to output a relationship between the input words in the input document in response to input of the input document and a plurality of input words; The process of acquiring the first information includes a process of inputting the first document and a first plurality of words as the first information into the first machine learning model to acquire a first relationship; The output process includes: obtaining, as the second information, a second relationship between the first plurality of words in the second document; outputting a relationship between the first plurality of words in the first document based on the first relationship and the second relationship; An output method according to any one of Supplementary Notes 8 to 13.

[0142] (Appendix 15) A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An information processing device comprising a control unit.

[0143] (Appendix 16) The storage unit stores information about the document in association with each of the documents, the control unit, in the process of identifying the second document, extracts, from the storage unit, the second information associated with the second document having the similar graph structure. 16. The information processing device according to claim 15.

[0144] (Appendix 17) the control unit, in the process of identifying the second document, inputs the second document corresponding to the similar graph structure into a second machine learning model to acquire the second information. 16. The information processing device according to claim 15.

[0145] (Appendix 18) the second machine learning model is the first machine learning model; 18. The information processing device according to claim 17.

[0146] (Appendix 19) The control unit, in the process of identifying the second document, obtaining a first vector obtained by vectorizing a graph structure of the first document; For each of the documents stored in the storage unit, a vector is obtained by vectorizing a graph structure of the document; identifying, as the similar graph structure, a graph structure corresponding to a second vector, the similarity between which satisfies a predetermined condition and the first vector, among the vectors of each of the plurality of documents; The information processing device according to any one of Supplementary claims 15 to 18.

[0147] (Appendix 20) the first machine learning model is a machine learning model trained to output a relationship between the input words in the input document in response to input of the input document and a plurality of input words; The control unit is In the process of acquiring the first information, the first document and a first plurality of words are input to the first machine learning model as the first information to acquire a first relationship; In the output process, obtaining, as the second information, a second relationship between the first plurality of words in the second document; outputting a relationship between the first plurality of words in the first document based on the first relationship and the second relationship; The information processing device according to any one of Supplementary claims 15 to 19. [Explanation of symbols]

[0148] 1. Computer 10 Server 11 Memory section 12 Training Department 13 Relationship Extraction 14 Search section 15 Prediction result generation unit 16 Control section 11a Relationship Extraction Model 11b Vector conversion model 11c Text collection 11d,102 Prediction results 100,144 NN 101 Input Data 101a,111 Text 101b,112 Target Pair 103 Results 110, 110A, 110B Search corpus 113,119 Relationship Labels 114,114a,114b Graph 115 nodes 116 Edge 117 Similar Graphs 118 Similar Texts 120 Training Engine 121 Training Data 122 Teacher Data 123,131 Parameters 130 Classification Engine 140 Search processing unit 141,141a,141b Graph structure transformation engine 142 Similarity Calculation Engine 143 Pseudo Labeling Module

Claims

1. A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An output program that causes a computer to carry out processing.

2. The storage unit stores information about the document in association with each of the documents, the process of identifying the second document includes a process of extracting, from the storage unit, the second information associated with the second document having the similarity graph structure; The output program according to claim 1 .

3. The process of identifying the second document includes a process of inputting the second document corresponding to the similar graph structure into a second machine learning model to obtain the second information. The output program according to claim 1 .

4. The process of identifying the second document includes: A first vector is obtained by vectorizing a graph structure of the first document; For each of the documents stored in the storage unit, a vector is obtained by vectorizing a graph structure of the document; identifying, as the similar graph structure, a graph structure corresponding to a second vector, the similarity between which satisfies a predetermined condition and the first vector, among the vectors of each of the plurality of documents; The output program according to any one of claims 1 to 3.

5. the first machine learning model is a machine learning model trained to output a relationship between the input words in the input document in response to input of the input document and a plurality of input words; The process of acquiring the first information includes a process of inputting the first document and a first plurality of words as the first information into the first machine learning model to acquire a first relationship; The output process includes: obtaining, as the second information, a second relationship between the first plurality of words in the second document; outputting a relationship between the first plurality of words in the first document based on the first relationship and the second relationship. The output program according to any one of claims 1 to 3.

6. A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An output method in which processing is performed by a computer.

7. A first document is input to a first machine learning model trained to output information about the input document in response to input of the input document to obtain first information; Identifying a second document having a similar graph structure to the graph structure of the first document from a storage unit that stores a plurality of documents; outputting information about the first document based on second information about the identified second document and the first information; An information processing device comprising a control unit.

Images