Information processing device, operating method for information processing device, and operating program for information processing device

Abstract

To provide an information processing device for preparing a learning input sentence which contributes to effective learning of a natural language processing model, a method for operating the information processing device, and an operation program of the information processing device.SOLUTION: In a learning data preparation device 11, a CPU 32X includes an extraction part 53, a derivation part 54 and a preparation part (selection part) 55. The extraction part 53 extracts a plurality of unique expressions being specific phrases from a medical document. The derivation part 54 derives a degree of association showing the degree of a correlation between unique expressions. The preparation part (selection part) 55 selects an object phrase being a phrase to be a mask object on the basis of the degree of association.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an information processing device, an operating method for an information processing device, and an operating program for an information processing device. [Background technology]

[0002] In the field of natural language processing (NLP), BERT (Bidirectional Encoder Representations from Transformers), which uses a transformer encoder, is attracting attention. BERT uses Masked Language Modeling (MLM) as pre-training. MLM is a fill-in-the-blank problem in which the system is asked to predict what words will fit into the masked parts of training input sentences where some of the words are masked. After this pre-training, BERT undergoes fine-tuning according to the desired natural language processing task. A phrase is a single word and / or a phrase consisting of a combination of one or more words.

[0003] Typically, the words to be masked in the training input sentences used in MLM (hereafter referred to as target words) are selected randomly. However, in this case, words that are not very important in the target document of the natural language processing task are often selected as target words. When MLM is performed using such training input sentences, the opportunity to learn important words in the target document of the natural language processing task is lost, which could result in ineffective pre-training.

[0004] In particular, in fields where it is difficult to obtain a large number of documents that serve as the source of training input sentences, such as the medical field, the risk of losing learning opportunities for important terms increases. For example, in the sentence "Early thickening of the tumor observed in liver S3 is observed," if less important terms such as the particle "ni" are selected as target terms and important terms such as "tumor" and "early thickening" are not selected as target terms, the opportunity to learn important terms is lost, and effective pre-training is not performed. Therefore, a method for selecting target terms has been proposed, as in Non-Patent Document 1, for example.

[0005] Non-Patent Document 1 describes a technology for creating training input sentences to be used in MLM of BERT for medical documents. Specifically, named entities, which are terms specific to medical documents such as disorders, signs, symptoms, medications, anatomical locations, procedures, and time expressions, are extracted, and target terms are selected based mainly on the extracted named entities. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Chen Lin, et al: EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain, Proceedings of the BioNLP 2021 workshop, pages 191-201 June 11, 2021. Summary of the Invention [Problem to be solved by the invention]

[0007] If the training input sentences used in MLM do not contain words necessary for predicting the masked words, it is naturally difficult to predict the masked words. Taking a medical document as an example, the word "early thickening," which describes the characteristics of a liver tumor, is highly associated with hepatocellular carcinoma (HCC). However, if both "early thickening" and "HCC" in the sentence "Early thickening was observed in the tumor observed in liver S3, and HCC is suspected," are masked, it is difficult to predict either "early thickening" or "HCC" from the other because the words necessary for predicting "early thickening" and "HCC" do not exist in the sentence. Non-Patent Document 1 does not take this issue into consideration. Therefore, there is still a risk that effective pre-training will not be performed.

[0008] One embodiment of the technology disclosed herein provides an information processing device, an operating method for an information processing device, and an operating program for an information processing device that are capable of creating training input sentences that contribute to effective learning of a natural language processing model. [Means for solving the problem]

[0009] The information processing device disclosed herein is an information processing device that creates training input sentences from documents, in which some words are masked, and is used to train a natural language processing model. The information processing device includes a processor that extracts multiple specific words from the document, derives a degree of association that indicates the degree of association between the multiple specific words, and selects target words that are to be masked based on the degree of association.

[0010] It is preferable that the processor excludes, from the candidates for the target phrase, specific phrases whose relevance with the phrase selected as the target phrase satisfies a preset condition.

[0011] The processor preferably determines sentences of the same type as sentences containing the phrase selected as the target phrase, and sets the sentences of the same type as sentences containing the target phrase as training input sentences.

[0012] The processor preferably determines that sentences containing specific phrases whose degrees of relevance to the phrase selected as the target phrase satisfy a preset condition are of the same type.

[0013] The document is a medical document, and the processor preferably determines that sentences describing the same organ as the organ related to the term selected as the target term are of the same type.

[0014] It is preferable that the processor derives the degree of association using association information in which the degree of association is preset according to a combination of types of specific words.

[0015] The document is a medical document, and the type preferably includes at least one of an anatomical location, a quantity, a lesion name, a lesion characteristic, and a disease name.

[0016] Preferably, the processor derives the relevance using a trained derivation model.

[0017] The derived model is preferably created based on a trained natural language processing model.

[0018] Preferably, the processor trains the natural language processing model using the training input sentences.

[0019] It is preferable that the processor repeats a cycle of the following process: creating an extraction model that extracts at least specific words and phrases from the natural language processing model; using the extraction model to create training documents from which the specific words and phrases have been extracted; creating training input sentences from the training documents; and updating the natural language processing model by training the natural language processing model using the created training input sentences.

[0020] In the first cycle, the processor preferably creates an extraction model from a natural language processing model created by training using training input sentences in which target phrases are randomly selected regardless of relevance.

[0021] Preferably, the processor further creates a derivation model for deriving relevance from the natural language processing model, and creates training documents labeled with relevance using the derivation model.

[0022] Preferably, the documents are medical documents, and the processor further creates a determination model from the natural language processing model that determines which organ a sentence describes, and uses the determination model to create training documents in which the organs are labeled.

[0023] The disclosed method of operating an information processing device is a method of operating an information processing device that creates, from a document, training input sentences used for training a natural language processing model, in which some words are masked, and includes extracting multiple specific words from the document, deriving a degree of association that indicates the degree of association between the multiple specific words, and selecting, based on the degree of association, target words that are to be masked.

[0024] The operating program of the information processing device disclosed herein is an operating program of an information processing device that creates, from a document, training input sentences used for training a natural language processing model, training input sentences in which some words are masked, and causes a computer to perform processes including extracting multiple specific words from the document, deriving a relevance indicating the degree of relevance between the multiple specific words, and selecting target words that are to be masked based on the relevance. [Effects of the Invention]

[0025] According to the technology disclosed herein, it is possible to provide an information processing device, an operating method for an information processing device, and an operating program for an information processing device that are capable of creating training input sentences that contribute to the effective learning of a natural language processing model. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a diagram illustrating a learning system including a learning data creation device and a learning device. [Figure 2]FIG. 10 is a diagram showing learning input sentences and correct answer data created from original medical documents. [Figure 3] FIG. 1 is a diagram illustrating a learning procedure for a natural language processing model. [Figure 4] FIG. 2 is a block diagram showing a computer that constitutes the learning data creation device and the learning device. [Figure 5] FIG. 2 is a block diagram showing a processing unit of a CPU of the learning data creation device. [Figure 6] FIG. 10 is a diagram illustrating processing performed by a preprocessing unit. [Figure 7] FIG. 10 is a diagram illustrating the processing of an extraction unit. [Figure 8] FIG. 10 is a diagram illustrating association level information. [Figure 9] FIG. 10 is a diagram showing the derivation results. [Figure 10] FIG. 10 is a diagram showing selection conditions. [Figure 11] FIG. 10 is a diagram illustrating the process of the creation unit. [Figure 12] FIG. 10 is a diagram summarizing the processing of the learning data creation device. [Figure 13] FIG. 2 is a block diagram showing a processing unit of a CPU of the learning device. [Figure 14] FIG. 1 is a diagram illustrating an overview of the MLM pre-learning process for a natural language processing model. [Figure 15] 10 is a flowchart showing a processing procedure of the learning data creation device. [Figure 16] 10 is a flowchart showing a processing procedure of the learning device. [Figure 17] FIG. 10 is a diagram illustrating a second embodiment in which a degree of association is derived using a trained derivation model. [Figure 18] FIG. 10 is a diagram showing a derivation result in the second embodiment. [Figure 19] FIG. 10 is a diagram showing a derivation result in the 3_1 embodiment. [Figure 20] FIG. 10 is a diagram showing the processing of the creation unit in the 3_1 embodiment. [Figure 21] FIG. 10 is a diagram summarizing the processing of the learning data creation device in the 3_1 embodiment. [Figure 22]FIG. 10 is a diagram showing a processing unit of a third embodiment including a determination unit that determines which organ a sentence in a medical document describes. [Figure 23] FIG. 10 is a diagram showing a determination result. [Figure 24] FIG. 10 is a diagram showing the processing of the creation unit in the third embodiment. [Figure 25] FIG. 10 is a diagram summarizing the processing of the learning data creation device in the third embodiment. [Figure 26] 10 is a flowchart showing a processing procedure according to a fourth embodiment. [Figure 27] FIG. 10 is a diagram showing a processing procedure according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0027] [First embodiment] As an example, as shown in FIG. 1, a training system 2 is a system for training a natural language processing model 10 (see FIG. 3), which is a BERT using a transformer encoder, and is composed of a training data creation device 11 and a training device 12. The training data creation device 11 and the training device 12 are, for example, desktop personal computers or workstations. The training data creation device 11 and the training device 12 are connected via a network (not shown) such as the Internet. The training data creation device 11 and the training device 12 are examples of an "information processing device" according to the technology of the present disclosure.

[0028] A medical document database (hereinafter abbreviated as DB (Data Base)) server 13 is connected to the training data creation device 11 via a network (not shown) such as the Internet. The training data creation device 11 receives a medical document group 14G from the medical document DB server 13. The medical document group 14G is a collection of multiple medical documents 14 such as medical papers and radiology reports.

[0029] The training data creation device 11 creates multiple training data 15 from medical documents 14. The training data 15 is a set of training input sentences 16 and correct answer data 17. The training data creation device 11 distributes a training data group 15G, which is a collection of multiple training data 15, to the training device 12.

[0030] As an example, as shown in FIG. 2, a training input sentence 16 is a sentence in which some words of an original sentence 18 contained in a medical document 14 have been masked. Correct answer data 17 is data indicating what the masked words are. FIG. 2 illustrates an original sentence 18 that reads, "A mass measuring 6 cm in diameter was found in liver S3, showing early hyperplasia and washout, and HCC is suspected." Also illustrated are training input sentence 16 in which four words (mask1 to mask4) - "diameter," "large," "found," and "HCC" - have been masked, and correct answer data 17 indicating each of the words in mask1 to mask4.

[0031] The learning device 12 receives the training data group 15G from the training data creation device 11. The learning device 12 creates a task adaptation model 20 from the natural language processing model 10 by training the natural language processing model 10 using the training data group 15G.

[0032] More specifically, as an example, as shown in FIG. 3, the learning device 12 performs two stages of learning: pre-learning and fine-tuning. In the pre-learning, MLM using a training data set 15G and NSP (Next Sentence Prediction) are performed. MLM is a task that causes the natural language processing model 10 to predict what words will be placed in the masked portion of a training input sentence 16. NSP is a task that causes the natural language processing model 10 to determine whether two training input sentences, prepared separately from the training input sentence 16, are consecutive sentences. Since the technology disclosed herein mainly focuses on MLM, a detailed description of NSP will be omitted.

[0033] The natural language processing model 10 is pre-trained to become a pre-trained natural language processing model 19. Fine tuning is performed on this pre-trained natural language processing model 19. The pre-trained natural language processing model 19 is then fine-tuned to become a task adaptation model 20. The task adaptation model 20 is distributed from the learning device 12 to an operation device (not shown).

[0034] Fine tuning is a learning process that adjusts the values ​​of various parameters of the pre-trained natural language processing model 19 set in pre-training to values ​​suitable for a desired natural language processing task. In fine tuning, medical documents 14 labeled with classes related to the desired natural language processing task are prepared as training data.

[0035] Natural language processing tasks include Multi-Genre Natural Language Inference (MNLI), a task that determines whether two input sentences are semantically entailed, contradictory, or neutral, and Named Entity Recognition (NER). Other natural language processing tasks include Quora Question Pairs (QQP), a task that classifies whether questions are the same, and Semantic Textual Similarity Benchmark (STS-B), a task that evaluates the similarity between two sentences. As with NSP, the technology disclosed herein primarily focuses on MLM, and therefore a detailed description of fine-tuning will be omitted.

[0036] 4, the computers that make up the learning data creation device 11 and the learning device 12 have the same basic configuration, and include a storage 30, a memory 31, a CPU (Central Processing Unit) 32, a communication unit 33, a display 34, and an input device 35. These are interconnected via a bus line 36.

[0037] The storage 30 is a hard disk drive built into the computer that constitutes the learning data creation device 11 and the learning device 12, or connected via a cable or network. Alternatively, the storage 30 is a disk array consisting of multiple hard disk drives. The storage 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0038] The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads a program stored in the storage 30 into the memory 31 and executes processing in accordance with the program. In this way, the CPU 32 comprehensively controls each part of the computer. The CPU 32 is an example of a "processor" according to the technology of the present disclosure. The memory 31 may be built into the CPU 32.

[0039] The communication unit 33 controls the transmission of various information to and from external devices. The display 34 displays various screens. Each screen is equipped with an operation function using a GUI (Graphical User Interface). The computers constituting the training data creation device 11 and the learning device 12 accept operation instructions input from the input device 35 via the various screens. The input device 35 is a keyboard, mouse, touch panel, microphone for voice input, etc. In the following description, the symbols of the parts of the training data creation device 11 are distinguished by adding the suffix "X" to the symbols, and the symbols of the parts of the learning device 12 are distinguished by adding the suffix "Y" to the symbols.

[0040] As an example, as shown in FIG. 5 , an operating program 40 is stored in the storage 30X of the training data creation device 11. The operating program 40 is an application program for causing a computer to function as the training data creation device 11. In other words, the operating program 40 is an example of an "operating program for an information processing device" according to the technology of the present disclosure. The storage 30X also stores a medical document group 14G and a training data group 15G. The storage 30X also stores an extraction model 41, relevance information 42, and selection conditions 43.

[0041] When the operating program 40 is started, the CPU 32X of the computer constituting the learning data creation device 11, in cooperation with the memory 31 and the like, functions as an acquisition unit 50, a read / write (hereinafter abbreviated as RW (Read Write)) control unit 51, a pre-processing unit 52, an extraction unit 53, a derivation unit 54, a creation unit 55, and a distribution control unit 56.

[0042] The acquiring unit 50 acquires the medical document group 14G from the medical document DB server 13. The acquiring unit 50 outputs the medical document group 14G to the RW control unit 51.

[0043] The RW control unit 51 controls the storage of various data in the storage 30X and the reading of various data from the storage 30X. For example, the RW control unit 51 stores the medical document group 14G from the acquisition unit 50 in the storage 30X. The RW control unit 51 also reads the medical document group 14G from the storage 30X and outputs the medical document group 14G to the preprocessing unit 52.

[0044] The RW control unit 51 reads the extraction model 41 from the storage 30X and outputs the extraction model 41 to the extraction unit 53. The RW control unit 51 also reads the relevance information 42 from the storage 30X and outputs the relevance information 42 to the derivation unit 54. The RW control unit 51 also reads the selection conditions 43 from the storage 30X and outputs the selection conditions 43 to the creation unit 55.

[0045] The preprocessing unit 52 performs preprocessing on each sentence of each medical document 14 that constitutes the medical document group 14 G. The preprocessing is a process of converting the sentence into data that can be handled by the extraction model 41.

[0046] The extraction unit 53 extracts multiple named entities from the medical document 14 using the extraction model 41. The extraction model 41 is a model created by applying fine tuning adapted to NER to an existing pre-trained natural language processing model that is different from the pre-trained natural language processing model 19 and that is created using a normal method of randomly selecting target phrases regardless of relevance. The extraction unit 53 outputs the named entity extraction result 60 to the derivation unit 54. The named entity is an example of a "specific phrase" according to the technology of the present disclosure.

[0047] Here, "named entity (NE)" is generally a general term for proper nouns such as people's names and place names, date expressions, time expressions, etc. In this example, "named entities" are specific words and phrases used in medical documents 14, and more specifically, words and phrases that represent anatomical locations, quantities, names of lesions, characteristics of lesions, and names of diseases (see Figure 7).

[0048] The derivation unit 54 uses the relevance information 42 to derive a degree of relevance indicating the degree of relevance between the multiple named entities extracted by the extraction unit 53. The derivation unit 54 outputs a derivation result 61 of the relevance to the creation unit 55.

[0049] The creation unit 55 selects target phrases to be masked based on the relevance so as to satisfy the selection condition 43, and creates training input sentences 16 and correct answer data 17, i.e., training data 15. The creation unit 55 creates multiple pieces of training data 15. The creation unit 55 outputs a training data group 15G, which is a collection of the multiple pieces of training data 15, to the RW control unit 51. The RW control unit 51 stores the training data group 15G in the storage 30X.

[0050] The RW control unit 51 reads out the learning data group 15G from the storage 30X and outputs the learning data group 15G to the distribution control unit 56. The distribution control unit 56 controls the distribution of the learning data group 15G to the learning device 12.

[0051] As an example, as shown in FIG. 6, the preprocessing performed by the preprocessing unit 52 on each sentence in the medical document 14 includes tokenization and vectorization. The tokenization process is a process of dividing the original text 18 contained in the medical document 14 into tokens such as "liver," "recognition," "early thickening," "," and "masu" to create token data 65. In FIG. 6, morphemes (including punctuation marks) are shown as examples of tokens. Each token divided by this tokenization process becomes a candidate for a target phrase to be selected by the creation unit 55.

[0052] The vectorization process is a process of converting each token of the token data 65 into a multidimensional, for example, 64-dimensional vector V1, V2, V3, ..., to create vector data 66. The preprocessing unit 52 outputs the vector data 66 to the extraction unit 53 as the aforementioned "data that can be handled by the extraction model 41."

[0053] As an example, as shown in FIG. 7 , the extraction unit 53 uses the extraction model 41 to extract words and phrases representing anatomical location, quantity, lesion name, lesion characteristics, and disease name as named entities from the medical document 14. The extraction unit 53 also labels the extracted named entities by type. The types of named entities are anatomical location, quantity, lesion name, lesion characteristics, and disease name. FIG. 7 shows an example in which “liver / S3,” “diameter / 6 cm / large,” “mass,” “early thickening,” “washout,” and “HCC” are extracted as named entities. The example also shows an example in which “liver / S3” is labeled as the anatomical location, “diameter / 6 cm / large” as the quantity, “mass” as the lesion name, “early thickening” and “washout” as the characteristics of the lesion, and “HCC” as the disease name. While the extraction model 41 actually handles vector data 66, FIG. 7 illustrates token data 65 for ease of explanation.

[0054] As an example, as shown in Fig. 8, the relevance information 42 registers a relevance for each combination of two types of named entities. The relevance is set in advance by an operator of the learning data creation device 11 based on the knowledge and experience acquired up to that point. For example, the combination of "characteristics of lesion-name of disease" is assigned the highest relevance value of 10. On the other hand, the combination of "quantity-characteristics of lesion" is assigned the lowest relevance value of 0.

[0055] As an example, as shown in Figure 9, a derived result 61 is a registered relevance derived using the relevance information 42 for each combination of two named entities. Figure 9 illustrates the derived result 61 for the sentence illustrated in Figure 7. For example, a relevance of 8 is registered for the combination of "liver S3-mass," which is a combination of "anatomical location-lesion name." Furthermore, a relevance of 10 is registered for the combination of "early dense-HCC" and "washout-HCC," which are combinations of "lesion characteristics-disease name."

[0056] As an example, as shown in FIG. 10 , the selection conditions 43 include the following three conditions. The first condition is to mask 15% of the tokens in a sentence. As shown in the example of FIG. 7 , if the number of tokens in a sentence is 25, 25 × 0.15 = 3.75, and therefore, the first condition is met by masking at least four tokens. The second condition is to select at least one named entity as the target phrase. In the example shown in FIG. 7 , the second condition is met by selecting at least one of “liver / S3,” “diameter / 6 cm / large,” “mass,” “early thickening,” “washout,” and “HCC” as the target phrase. The third condition is to exclude from the candidate target phrase at least one named entity with a relevance of 8 or more with the phrase selected as the target phrase. The third condition, “relevance of 8 or more,” is an example of a “preset condition” according to the technology of the present disclosure.

[0057] FIG. 11 illustrates a case in which the named entity "HCC" is selected as the target phrase in the example shown in FIG. 7. In this case, according to the derived result 61, the named entities having a degree of association with "HCC" of 8 or more are "tumor mass," "early thickening," and "washout." Therefore, in accordance with the third condition, the creation unit 55 excludes "tumor mass," "early thickening," and "washout" from the candidate target phrases. In other words, the creation unit 55 does not select "tumor mass," "early thickening," and "washout" as the target phrases, but leaves them in the training input sentence 16. On the other hand, the creation unit 55 does not exclude "liver / S3" and "diameter / 6cm / large," which are named entities having a degree of association with "HCC" of less than 8, from the candidate target phrases. Therefore, in FIG. 11, "diameter" and "large" in "diameter / 6cm / large" are selected as the target phrases. It should be noted that "HCC" is an example of a "target phrase" and a "phrase selected as a target phrase" according to the technology of the present disclosure.

[0058] 12 is a diagram summarizing the processing of the training data creation device 11. In the training data creation device 11, first, the extraction unit 53 extracts named entities from each sentence of the medical document 14. Then, the derivation unit 54 derives the degrees of association between the named entities. Finally, the creation unit 55 selects a target phrase based on the degrees of association. Specifically, at least one named entity with a degree of association of 8 or more with the phrase selected as the target phrase is excluded from the candidate target phrases. In this way, training input sentences 16 and correct answer data 17, i.e., training data 15, are created. 12 shows an example in which training input sentence P-1 and correct answer data P-1, training input sentence P and correct answer data P, and training input sentence P+1 and correct answer data P+1 (P is an integer from 2 to Q, and Q is the total number of training input sentences 16) are created from sentence J-1, sentence J, and sentence J+1 (J is an integer from 2 to N) out of N sentences 1 to N in medical document 14. Note that the training input sentences 16 also include a certain number of sentences created by a normal method of randomly selecting target phrases regardless of relevance.

[0059] As an example, as shown in Figure 13, an operating program 70 is stored in storage 30Y of learning device 12. The operating program 70 is an application program for causing a computer to function as learning device 12. In other words, like operating program 40, operating program 70 is an example of an "operating program for an information processing device" according to the technology of the present disclosure. Storage 30Y also stores a natural language processing model 10 and the like.

[0060] When the operating program 70 is started, the CPU 32Y of the computer constituting the learning device 12 functions as an acquisition unit 75, a RW control unit 76, a pre-learning unit 77, and a fine tuning unit 78 in cooperation with the memory 31 and the like.

[0061] The acquiring unit 75 acquires the training data group 15G from the training data generating device 11. The acquiring unit 75 outputs the training data group 15G to the RW control unit .

[0062] The RW control unit 76 controls the storage of various data in the storage 30Y and the reading of various data from the storage 30Y. For example, the RW control unit 76 stores the training data group 15G from the acquisition unit 75 in the storage 30Y. The RW control unit 76 also reads the training data group 15G from the storage 30Y and outputs the training data group 15G to the pre-training unit 77. The RW control unit 76 also reads the natural language processing model 10 from the storage 30Y and outputs the natural language processing model 10 to the pre-training unit 77.

[0063] The pre-training unit 77 pre-trains the natural language processing model 10 using the training data 15, and converts the natural language processing model 10 into a pre-trained natural language processing model 19. The pre-training unit 77 outputs the pre-trained natural language processing model 19 to the RW control unit 76. The RW control unit 76 stores the pre-trained natural language processing model 19 in the storage 30Y.

[0064] The RW control unit 76 reads the pre-trained natural language processing model 19 from the storage 30Y and outputs the pre-trained natural language processing model 19 to the fine tuning unit 78. The fine tuning unit 78 performs fine tuning on the pre-trained natural language processing model 19 and sets the pre-trained natural language processing model 19 as a task adaptation model 20. The fine tuning unit 78 outputs the task adaptation model 20 to the RW control unit 76. The RW control unit 76 stores the task adaptation model 20 in the storage 30Y.

[0065] The task adaptation model 20 is distributed to the operation device by a distribution control unit (not shown) and used in the operation device. Alternatively, the learning device 12 may function as the operation device and use the task adaptation model 20 in the learning device 12.

[0066] As an example, as shown in FIG. 14 , in pre-learning, a training input sentence 16 is input to a natural language processing model 10. The natural language processing model 10 outputs a training prediction result 80 for the training input sentence 16. A loss calculation is performed on the natural language processing model 10 using a loss function based on the training prediction result 80 and ground truth data 17. Then, the values ​​of various parameters of the natural language processing model 10 are updated according to the result of the loss calculation, and the natural language processing model 10 is updated according to the updated settings. The training input sentence 16 is converted into vector data 66 in the natural language processing model 10. The training prediction result 80 is a predicted phrase of a masked phrase expressed as vector data 66.

[0067] In pre-training, the above series of processes, including input of training input sentences 16 to the natural language processing model 10, output of training prediction results 80 from the natural language processing model 10, loss calculation, update setting, and update of the natural language processing model 10, are repeatedly performed while exchanging training data 15. The repetition of the above series of processes is terminated when the prediction accuracy of the training prediction results 80 for the supervised data 17 reaches a predetermined set level. The natural language processing model 10 whose prediction accuracy has thus reached the set level is treated as a pre-trained natural language processing model 19. Note that pre-training may be terminated when the above series of processes has been repeated a set number of times, regardless of the prediction accuracy of the training prediction results 80 for the supervised data 17.

[0068] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 15 and 16. First, when the operating program 40 is started in the training data creation device 11, the CPU 32X of the training data creation device 11 functions as an acquisition unit 50, an RW control unit 51, a preprocessing unit 52, an extraction unit 53, a derivation unit 54, a creation unit 55, and a distribution control unit 56, as shown in Figure 5.

[0069] The operator of the learning data creation device 11 transmits a request for distribution of the medical document group 14G to the medical document DB server 13. As shown in Fig. 15, the medical document group 14G distributed from the medical document DB server 13 in response to this distribution request is acquired by the acquisition unit 50 (step ST100). The medical document group 14G is output from the acquisition unit 50 to the RW control unit 51 and stored in the storage 30X under the control of the RW control unit 51.

[0070] The medical document group 14G is read from the storage 30X by the RW control unit 51. The medical document group 14G is output from the RW control unit 51 to the preprocessing unit 52. Then, as shown in FIG. 6, preprocessing including tokenization processing and vectorization processing is performed on each sentence of the medical document 14 in the preprocessing unit 52 (step ST110).

[0071] 7, the extraction unit 53 extracts a plurality of named entities from the medical document 14 using the extraction model 41 (step ST120). The extraction result 60 of the named entities is output from the extraction unit 53 to the derivation unit 54.

[0072] In the derivation unit 54, the relevance information 42 shown in Fig. 8 is used to derive the relevance between a plurality of named entities, as in the derivation result 61 shown in Fig. 9 (step ST130). The derivation result 61 is output from the derivation unit 54 to the creation unit 55.

[0073] 11 based on the relevance so as to satisfy the selection condition 43 shown in Fig. 10, and training input sentences 16 and, in turn, training data 15 are generated (step ST140). More specifically, 15% of the tokens in one sentence are masked, at least one named entity is selected as the target phrase, and named entities having a relevance of 8 or more with the phrase selected as the target phrase are excluded from the candidates for the target phrase. The training data group 15G is output from the generation unit 55 to the RW control unit 51 and stored in storage 30X under the control of the RW control unit 51.

[0074] The RW control unit 51 reads the learning data group 15G from the storage 30X. The learning data group 15G is output from the RW control unit 51 to the distribution control unit 56. Under the control of the distribution control unit 56, the learning data group 15G is distributed to the learning device 12 (step ST150).

[0075] When the operating program 70 is started in the learning device 12, the CPU 32Y of the learning device 12 functions as an acquisition unit 75, a RW control unit 76, a pre-learning unit 77, and a fine tuning unit 78, as shown in FIG.

[0076] The operator of the learning device 12 transmits a distribution request for the learning data group 15G to the learning data creation device 11. As shown in Fig. 16, the learning data group 15G distributed from the learning data creation device 11 in response to the distribution request is acquired by the acquisition unit 75 (step ST200). The learning data group 15G is output from the acquisition unit 75 to the RW control unit 76 and stored in the storage 30Y under the control of the RW control unit 76.

[0077] The RW control unit 76 reads out the training data group 15G and the natural language processing model 10 from the storage 30Y. The training data group 15G and the natural language processing model 10 are output from the RW control unit 76 to the pre-training unit 77. Then, as shown in FIG. 14, the pre-training unit 77 pre-trains the natural language processing model 10 using the training data 15. As a result, the natural language processing model 10 becomes a pre-trained natural language processing model 19 (step ST210). The pre-trained natural language processing model 19 is output from the pre-training unit 77 to the RW control unit 76 and stored in the storage 30Y under the control of the RW control unit 76.

[0078] The RW control unit 76 reads the pre-trained natural language processing model 19 from the storage 30Y. The pre-trained natural language processing model 19 is output from the RW control unit 76 to the fine tuning unit 78. Then, the fine tuning unit 78 performs fine tuning on the pre-trained natural language processing model 19. As a result, the pre-trained natural language processing model 19 is used as the task adaptation model 20 (step ST220). The task adaptation model 20 is output from the fine tuning unit 78 to the RW control unit 76 and stored in the storage 30Y under the control of the RW control unit 76.

[0079] As described above, the CPU 32X of the training data creation device 11 includes an extraction unit 53, a derivation unit 54, and a creation unit 55. The extraction unit 53 extracts multiple named entities, which are specific phrases, from the medical documents 14. The derivation unit 54 derives relevance values ​​indicating the degree of relevance between the multiple named entities. The creation unit 55 selects target phrases, which are phrases to be masked, based on the relevance values. This makes it possible to create training input sentences 16 that contribute to effective training of the natural language processing model 10, compared to typical methods that randomly select target phrases regardless of relevance, and the method of Non-Patent Document 1, which does not consider relevance and may mask phrases necessary for predicting target phrases.

[0080] The creation unit 55 excludes named entities whose relevance to the word selected as the target word satisfies a preset condition from the candidates for the target word. This allows named entities necessary for predicting the word selected as the target word to remain in the training input sentence 16. This reduces the risk that it will be difficult to predict the masked word, which ultimately contributes to more effective training of the natural language processing model 10.

[0081] The derivation unit 54 derives the relevance using the relevance information 42 in which the relevance corresponding to the combination of the types of named entities is preset. Therefore, once named entities are extracted and the type of each named entity is labeled, the relevance can be easily derived from the relevance information 42. Furthermore, when the operator sets the relevance in the relevance information 42 as in this example, the operator's thoughts can be reflected in the creation of the training input sentences 16.

[0082] The document is a medical document 14, and the types of named entities include anatomical location, quantity, lesion name, lesion characteristics, and disease name. It is difficult to obtain large quantities of medical documents 14, which are the documents from which training input sentences are generated. This increases the risk of missing learning opportunities for important terms and preventing effective pre-learning. Therefore, by using medical documents 14 as the documents and setting the types of named entities to anatomical location, quantity, lesion name, lesion characteristics, and disease name as in this example, it is possible to more effectively generate training input sentences 16 that contribute to effective training of the natural language processing model 10. The types of named entities may include at least one of anatomical location, quantity, lesion name, lesion characteristics, and disease name.

[0083] The pre-learning unit 77 of the learning device 12 pre-learns the natural language processing model 10 using the training input sentences 16 created as described above by the training data creation device 11. This allows the natural language processing model 10 to be effectively trained.

[0084] [Second embodiment] In the first embodiment, the relevance is derived using the relevance information 42, but this is not limiting. For example, the relevance may be derived using a derived model 85 as shown in FIG.

[0085] In FIG. 17 , a derivation unit 86 of the second embodiment derives a degree of relevance using a derived model 85 and outputs the derivation result 87 to the creation unit 55. The derived model 85 is stored in a storage 30X. The derived model 85 is a model created by applying fine tuning adapted to the derivation of a degree of relevance to a pre-trained natural language processing model 88. The pre-trained natural language processing model 88 is a model different from the pre-trained natural language processing model 19. The pre-trained natural language processing model 88 is a model created by training using training input sentences 16 created by a normal method of randomly selecting target phrases regardless of their degree of relevance. The derived model 85 is an example of a "trained derived model" according to the technology of the present disclosure. Furthermore, the pre-trained natural language processing model 88 is an example of a "trained natural language processing model" according to the technology of the present disclosure.

[0086] As an example, as shown in FIG. 18 , a combination of named entities extracted by the extraction unit 53 is input to the derivation model 85 as input data 90. The derivation model 85 outputs output data 91 in response to the input data 90. The output data 91 is a score representing the degree of association between the named entities in the input data 90, expressed as a numerical value between 0 and 1. The derivation unit 86 registers the score of the output data 91 as the degree of association in the derivation result 87. FIG. 18 illustrates an example in which the combination of “early dense staining-HCC” is input to the derivation model 85 as the input data 90, and 0.98 is output as the output data 91. In this case, the third condition of the selection condition 43 is, for example, to exclude at least one named entity having a degree of association of 0.8 or more with the term selected as the target term from the candidate target terms.

[0087] As described above, in the second embodiment, the derivation unit 86 derives the relevance using the trained derivation model 85. Therefore, as long as the named entity is extracted, the relevance can be easily derived using the derivation model 85. Furthermore, unlike the relevance information 42, it is possible to eliminate the need for an operator to set the relevance. Furthermore, the derivation model 85 is created based on the pre-trained natural language processing model 88. Therefore, the derivation model 85 can be easily created.

[0088] [Third embodiment] In the third embodiment shown in FIGS. 19 to 25, sentences of the same type as sentences containing a phrase selected as a target phrase are determined, and the sentences of the same type as sentences containing the target phrase are set as learning input sentences 16.

[0089] [3_1 embodiment] 19 shows a derived result 61 for an original text 18 consisting of four sentences: "A mass measuring 6 cm in diameter was found in liver S3. It showed early enhancement and washout. HCC is suspected. There was no significant change in the cyst in the right kidney." In this case, in addition to "liver S3," "early enhancement," and "HCC" in the first embodiment, the named entities "right kidney," "cyst," and "significant change" are extracted, and the relevance of "right kidney," "cyst," and "significant change" is also derived.

[0090] FIG. 20 illustrates a case where the named entity “HCC” is selected as the target phrase, similar to the example shown in FIG. 11 . In this case, according to the derived result 61, the named entities with a relevance of 8 or more to “HCC” are “tumor mass,” “early thickening,” and “washout.” Therefore, the creation unit 55 excludes “tumor mass,” “early thickening,” and “washout” from the candidate target phrases. The creation unit 55 also determines that “A tumor measuring 6 cm in diameter was found in liver S3.” and “Early thickening and washout were observed.”, which contain “tumor mass,” “early thickening,” and “washout,” are similar to the sentence “HCC is suspected.”, which contains “HCC.” The creation unit 55 then determines “HCC is suspected.”, “A tumor measuring 6 cm in diameter was found in liver S3.”, and “Early thickening and washout were observed.” as training input sentences 16 to be input simultaneously to the natural language processing model 10. "HCC is suspected." is an example of a "sentence including a phrase selected as a target phrase" and a "sentence including a target phrase" according to the technology of the present disclosure. Also, "A mass measuring 6 cm in diameter was found in liver S3." and "It showed early staining and washout." are examples of "similar sentences" according to the technology of the present disclosure.

[0091] FIG. 21 is a diagram summarizing the processing of the training data creation device in the third embodiment. In the training data creation device of the third embodiment, as in the first embodiment, first, the extraction unit 53 extracts named entities from each sentence of the medical document 14. Then, the derivation unit 54 derives the degrees of association between the named entities. Finally, the creation unit 55 selects a target phrase based on the degrees of association. Specifically, at least one named entity having a degree of association of 8 or more with the phrase selected as the target phrase is excluded from the candidate target phrases. The creation unit 55 also determines sentences of the same type as the sentences containing the phrase selected as the target phrase, and sets the sentences of the same type as the sentences containing the target phrase as training input sentences 16. In this way, the training input sentences 16 and correct answer data 17, i.e., the training data 15, are created. Figure 21 shows an example in which, out of N sentences from sentences 1 to N in medical document 14, training input sentence P-1 and correct answer data P-1 are created from sentence J-1, and training input sentence P and correct answer data P are created from sentences J and J+1.

[0092] [Third embodiment] As an example, as shown in FIG. 22, the CPU of the training data creation device of the third embodiment functions as a determination unit 100 in addition to the units 50 to 56 of the first embodiment (only the preprocessing unit 52 and the creation unit 55 are shown in FIG. 22). The determination unit 100 uses a determination model 101 stored in the storage 30X to determine which organ a sentence in the medical document 14 describes. The determination unit 100 outputs a determination result 102 to the creation unit 55. Although not shown, the determination model 101 is a model created by fine-tuning adapted to organ determination on a pre-trained natural language processing model that has been pre-trained using training input sentences 16 created by a normal method of randomly selecting target phrases regardless of their relevance, similar to the derived model 85.

[0093] As an example, as shown in FIG. 23, the determination result 102 is obtained by labeling each sentence of the medical document 14 with an organ. Similar to the case of the third embodiment, FIG. 23 illustrates an original text 18 consisting of four sentences: "A mass measuring 6 cm in diameter was found in the liver S3. It showed early staining and washout. HCC is suspected. There is no significant change in the cyst in the right kidney." This original text 18 is then illustrated with an example in which organs are labeled. That is, "A mass measuring 6 cm in diameter was found in the liver S3," "It showed early staining and washout," and "HCC is suspected." are labeled with "liver" as the organ. On the other hand, "There is no significant change in the cyst in the right kidney." is labeled with "kidney" as the organ. That is, in this case, the determination unit 100 determines that all three sentences, "A mass measuring 6 cm in diameter was found in liver S3," "Shows early staining and washout," and "HCC is suspected," are sentences describing the liver. Note that the determination model 101 may also determine that the organ is unknown. In such a case, the organ is not labeled in a sentence determined to be unknown.

[0094] FIG. 24 illustrates a case where the named entity “HCC” is selected as the target phrase, similar to the example shown in FIG. 11 and other figures. In this case, according to the derived result 61, the named entities with a relevance of 8 or more to “HCC” are “tumor mass,” “early thickening,” and “washout.” Therefore, the creation unit 55 excludes “tumor mass,” “early thickening,” and “washout” from the candidate target phrases. Furthermore, according to the determination result 102, the sentences describing the same liver as “HCC suspected,” which is a sentence containing “HCC,” are “A tumor measuring 6 cm in diameter was found in liver S3.” and “Early thickening was observed and washout was present.” Therefore, the creation unit 55 selects “HCC suspected,” “A tumor measuring 6 cm in diameter was found in liver S3,” and “Early thickening was observed and washout was present” as training input sentences 16 to be input simultaneously to the natural language processing model 10. "HCC is suspected." is an example of a "sentence containing a phrase selected as a target phrase" and a "sentence containing a target phrase" according to the technology of the present disclosure. "A mass measuring 6 cm in diameter was found in liver S3." and "It showed early staining and washout." are examples of a "similar sentence" and a "sentence describing the same organ as the organ related to the phrase selected as a target phrase" according to the technology of the present disclosure.

[0095] FIG. 25 is a diagram summarizing the processing of the training data creation device in the third embodiment. In the training data creation device of the third embodiment, as in the first embodiment, first, the extraction unit 53 extracts named entities from each sentence in the medical document 14. Then, the derivation unit 54 derives the degrees of association between the named entities. Furthermore, in the training data creation device of the third embodiment, the determination unit 100 determines which organ each sentence in the medical document 14 describes. Finally, the creation unit 55 selects a target phrase based on the degrees of association. Specifically, at least one named entity having a degree of association of 8 or more with the phrase selected as the target phrase is excluded from the candidate target phrases. Furthermore, the creation unit 55 sets sentences containing the target phrase and sentences of the same type as the target phrase as training input sentences 16. In this way, the training input sentences 16 and correct answer data 17, i.e., the training data 15, are created. Figure 25 shows an example in which, out of N sentences from sentences 1 to N in medical document 14, learning input sentence P-1 and correct answer data P-1 are created from sentence J-1, whose organ is determined to be R, and learning input sentence P and correct answer data P are created from sentences J and J+1, whose organs are both determined to be S.

[0096] As described above, in the third embodiment, the CPU of the training data creation device determines sentences of the same type as sentences containing the target phrase. Then, the sentences containing the target phrase and the sentences of the same type are set as training input sentences 16. Therefore, for example, it is possible to reduce the risk of creating training input sentences 16 containing only the single sentence "I suspect HCC," with "HCC" masked, making it difficult to predict the masked phrase. As a result, it is possible to contribute to more effective training of the natural language processing model 10.

[0097] The CPU of the training data creation device of the third embodiment determines that sentences containing named entities whose relevance to the target phrase satisfies a preset condition are the same type of sentences. Therefore, the named entities necessary for predicting the target phrase can be left in the training input sentences 16. This reduces the risk that it will be difficult to predict the masked phrase, which ultimately contributes to more effective training of the natural language processing model 10.

[0098] The document is a medical document 14, and the determination unit 100 determines that sentences describing the same organ as the organ related to the term selected as the target term are the same type of sentence. Therefore, sentences highly related to sentences containing the target term can be used as learning input sentences 16 to be input all at once to the natural language processing model 10. Note that the same type of sentences may also be sentences describing the same disease name as the disease name related to the term selected as the target term.

[0099] [Fourth embodiment] As an example, in the fourth embodiment shown in FIGS. 26 and 27, the following series of processing cycles are repeated.

[0100] In the fourth embodiment, first, a first process is performed to create an extraction model 110 that extracts named entities from the natural language processing model 10, a derivation model 111 that derives the relevance between named entities, and a determination model 112 that determines which organ a sentence in a medical document 14 describes. Next, a second process is performed to create training documents 113 from which named entities are extracted and in which the relevance and organs are labeled using the extraction model 110, the derivation model 111, and the determination model 112. Then, a third process is performed to create training input sentences 16 from the training documents 113 created in the second process using a technique according to the technology disclosed herein. Furthermore, a fourth process is performed to pre-train the natural language processing model 10 using the training input sentences 16 created in the third process, thereby creating a pre-trained natural language processing model 19 (updating the natural language processing model 10). In the fourth embodiment, a cycle of the first to fourth processes is repeated.

[0101] 26 and 27, in the first cycle, the CPU of the training data creation device creates training input sentences 16_0 and supervised data 17_0 from medical documents 14 by a normal method of randomly selecting target phrases (step ST300). Training data 15_0, which is a set of training input sentences 16_0 and supervised data 17_0, is referred to as training data V0.

[0102] In the CPU of the learning device, a natural language processing model 10_0 (referred to as natural language processing model V0) is pre-trained using the training data V0 to obtain a pre-trained natural language processing model 19_0 (referred to as pre-trained natural language processing model V0) (step ST310). Then, by performing fine tuning on the pre-trained natural language processing model V0, an extraction model 110_0 (referred to as extraction model V0), a derived model 111_0 (referred to as derived model V0), and a determination model 112_0 (referred to as determination model V0) are created (step ST320). This completes the first process of the first cycle.

[0103] In the CPU of the training data creation device, named entities are extracted from the medical document 14 using the extraction model V0, the derivation model V0, and the determination model V0, and a training document 113_1 (referred to as training document V1) in which the relevance and organs are labeled is created (step ST330). This completes the second process of the first cycle.

[0104] Next, as shown in the first embodiment, a target phrase is selected based on the degree of association, and as shown in the third embodiment, a sentence containing the target phrase and a sentence of the same type are set as training input sentences 16_1, and training input sentences 16_1 and supervised answer data 17_1 are created from training document V1 (step ST340). This completes the third process of the first cycle. Training data 15_1, which is a set of training input sentences 16_1 and supervised answer data 17_1, is referred to as training data V1.

[0105] In the CPU of the learning device, the natural language processing model 10_1 (referred to as the natural language processing model V1) is pre-trained using the training data V1 to obtain a pre-trained natural language processing model 19_1 (referred to as the pre-trained natural language processing model V1) (step ST350). The natural language processing model V1 is nothing but the pre-trained natural language processing model V0. That is, in step ST350, the pre-trained natural language processing model V0 is re-trained using the training data V1. This completes the fourth process of the first cycle.

[0106] By performing fine tuning on the pre-trained natural language processing model V1, an extraction model 110_1 (referred to as extraction model V1), a derived model 111_1 (referred to as derived model V1), and a judgment model 112_1 (referred to as judgment model V1) are created (step ST360). This completes the first process of the second cycle.

[0107] In the CPU of the training data creation device, named entities are extracted from the medical document 14 using the extraction model V1, the derivation model V2, and the determination model V2, and a training document 113_2 (referred to as training document V2) in which the relevance and organs are labeled is created (step ST370). This completes the second process of the second cycle.

[0108] Next, as shown in the first embodiment, a target phrase is selected based on the degree of association, and as shown in the third embodiment, a sentence containing the target phrase and a sentence of the same type are set as training input sentences 16_2, and training input sentences 16_2 and supervised answer data 17_2 are created from training document V2 (step ST380). This completes the third process of the second cycle. Training data 15_2, which is a set of training input sentences 16_2 and supervised answer data 17_2, is referred to as training data V2.

[0109] In the CPU of the learning device, the natural language processing model 10_2 (referred to as natural language processing model V2) is pre-trained using the training data V2 to obtain a pre-trained natural language processing model 19_2 (referred to as pre-trained natural language processing model V2, not shown in FIG. 27) (step ST390). The natural language processing model V2 is nothing but the pre-trained natural language processing model V1. That is, in step ST390, the pre-trained natural language processing model V1 is re-trained using the training data V2. This completes the fourth process of the second cycle.

[0110] As described above, in the fourth embodiment, the CPU of the training data creation device and the CPU of the learning device create the extraction model 110, the derivation model 111, and the determination model 112 from the natural language processing model 10. Next, the extraction model 110, the derivation model 111, and the determination model 112 are used to create training documents 113 from which named entities are extracted and whose relevance and organs are labeled, and training input sentences 16 are created from the training documents 113. Finally, the natural language processing model 10 is trained using the created training input sentences 16, thereby updating the natural language processing model 10. This series of processing cycles is then repeated.

[0111] Therefore, as the cycle progresses, the performance of the extraction model 110, derivation model 111, and determination model 112 gradually improves, and accordingly, the accuracy of extracting named entities from the training documents 113 and the accuracy of the relevance and organ labeling also gradually improves. Therefore, the training input sentences 16 created from the training documents 113 gradually become more suitable for pre-training, and as a result, the performance of the pre-trained natural language processing model 19 can be gradually improved.

[0112] In the first cycle, the CPU of the learning device creates extraction models 110_0 and the like from pre-trained natural language processing models 19_0 created by learning using training input sentences 16 in which target phrases are randomly selected regardless of relevance. This eliminates the need to manually create training documents 113 from which fixed expressions have been extracted, thereby significantly reducing the amount of work required.

[0113] The CPU of the learning device further creates a derived model 111 that derives relevance from the pre-trained natural language processing model 19. The CPU of the training data creation device uses the derived model 111 to create training documents 113 labeled with relevance. This eliminates the need to manually label relevance. Furthermore, it is not necessary to use the relevance information 42 described in the first embodiment.

[0114] The document is a medical document 14, and the CPU of the learning device further creates a determination model 112 from the pre-trained natural language processing model 19 to determine which organ a sentence describes. The CPU of the training data creation device uses the determination model 112 to create training documents 113 in which organs are labeled. This eliminates the need to manually label organs. Note that the model created from the natural language processing model 10 in the first process may be only the extraction model 110.

[0115] In the above embodiments, the training input sentences 16 are created by a training data creation device, and the natural language processing model 10 is trained by a training device, but this is not limiting. The creation of the training input sentences 16 and the training of the natural language processing model 10 may be performed by a single computer.

[0116] The relevance level is not limited to the exemplary numerical values ​​of 1 to 10 or 0 to 1. It may also be a level such as high, medium, or low. Furthermore, while medical documents 14 are given as examples of documents, this is not limiting. Documents may also include novels, newspaper articles, or patent documents. Furthermore, documents in narrower fields, such as papers on COVID-19, are also acceptable. Furthermore, named entities are given as examples of specific terms, but this is not limiting. Keywords set by the operator may also be used.

[0117] In each of the above embodiments, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition units 50 and 75, the RW control units 51 and 76, the extraction unit 53, the derivation units 54 and 86, the creation unit 55, the distribution control unit 56, the pre-learning unit 77, the fine-tuning unit 78, and the determination unit 100. As described above, the various processors include the CPUs 32X and 32Y, which are general-purpose processors that execute software (operation programs 40 and 70) and function as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as programmable logic devices (PLDs) that are processors whose circuit configuration can be changed after manufacture, such as field programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs).

[0118] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0119] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by client and server computers. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by System on Chip (SoC). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0120] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0121] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0122] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0123] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0124] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0125] 2. Learning System 10 Natural Language Processing Models 11 Learning data creation device 12 Learning Device 13 Medical Document Database Server (Medical Document DB Server) 14 Medical Documents 14G Medical Document Group 15 Training data 15G training data set 16 Training input sentences 17 Correct data 18 Original text 19, 88 Pre-trained natural language processing models 20 Task Adaptation Model 30, 30X, 30Y Storage 31 memory 32, 32X, 32Y CPUs 33 Communications Department 34 Display 35 Input Devices 36 Bus Line 40, 70 operating programs 41, 110 Extraction model 42 Relevance Information 43 Selection criteria 50, 75 Acquisition Department 51, 76 Read / write control unit (RW control unit) 52 Pre-processing section 53 Extraction part 54, 86 Derivation part 55 Creation Department 56 Distribution control section 60 Extraction results 61, 87 Derived results 65 Token Data 66 Vector Data 77 Pre-Learning Section 78 Fine Tuning Section 80 Prediction results for training 85, 111 Derived model 90 Input Data 91 Output Data 100 Judgment section 101, 112 Decision Model 102 Judgment result 113 Study Documents ST100, ST110, ST120, ST130, ST140, ST150, ST200, ST210, ST220, ST300, ST310, ST320, ST330, ST340, ST350, ST360, ST370, ST380, ST390 Step

Claims

1. An information processing device that creates training input sentences from documents, the training input sentences being used to train a first natural language processing model, with some words masked, comprising: a processor; The processor: extracting a plurality of specific phrases from the document; deriving a degree of association indicating a degree of association between the plurality of specific words and phrases; selecting target phrases that are to be masked based on the relevance; Information processing device.

2. The processor: The information processing apparatus according to claim 1 , wherein the specific phrase whose degree of association with the phrase selected as the target phrase satisfies a preset condition is excluded from the candidates for the target phrase.

3. The processor: Identifying sentences of the same type as the sentences containing the phrase selected as the target phrase; 3. The information processing apparatus according to claim 1, wherein the sentences including the target phrase and the sentences of the same type are set as the learning input sentences.

4. The processor: The information processing apparatus according to claim 3 , wherein a sentence including the specific phrase whose degree of association with the phrase selected as the target phrase satisfies a preset condition is determined to be the same type of sentence.

5. the document is a medical document; The processor: The information processing apparatus according to claim 3 , wherein a sentence describing the same organ as an organ related to the term selected as the target term is determined to be the same type of sentence.

6. The processor: The information processing apparatus according to claim 1 , wherein the degree of association is derived using association information in which the degree of association is preset according to a combination of types of the specific phrases.

7. the document is a medical document; The information processing device according to claim 6 , wherein the type includes at least one of an anatomical location, a quantity, a name of a lesion, a property of a lesion, and a name of a disease.

8. The processor: The information processing device according to claim 1 , wherein the degree of association is derived using a trained derivation model.

9. The information processing device according to claim 8 , wherein the derived model is created based on a trained second natural language processing model.

10. The processor: The information processing apparatus according to claim 1 , wherein the first natural language processing model is trained using the training input sentences.

11. The processor: creating an extraction model that extracts at least the specific phrase from the first natural language processing model; Using the extraction model, a learning document is created from which the specific phrases are extracted; creating the training input sentences from the training documents; updating the first natural language processing model by training the first natural language processing model using the created training input sentences; The information processing apparatus according to claim 10, wherein the above cycle of processing is repeated.

12. The processor: The information processing device according to claim 11, wherein in the first cycle, the extraction model is created from the first natural language processing model created by learning using training input sentences in which the target phrases are randomly selected regardless of the relevance.

13. The processor: creating a derived model from the first natural language processing model to derive the relevance; The information processing apparatus according to claim 11 or 12, wherein the learning documents labeled with the relevance are created using the derived model.

14. the document is a medical document; The processor: a determination model is created from the first natural language processing model to determine which organ a sentence describes; The information processing device according to claim 11 , wherein the learning document in which the organ is labeled is created using the determination model.

15. 1. A method for operating an information processing device that creates, from a document, training input sentences used in training a natural language processing model, the training input sentences having some words masked, the method comprising: extracting a plurality of specific phrases from the document; Deriving a degree of association indicating the degree of association between the plurality of specific words and phrases; and A method for operating an information processing device, comprising: selecting a target phrase that is a phrase to be masked based on the relevance.

16. An operating program for an information processing device that creates, from a document, training input sentences used in training a natural language processing model, in which some words and phrases are masked, the training input sentences being: extracting a plurality of specific phrases from the document; Deriving a degree of association indicating the degree of association between the plurality of specific words and phrases; and An operating program for an information processing device that causes a computer to execute a process including selecting target phrases that are to be masked based on the relevance.

Images