Learning data generation device, learning data generation method and program, and learning device

Abstract

Provided are: a training data generation device that generates training data that improves the performance of a training model; a training data generation method and program; and a training device. According to the present invention, a first non-labeled image and a first text attached to the first non-labeled image are acquired, at least one piece of first information from among the region, type, and state of the first non-labeled image is acquired from the first non-labeled image, at least one piece of second information from among the region, type, and state of the first non-labeled image is acquired from the first text, an output obtained by inputting the first non-labeled image into a first training model is acquired as a first pseudo label of the first non-labeled image, and the first pseudo label is subjected to filtering on the basis of the first information and the second information.

Description

Technical Field

[0001] The present invention relates to a learning data generation device, a learning data generation method and program, and a learning device, and particularly relates to a technique for generating highly accurate learning data.

Background Art

[0002] In the learning of deep neural networks, it is known that the number of data has a great influence on performance. Although there are several large-scale datasets such as ImageNet and MS COCO, there are also fields where data is not sufficiently present, such as the medical field. As one countermeasure method in the case of insufficient data, Patent Document 1 discloses a method using pseudo labels. Specifically, unlabeled data is input into a model learned using labeled data, and the output is used as a pseudo label, and the model is learned again using the labeled data and the unlabeled data.

[0003] However, in the technique described in Patent Document 1, since output filtering is not performed, depending on the performance of the model learned with labeled data, a large amount of noise may be generated in the output, and the quality of the pseudo label may deteriorate.

[0004] On the other hand, Non-Patent Document 1 discloses a technique for filtering output results by SVM (Support Vector Machine).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] However, the technology described in Non-Patent Document 1 had the drawback of requiring a pre-trained SVM.

[0008] This invention has been made in view of these circumstances, and aims to provide a learning data generation device, a learning data generation method and program, and a learning device that generate learning data to improve the performance of a learning model. [Means for solving the problem]

[0009] One embodiment of a learning data generation device for achieving the above objective comprises at least one processor and at least one memory for storing instructions to be executed by the at least one processor, wherein the at least one processor acquires a first unlabeled image and first text attached to the first unlabeled image, acquires at least one first piece of information from the region, type, and state of the subject of the first unlabeled image from the first unlabeled image, acquires at least one second piece of information from the region, type, and state of the subject of the first unlabeled image from the first text, inputs the first unlabeled image to a first learning model that outputs a label relating to the region of interest of the image when an image is input, acquires the output of the first learning model as a first pseudo-label of the first unlabeled image, and filters the first pseudo-label based on the first and second pieces of information. According to this embodiment, it is possible to generate learning data that improves the performance of the learning model.

[0010] The first unlabeled image is a medical image, and the area, type, and condition of the subject are preferably the location of the lesion, the organ name, and the disease name, respectively. This embodiment is suitable for medical images, and the first pseudo-label can be filtered based on the information of the location of the lesion, the organ name, and the disease name.

[0011] Preferably, at least one processor inputs a first unlabeled image into an image recognition learning model that outputs at least one piece of information from the region, type, and state of the subject in the image when an image is input, and obtains the first piece of information. Also, preferably, at least one processor inputs a first piece of text into a text analysis learning model that outputs at least one piece of information from the region, type, and state of the text when text is input, and obtains the second piece of information. By using the learning models, the first and second pieces of information can be appropriately obtained.

[0012] It is preferable that at least one processor removes the first pseudolabel that differs from at least one of the first information and the second information. This leaves only the first pseudolabel that matches the first information and the second information.

[0013] In one embodiment of a learning device for achieving the above objective, it is preferable that the learning data generation device described above comprises at least one processor which trains a first learning model using labeled images and the labels of the labeled images as learning data, and trains a second learning model which outputs a label relating to a region of interest in an image when an image is input, using a first unlabeled image and a first pseudo-label (a first label) after filtering as learning data. According to this embodiment, the performance of the learning model can be improved.

[0014] The first and second learning models preferably include convolutional neural networks. Furthermore, the first and second learning models preferably include convolutional neural networks with the same configuration. According to this embodiment, the performance of the convolutional neural network can be improved.

[0015] When the initial value of the variable M is 1 and N is an integer of 2 or more, it is preferable that at least one processor obtains the Mth unlabeled image and the Mth text attached to the Mth unlabeled image, obtains at least one first piece of information about the Mth unlabeled image from the Mth unlabeled image, which is one of the region, type, and state of the subject of the Mth unlabeled image, obtains at least one second piece of information about the Mth unlabeled image from the Mth text, inputs the Mth unlabeled image into the Mth learning model, obtains the output of the Mth learning model as the Mth pseudo-label of the Mth unlabeled image, filters the Mth pseudo-label based on the first and second pieces of information about the Mth unlabeled image, trains the (M+1)th learning model using the Mth unlabeled image and the filtered Mth pseudo-label as training data, outputs a label about the region of interest in the image when an image is input, and adds 1 to M, repeating this process N times. This allows for the sequential generation of improved learning models.

[0016] One embodiment of a training data generation method for achieving the above objective is a training data generation method comprising: an image acquisition step of acquiring a first unlabeled image and a first text attached to the first unlabeled image; a first information acquisition step of acquiring at least one first piece of information from the region, type, and state of the subject of the first unlabeled image from the first unlabeled image; a second information acquisition step of acquiring at least one second piece of information from the region, type, and state of the subject of the first unlabeled image from the first text; a pseudo-label acquisition step of inputting the first unlabeled image to a first learning model that outputs a label relating to the region of interest of the image when an image is input, and acquiring the output of the first learning model as a first pseudo-label of the first unlabeled image; and a filtering step of filtering the first pseudo-label based on the first and second pieces of information. According to this embodiment, training data that improves the performance of the learning model can be generated.

[0017] One aspect of the program for achieving the above object is a program for causing a computer to execute the above learning data generation method. A computer-readable non-transitory storage medium on which this program is recorded may also be included in this aspect.

Advantages of the Invention

[0018] According to the present invention, learning data for improving the performance of a learning model can be generated.

Brief Description of the Drawings

[0019] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to the present embodiment. [Figure 2] FIG. 2 is a flowchart showing each step of the learning data generation method. [Figure 3] FIG. 3 is a diagram showing data processing in the learning data generation method. [Figure 4] FIG. 4 is a diagram showing details of data processing in the learning method. [Figure 5] FIG. 5 is a flowchart showing each step of the learning method. [Figure 6] FIG. 6 is a flowchart showing each step of the learning method.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail according to the accompanying drawings.

[0021] 〔Overall Configuration of Information Processing Apparatus〕 FIG. 1 is a block diagram of an information processing apparatus 10. The information processing apparatus 10 generates learning data for training a learning model that detects an object from an image. Further, the information processing apparatus 10 trains a learning model using the learning data. Here, an example of a learning model for extracting a lesion region from a medical image will be described. As shown in FIG. 1, the information processing apparatus 10 includes a processor 12 and a memory 14.

[0022] The processor 12 executes instructions stored in memory 14. The hardware structure of the processor 12 consists of various types of processors, as shown below. These types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute software (programs) and act as various functional units; GPUs (Graphics Processing Units), which are processors specialized for image processing; PLDs (Programmable Logic Devices), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing.

[0023] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, multiple functional units may be composed of a single processor. Examples of composing multiple functional units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, as is typical of computers such as client or server computers, and this processor acts as multiple functional units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple functional units, on a single IC (Integrated Circuit) chip, as is typical of SoCs (System On Chip). Thus, various functional units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0024] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0025] Memory 14 stores instructions for the processor 12 to execute. Memory 14 includes RAM (Random Access Memory) and ROM (Read Only Memory), which are not shown. The processor 12 uses RAM as a working area and executes software using various programs and parameters, including learning data generation methods, stored in ROM, and also uses parameters stored in ROM, etc., to execute various processes of the information processing device 10.

[0026] [Functional configuration of information processing equipment] As shown in Figure 1, the information processing device 10 includes a learning data generation device 16 and a learning device 18. The functions of the learning data generation device 16 and the learning device 18 are realized by the processor 12.

[0027] The learning data generation device 16 includes an image / text acquisition unit 20, a first learning model 22, an image recognition unit 24, a text analysis unit 26, and a filtering processing unit 28.

[0028] The image / text acquisition unit 20 acquires medical images and interpretation reports attached to the medical images. Medical images are images taken by medical imaging devices such as X-ray imaging devices, CT (Computed Tomography) devices, MRI (Magnetic Resonance Imaging) devices, PET (Positron Emission Tomography) devices, ultrasound devices, and CR (Computed Radiography) devices using a flat X-ray detector. An interpretation report is a report created by a radiologist after interpreting a medical image, and includes a report (an example of "text") which is textual information describing at least one piece of information from the lesion, organ, and disease of the subject of the medical image. Here, information about a lesion refers to information about the location where the lesion occurs (location of the lesion). The location of a lesion is a specific region contained within an organ; for example, if the organ is the brain, it refers to each part contained within the brain such as the frontal lobe, right frontal lobe, cerebrum, and cerebellum, the regions contained within each part, or coordinate information associated with each part and region. Information about organs refers to information indicating the name of the organ where the lesion is occurring, such as the brain, heart, and lungs. Information about diseases refers to information indicating the name or type of disease causing the lesion, such as cerebral infarction, myocardial infarction, pneumonia, tumors, and nodules. The medical images acquired by the image / text acquisition unit 20 include medical images with correct labels (labeled images) and medical images without correct labels (unlabeled images). The correct labels will be described later.

[0029] The first learning model 22 is a machine learning model that, when an image is input, outputs a label relating to the region of interest in the input image. The first learning model 22 may be a pre-trained model. The first learning model 22 is composed of, for example, a convolutional neural network (CNN). The first learning model 22 is trained using pairs of training images and ground truth labels containing information about lesions in the training images (ground truth dataset). The ground truth labels are manually created and include, for example, at least one piece of information indicating the location of the lesion in the training image, the name or type of organ in which the lesion occurs, and the name or type of disease causing the lesion. The information regarding the location of the lesion may be coordinate information of the lesion in the image, or it may be information indicating a specific region in which the lesion exists from among several regions pre-defined for each organ. Although the assigned ground truth labels are accurate, it is difficult to obtain a large number of images with ground truth labels. For this reason, the first learning model 22 is trained with relatively little training data and has relatively low accuracy.

[0030] The first learning model 22 receives medical images acquired by the image / text acquisition unit 20 as input. The first learning model 22 outputs labels related to the lesion area of ​​the input medical image. The labels output by the first learning model 22 are input to the filtering processing unit 28, described later, as first pseudo-labels. The labels related to the lesion area include information about at least one of the following in the image: the location of the lesion (location of the lesion), the name or type of organ in which the lesion occurs, and the name or type of disease causing the lesion. The information regarding the location of the lesion may be coordinate information of the lesion in the image, as described above, or it may be information indicating a specific region in which the lesion is located from among multiple regions pre-set for each organ.

[0031] When an image is input to the image recognition unit 24, it acquires at least one piece of first information from the region, type, and state of the subject based on the input image. The region information is information about the location where a lesion exists in the image (location of the lesion). The type information is information indicating the name or type of organ in which the lesion is occurring. The state information is information indicating the name or type of disease causing the lesion. Here, the image recognition unit 24 includes an image recognition learning model 24A. The image recognition learning model 24A is a learning model that, when a medical image is input, outputs at least one piece of first information from among the location of the lesion, the organ name, and the disease name based on the medical image. The image recognition learning model 24A is composed of, for example, a convolutional neural network.

[0032] The image recognition unit 24 may acquire the first information using known techniques described in Japanese Patent Publication No. 7-323024 and Japanese Patent Publication No. 7-031591.

[0033] When the text of the image interpretation report is input to the text analysis unit 26, it obtains at least one second piece of information from the input text, which is one of the following: region, type, and state. The region, type, and state of the second piece of information are the same as the region, type, and state of the first piece of information. That is, the region information is information about the location of the lesion. The type information is information indicating the name or type of organ in which the lesion is occurring. The state information is information indicating the name or type of disease causing the lesion. The text may be a sentence or a list of structured words. The text may also be image tag information. Here, the text analysis unit 26 includes a text analysis learning model 26A. When text is input to the text analysis learning model 26A, it obtains at least one second piece of information from the text, which is one of the following: location of the lesion, organ name, and disease name. The text analysis learning model 26A is composed of, for example, a neural network.

[0034] The text analysis unit 26 may obtain second information from the input text using a dictionary, or it may obtain second information from the input text using an If-Then rule. Alternatively, the text analysis unit 26 may obtain second information from the input text using the techniques described in Japanese Patent Publication No. 2013-200592 and Japanese Patent Publication No. 2012-63919, which apply known morphological analysis.

[0035] The filtering processing unit 28 acquires the label output by the first learning model 22 as the first pseudo-label. The filtering processing unit 28 also acquires first information from the image recognition unit 24. Furthermore, the filtering processing unit 28 acquires second information from the text analysis unit 26. The filtering processing unit 28 also filters the first pseudo-label based on the first and second information. For example, the filtering processing unit 28 deletes the first pseudo-label or information contained in the first pseudo-label that differs from at least one of the first and second pieces of information. In this way, the filtering processing unit 28 performs filtering on the first pseudo-label and generates the first label. That is, the first label is the pseudo-label that has been filtered by the filtering processing unit 28 from the first pseudo-label output by the first learning model 22. The first label is a pseudo-label with higher accuracy than the first pseudo-label.

[0036] Furthermore, the learning device 18 includes a second learning model 30. The second learning model 30 is a machine learning model that outputs a label relating to a region of interest in an image when an image is input. The second learning model 30 is composed of, for example, a convolutional neural network. The second learning model 30 may be a convolutional neural network with the same configuration as the first learning model 22. The learning device 18 inputs the ground truth dataset into the second learning model 30 to train the second learning model 30. In addition, the learning device 18 inputs a learning dataset of unlabeled medical images acquired by the image / text acquisition unit 20 and the first labels into the second learning model 30 to train the second learning model 30.

[0037] For example, the learning device 18 includes a loss value calculation unit (not shown) and a parameter control unit (not shown). When an unlabeled medical image from the learning dataset is input, the learning device 18 calculates the loss value between the pseudo-label (second pseudo-label) output from the second learning model 30 and the first label of the learning dataset. Furthermore, based on the calculated loss value, the learning device 18 adjusts the parameters of the second learning model 30 using backpropagation to maximize the similarity between the second pseudo-label output from the second learning model 30 and the label of the ground truth data.

[0038] [First Embodiment: Method for Generating Training Data] Figure 2 is a flowchart showing each step of the learning data generation method by the information processing device 10. Figure 3 is a diagram showing the data processing in the learning data generation method. The learning data generation method is realized by the processor 12 executing a learning data generation program stored in memory 14. The learning data generation program may be provided by a computer-readable non-temporary storage medium. In this case, the information processing device 10 may read the learning data generation program from the non-temporary storage medium and store it in memory 14.

[0039] In step ST1, the image / text acquisition unit 20 acquires a medical image via an input interface (not shown) (an example of the "image acquisition process"). Here, the image / text acquisition unit 20 acquires an unlabeled CT image (an example of the "first unlabeled image") and an interpretation report attached to the unlabeled CT image. An unlabeled CT image is an image taken with a CT scanner that does not have labels such as bounding boxes attached to the region of interest. The interpretation report also includes text (an example of the "first text").

[0040] In step ST1, the image / text acquisition unit 20 inputs the acquired unlabeled CT images to the first learning model 22 and the image recognition unit 24. The image / text acquisition unit 20 also inputs the text of the acquired image interpretation report to the text analysis unit 26.

[0041] In step ST2, the first learning model 22 outputs bounding boxes at the location of lesions in the input unlabeled CT image. Specifically, the first learning model 22 outputs text information to be written in the bounding boxes, associating it with the location information of lesions in the unlabeled CT image. The first learning model 22 may make the output more stable by averaging it through data augmentation (test-time augmentation) of the input, or it may increase its sensitivity by increasing the number of pseudo-labels. In step ST2, the filtering processing unit 28 acquires the bounding boxes output by the first learning model 22 as the first pseudo-labels of the unlabeled CT image (an example of the "first pseudo-label acquisition process").

[0042] Figure 3 shows CT image P1, an unlabeled CT image, the image interpretation report R1 attached to CT image P1, and the first pseudo-label L1 output by the first learning model 22. CT image P1 is the image used by the information processing device 10 to generate training data. Image interpretation report R1 is a report created by a radiologist after interpreting CT image P1, and is stored in association with CT image P1.

[0043] The first pseudo-label L1 is the output of the first learning model 22 to which the CT image P1 is input. Here, the first pseudo-label L1 includes bounding boxes B1 and B2. The first pseudo-label L1 only needs to have the positional information of bounding boxes B1 and B2 on the CT image P1, and does not need to have the image information of the CT image P1.

[0044] Returning to the explanation of Figure 2, in step ST3, the image recognition unit 24 inputs the CT image P1 to the image recognition learning model 24A and obtains at least one piece of first information from the CT image P1, which includes the location of the lesion, the name of the organ, and the name of the disease (an example of the "first information acquisition step"). In other words, the image recognition unit 24 obtains information from the CT image P1 that includes at least one of the coordinate location or region where the lesion exists, the name of the organ where the lesion exists, and the name of the disease causing the lesion.

[0045] In step ST4, the text analysis unit 26 inputs the text of the image interpretation report R1 into the text analysis learning model 26A and obtains at least one second piece of information from the text, which is the location of the lesion in the CT image P1, the name of the organ, and the name of the disease (an example of the "second information acquisition step").

[0046] In step ST5, the filtering processing unit 28 filters the first pseudo-label obtained in step ST2 based on the first information obtained in step ST3 and the second information obtained in step ST4, and obtains the first label, which is the first pseudo-label after filtering (an example of a "filtering process"). Specifically, the filtering processing unit 28 compares the first information and the second information with the information contained in the first pseudo-label, and if there is any information that does not match, it deletes the first pseudo-label or the bounding box information contained in the first pseudo-label. The remaining first pseudo-label or bounding box information that is not deleted becomes the first label. For example, the filtering processing unit 28 deletes pseudo-label or bounding box information that has been assigned to an impossible area as a result of considering the first information or the second information.

[0047] The filtering processing unit 28 may filter the first pseudo-labels based on at least one of the first information and the second information. In addition to the first and second information, the filtering processing unit 28 may also filter according to the confidence level of the first learning model 22 (the output value of the first learning model 22). The filtering here is rule-based. Furthermore, if the target of examination is limited to a specific organ, the filtering processing unit 28 may filter by organ name to exclude information on bounding boxes for organs other than the specific organ. In addition, if a lesion is detected outside the body (if a bounding box is attached outside the body), the filtering processing unit 28 may filter to exclude bounding boxes attached outside the body.

[0048] The filtering processing unit 28 may perform learning-based filtering, for example, by using a filtering method based on the research described in Non-Patent Document 2. In this case, the filtering processing unit 28 is given an image and text, outputs multiple bounding boxes from the image, matches them with the text, and associates them with the corresponding locations. Therefore, the filtering processing unit 28 can filter out bounding boxes that are not associated with text.

[0049] Figure 3 shows the first information I1 acquired by the image recognition unit 24, the second information I2 acquired by the text analysis unit 26, and the first label L2 which is the output of the filtering processing unit 28.

[0050] The first label L2 shown in Figure 3 has its bounding box B2 removed, which differs from at least one of the first information I1 and the second information I2, leaving only the bounding box B1.

[0051] The first label L2 and CT image P1 obtained in this way become the training dataset generated by the information processing device 10.

[0052] Here, the filtering processing unit 28 removed the bounding box B2 from the first pseudo-label L1 that differed from at least one of the first information and the second information. However, if the first pseudo-label L1 has a bounding box that differs from at least one of the first information and the second information, the first pseudo-label L1 and the unlabeled CT image P1 on which the first pseudo-label L1 was output do not need to be used as training data.

[0053] The first learning model 22 does not have high accuracy due to the small amount of training data, and may assign false labels to areas other than the lesion region. According to the first embodiment, by filtering the first false labels using the first information and the second information, the accuracy of the first false labels can be relatively improved, and accurate training data can be generated. Therefore, using the filtered first false labels (first labels) as the training dataset rather than the training dataset using the first false labels before filtering can improve the performance of the learning model.

[0054] [Second Embodiment] Figure 4 shows the details of data processing in the learning method. Parts common to both Figure 4 and Figure 3 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0055] The first piece of information I1 is the output of the image recognition unit 24 to which the CT image P1 has been input. The first piece of information I1 is added to bounding boxes B1 and B2, respectively. In the example shown in Figure 4, the information "liver S8 tumor" is added to the region of bounding box B1, and the information "lung S1 nodule" is added to the region of bounding box B2. Here, "S8" in "liver S8 tumor" indicates a region of the liver, and "S1" in "lung S1 nodule" indicates a region of the lung. Here, a region is a small area into which the whole of an organ is divided into several parts. That is, both the liver and the lungs each have multiple pre-defined regions, and S8 is one region included in the entire region of the liver, and S1 is also one region included in the entire region of the lung.

[0056] The second piece of information, I2, is the output of the text analysis unit 26 to which the image interpretation report R1 has been input. For example, if the image interpretation report R1 states "Hemangioma present in S8", the text analysis unit 26 will obtain "S8 hemangioma" as the second piece of information, I2. In the example shown in Figure 4, the second piece of information, I2, includes "Liver S8 mass".

[0057] The first label L2 is obtained by removing the bounding box B2 that differs from at least one of the first information I1 and the second information I2 from the first pseudo-label L1, leaving only the bounding box B1. Here, the information "liver S8 tumor" in the region of bounding box B1 matches the information "liver S8 tumor" in the second information I2, and is therefore adopted as the correct label. On the other hand, the information "lung S1 nodule" in the region of bounding box B2 differs from the information "liver S8 tumor" in the second information I2. Therefore, the information "lung S1 nodule" in bounding box B2 is judged to be an incorrect label and is deleted.

[0058] The first label L2 and CT image P1 obtained in this way become the training dataset.

[0059] If the image interpretation report R1 states "No abnormalities," the text analysis unit 26 obtains "No abnormalities" as the second piece of information I2. In this case, the filtering processing unit 28 deletes bounding boxes B1 and B2.

[0060] [Third Embodiment: Learning Method] Figure 5 is a flowchart showing each step of the learning method by the information processing device 10. Parts common to the flowchart in Figure 3 are denoted by the same reference numerals, and their detailed explanations are omitted.

[0061] In step ST11, the processor 12 trains the first learning model 22 using CT images with ground truth labels. CT images with ground truth labels are data in which bounding boxes are assigned to areas of interest, such as lesions. Here, the processor 12 inputs the learning dataset (ground truth dataset) of CT images with ground truth labels acquired by the image / text acquisition unit 20 and the ground truth labels of those CT images into the first learning model 22, and trains the first learning model 22.

[0062] For example, the processor 12 includes a loss value calculation unit (not shown) and a parameter control unit (not shown). When a CT image with a ground truth label from the training dataset is input to the processor 12, the processor 12 calculates a loss value between the first pseudo-label output from the first training model 22 and the ground truth label from the training dataset. Based on the calculated loss value, the processor 12 adjusts the parameters of the first training model 22 using backpropagation to maximize the similarity between the first pseudo-label output from the first training model 22 and the ground truth label.

[0063] Next, by processing steps ST1 to ST5 in the same manner as in the first embodiment, the first pseudo-labels output by the first learning model 22 are filtered with the first information and the second information to generate training data of pairs of unlabeled CT images and the first labels.

[0064] Finally, in step ST12, the learning device 18 trains the second learning model 30 using the pairs of labeled images and their corresponding labels used in step ST11 as the training dataset. Furthermore, the learning device 18 trains the second learning model 30 using the pairs of unlabeled CT images acquired in step ST1 and their corresponding first labels (the first pseudo-labels after filtering in step ST5) as the training dataset. Note that if only CT images containing lesions are used to train the second learning model 30, and the first label of a CT image does not have a bounding box indicating a lesion, that CT image will not be used to train the second learning model 30.

[0065] Here, the learning device 18 trains the second learning model 30, a CNN model with the same structure (number of layers, number of parameters) as the first learning model 22, from scratch (starting with weights initialized with random numbers). Note that the structure of the second learning model 30 may differ from the structure of the first learning model 22. The second learning model 30 may be retrained or fine-tuned by inheriting the parameters of the first learning model 22. Alternatively, the second learning model 30 may indirectly use the parameters of the first learning model 22. For example, the average value of the parameters of multiple layers of the first learning model 22 may be inherited as the parameter of one layer of the second learning model 30.

[0066] Alternatively, the process in step ST12 may be performed after obtaining a large number of training datasets by repeating the processes in steps ST1 to ST5.

[0067] Various variations are possible for the first method of utilizing pseudo-labels and the second method of initializing the learning model 30. For example, as described in Patent Document 1, the output distribution of the first learning model 22 can be used as the first pseudo-label. The same applies to confidence scores. The first learning model 22 can also be trained without using labeled data.

[0068] Unlabeled medical images without correct labels are readily available. Similarly, image interpretation reports accompanying medical images are also readily available. The first learning model 22 has relatively low accuracy in its output first pseudo-labels because it has limited training data. However, according to this embodiment, the first pseudo-labels are filtered, so the accuracy of the filtered labels is relatively improved. Therefore, the learning data generation device 16 can use a large number of medical images as training data, and the learning device 18 can improve the performance of the learning model with high-accuracy training data.

[0069] [Fourth Embodiment] It is also possible to repeatedly train multiple learning models. Figure 6 is a flowchart showing each step of the learning method by the information processing device 10. In the third embodiment, the second learning model 30 of the learning device 18 was trained using the first learning model 22 of the learning data generation device 16. On the other hand, in the fourth embodiment, although not shown in the figure, if M is an integer, the (M+1)th learning model of the learning device 18 is trained using the Mth learning model of the learning data generation device 16. Then, by incrementing the variable M, higher-order learning models are trained sequentially.

[0070] In step ST21, the processor 12 assigns 1 to the variable M. That is, the initial value of variable M is set to 1.

[0071] In step ST22, the processor 12 trains the Mth learning model using the Mth labeled medical images.

[0072] In step ST23, the image / text acquisition unit 20 acquires the Mth unlabeled medical image (an example of the "Mth unlabeled image") and the Mth reading report attached to the Mth unlabeled medical image (an example of the "Mth text"). The image / text acquisition unit 20 inputs the Mth unlabeled medical image into the Mth learning model that was learned in step ST22. The Mth learning model outputs bounding boxes at the location of the lesion area in the input Mth medical image. In step ST24, the processor 12 acquires the output of the Mth learning model as the Mth pseudo-label for the Mth unlabeled medical image.

[0073] In step ST25, the image recognition unit 24 inputs the Mth unlabeled medical image to the image recognition learning model 24A and obtains at least one of the following first pieces of information from the Mth unlabeled medical image: the location of the lesion, the organ name, and the disease name.

[0074] In step ST26, the text analysis unit 26 inputs the text of the Mth image interpretation report of the Mth unlabeled medical image into the text analysis learning model 26A, and obtains at least one second piece of information from the text, which is the location of the lesion, the organ name, and the disease name of the Mth unlabeled medical image.

[0075] In step ST27, the filtering processing unit 28 uses the first information of the Mth unlabeled image acquired in step ST25 and step ST2 6 Based on the second piece of information from the Mth unlabeled image obtained in step ST24, the pseudo-labels of the Mth obtained in step ST24 are filtered, and the label of the Mth, which is the filtered pseudo-label of the Mth, is obtained.

[0076] In step ST28, the learning device 18 has the Mth label used in step ST22. medical care The (M+1)th learning model is trained using the images. Furthermore, the learning device 18 trains the (M+1)th learning model using the Mth unlabeled medical image acquired in step ST23 and the Mth pseudo-labeled image filtered in step ST27 as the learning dataset. Note that the processing in step ST28 may be performed after obtaining a large number of learning datasets by repeating the processing in steps ST23 to ST27.

[0077] In step ST29, processor 12 adds 1 to variable M.

[0078] In step ST30, processor 12 determines whether or not to retrain the Mth learning model. For example, processor 12 determines whether the variable M has exceeded N, which is an integer greater than or equal to 2. That is, processor 12 repeats the process from steps ST23 to ST28 until it reaches N times. If retraining is to be performed, it returns to step ST23 and repeats the same process. If retraining is not to be performed, processor 12 terminates the process in this flowchart.

[0079] As described above, by filtering the pseudo-label of the Mth learning model, which is the output of the Mth learning model, to obtain the label of the Mth learning model, and using the label of the Mth learning model to train the (M+1)th learning model, and repeatedly increasing M, it is possible to ultimately generate a learning model with good accuracy.

[0080] In step ST23, the Mth unlabeled medical image input, and in step ST24, the Mth unlabeled medical image without a bounding box, cannot be filtered for the Mth pseudo-label in step ST27 and cannot be used as training data in step ST28. However, as the accuracy improves in higher-order learning models, it becomes possible to output the Mth pseudo-label even when such unlabeled medical images are input.

[0081] 〔others〕 In the above embodiment, the first information regarding the subject's condition was information about the disease name, but the first information regarding the subject's condition may also be information about the severity of the disease. For example, the severity of the disease may be information indicating whether it is severe, moderate, or mild. Furthermore, although the above embodiment describes an example of generating training data for medical images, the images handled by the information processing device 10 are not limited to medical images. That is, it is sufficient to obtain at least one first piece of information from an unlabeled image, consisting of the region, type, and state of the subject of the unlabeled image, and at least one second piece of information from the region, type, and state of the subject of the unlabeled image, consisting of the region, type, and state of the subject of the unlabeled image, from text information attached to the unlabeled image.

[0082] Furthermore, while we have used bounding boxes as an example to indicate the lesion area as pseudo-labels, the bounding boxes may also indicate the type of lesion. For example, bounding boxes that display tumors and nodules in different colors may also be used.

[0083] Furthermore, pseudo-labels are not limited to bounding boxes. For example, pseudo-labels may be segmentation indicating probability of existence, instance segmentation, or the result of converting segmentation results into bounding boxes. Segmentation may be segmentation indicating lesions and non-lesions, or segmentation that classifies down to the type of lesion.

[0084] The technical scope of the present invention is not limited to the scope described in the embodiments above. The configurations and other elements in each embodiment can be appropriately combined with those in each embodiment without departing from the spirit of the present invention. [Explanation of Symbols]

[0085] 10…Information Processing Devices 12… Processor 14…Memory 16…Training data generation device 18…Learning device 20… image / Text acquisition unit 22…First Learning Model 24…Image Recognition Unit 24A...Image recognition learning model 26…Text Analysis Department 26A...Text analysis learning model 28…Filtering Processing Unit 30…Second learning model B1...Bounding Box B2...Bounding Box I1…First piece of information I2...Second piece of information L1...First pseudo-label L2…First label P1...CT image R1...Image interpretation report ST1~ST5...Each step in the training data generation method ST11, ST12, ST21-ST30...Each stage of the learning method

Claims

1. At least one processor, At least one memory for storing instructions to be executed by the aforementioned at least one processor, Equipped with, The aforementioned at least one processor is Obtain a first unlabeled image and a first text attached to the first unlabeled image. From the first unlabeled image, obtain at least one piece of first information from the region, type, and state of the subject in the first unlabeled image. From the first text, obtain at least one second piece of information from the region, type, and state of the subject of the first unlabeled image. When an image is input, the first unlabeled image is input to a first learning model that outputs a label relating to the region of interest in the image, and the output of the first learning model is obtained as the first pseudo-label of the first unlabeled image. Based on the first information and the second information, the first pseudo-label is filtered. A device for generating training data.

2. The first unlabeled image is a medical image. The region, type, and condition of the subject are, respectively, the location of the lesion, the name of the organ, and the name of the disease. The learning data generation device according to claim 1.

3. The aforementioned at least one processor is The first unlabeled image is input to an image recognition learning model that, upon input of the aforementioned image, outputs at least one piece of information from the region, type, and state of the subject in the image, and the first piece of information is obtained. A learning data generation device according to claim 1 or 2.

4. The aforementioned at least one processor is The first text is input to a text analysis learning model that outputs at least one piece of information from region, type, and state when text is input, and the second piece of information is obtained. A learning data generation device according to any one of claims 1 to 3.

5. The aforementioned at least one processor is Delete the first pseudo-label which differs from at least one of the first information and the second information. A learning data generation device according to any one of claims 1 to 4.

6. A learning data generation device according to any one of claims 1 to 5, The aforementioned at least one processor is The first learning model is trained using the labeled image and the labels of the labeled image as training data. Using the first unlabeled image and the first pseudo-label (the first label) after filtering as training data, a second learning model is trained that outputs a label relating to the region of interest in the image when the image is input. Learning device.

7. The first learning model and the second learning model include a convolutional neural network. The learning device according to claim 6.

8. The first learning model and the second learning model include a convolutional neural network of the same configuration. The learning device according to claim 7.

9. If we set the initial value of the variable M to 1 and N to an integer greater than or equal to 2, The aforementioned at least one processor is Obtain the Mth unlabeled image and the Mth text attached to the aforementioned Mth unlabeled image. From the unlabeled image M, obtain at least one of the first pieces of information, including the region, type, and state of the subject in the unlabeled image M. From the text of M, obtain at least one of the following second pieces of information: region, type, and state of the subject of the unlabeled image of M. The unlabeled image M is input to the Mth learning model, and the output of the Mth learning model is obtained as the Mth pseudo-label. Based on the first and second information of the unlabeled image of M, the pseudo-label of M is filtered. Using the unlabeled image of M and the first label, which is a pseudo-label of M after filtering, as training data, a (M+1) learning model is trained that outputs a label relating to the region of interest in the image when the image is input. The process of adding 1 to the aforementioned M, Repeat the above N times. A learning device according to any one of claims 6 to 8.

10. A computer, Image acquisition step of acquiring a first unlabeled image and a first text attached to the first unlabeled image, A first information acquisition step of acquiring at least one piece of first information from the first unlabeled image, which includes the region, type, and state of the subject of the first unlabeled image, A second information acquisition step of obtaining at least one second piece of information from the first text, which is the region, type, and state of the subject of the first unlabeled image. A pseudo-label acquisition step is performed in which a first unlabeled image is input to a first learning model that outputs a label relating to a region of interest in the image when an image is input, and the output of the first learning model is obtained as a first pseudo-label of the first unlabeled image. A filtering step of filtering the first pseudo-label based on the first information and the second information, A method for generating training data to execute this.

11. A program for causing a computer to execute the learning data generation method described in claim 10.

Images