Label generation method and device, learned model generation method, machine learning device, image processing method and device, and program

Abstract

To provide a label generation method and device, a learned model generation method, a machine learning device, an image processing method and device, and a program for providing information with a certainty factor along the position and seriousness of a disease in a medical image.SOLUTION: A label generation method is such that one or more first processors execute the steps of: acquiring one or more candidate positions of a disease in a first division unit from a first medical image; acquiring diagnostic information in which the position of the disease is uncertain or the position of the disease is specified in a second division unit; converting the diagnostic information into a certainty factor label according to the seriousness of the disease; associating the certainty factor of the disease according to the certainty factor label with the candidate positions of the disease acquired from the first medical image; and acquiring the position of the disease and a correct answer label of the certainty factor generated through the association.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a label generation method and apparatus, a trained model generation method, a machine learning apparatus, an image processing method and apparatus, and a program, and particularly relates to an information processing technology that contributes to medical image diagnosis support.

Background Art

[0002] Patent Document 1 describes a method for supporting diagnosis of diseases using endoscopic images of digestive organs with a neural network. The method described in Patent Document 1 uses a first endoscopic image of a digestive organ and at least one definitive diagnosis result of a positive or negative disease of the digestive organ, a past disease, a severity level, or information corresponding to the imaged site corresponding to the first endoscopic image to train a neural network, and the trained neural network outputs at least one of the probability of a positive and / or negative disease of the digestive organ, the probability of a past disease, the severity level of the disease, or information corresponding to the imaged site based on a second endoscopic image of the digestive organ.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [Problem 1: Importance of estimating disease severity] When estimating the location of a disease from a medical image using machine learning, generally, training data (learning data) annotated by a medical professional with respect to the presence or absence and location of the disease in the medical image is used. A machine learning model trained using this training data estimates the location of the disease and its confidence level from the input image. In this case, the confidence level often depends not on the severity of the disease but on the appearance frequency of each disease pattern for each disease type. For example, in the case of a model for detecting pleural effusion from a chest X-ray image, even a small amount of pleural effusion typically results in a high confidence score.

[0005] Therefore, in a diagnostic support system that provides a doctor with information including the location and confidence level of a disease using such a model, the location of a disease with low severity (mild) will be provided as information with a high confidence level, providing information that does not conform to the intuition of a doctor who emphasizes the severity (grading) of the disease. For such problems, a system that provides diagnostic support information that conforms to the intuition of a doctor as much as possible is desired.

[0006] [Problem 2: Importance of Combining Information Indicating the Severity of a Disease and Information on the Location of the Disease] In order to realize a diagnostic support system that can provide confidence level information along with the severity of a disease to conform to the intuitive understanding of a doctor who emphasizes the severity of the disease, it is conceivable to generate a machine learning model that estimates the location of the disease and the confidence level along with the severity from a medical image using machine learning. To generate such a machine learning model, it is necessary to prepare a large number of pairs of data, namely, training medical images and label data indicating the correct answers of the location and confidence level of the disease in the medical image, that is, correct answer data.

[0007] When generating a correct label of confidence level according to the severity of a disease, it is conceivable to use diagnostic information as information indicating the severity of the disease. The diagnostic information includes information obtained by a definitive diagnosis test (hereinafter referred to as definitive diagnosis test information). Patent Document 1 describes training a neural network using the severity level which is the definitive diagnosis result for a first endoscopic image, but it is premised that the definitive diagnosis result in Patent Document 1 is data including information on anatomical imaging sites such as "pharynx" and "esophagus".

[0008] However, there is also data in the diagnostic information in which the disease location is not specified. For example, since sputum test information is a measured value obtained by measuring the total amount of bacteria discharged from the lungs, it is not possible to specify where (at which position) in the lungs the disease is present. When using such data with an indefinite disease location for learning, the machine learning model can estimate the severity of the disease, but it is not easy to identify the location of the disease. For the data in the diagnostic information with an indefinite disease location, the technique of Patent Document 1 cannot be applied.

[0009] Alternatively, even in the case of diagnostic information in which the disease location is recorded, there may be data in which the region division granularity of the location information is not the desired granularity. For example, while there is data in which the disease location is recorded in units of region division based on anatomical structures such as the name of the organ part as diagnostic information, the task to be realized by the machine learning model is a process of estimating the disease location and confidence level in pixel units from the input medical image, and there may be cases where the region division granularity of the location is different. Even in such a case, the technique of Patent Document 1 cannot be applied, and it is difficult to generate a machine learning model that realizes the target task.

[0010] The present disclosure has been made in view of such circumstances, and one of its objectives is to perform machine learning on the confidence level according to the position and severity of a disease in a medical image, and to provide information that matches the intuition of a doctor as much as possible. In relation to this objective, one of the objectives of the present disclosure is to provide a label generation method, an apparatus, and a program that can efficiently generate a correct label that can contribute to the generation of a machine learning model for estimating the confidence level according to the position and severity of a disease from a medical image.

[0011] Another objective of the present disclosure is to provide a trained model generation method, a machine learning apparatus, and a program for performing machine learning using the correct label generated by the label generation method of the present disclosure. Furthermore, one of the other objectives of the present disclosure is to provide an image processing apparatus and a program that can generate information indicating the position and confidence level of a disease in a medical image using a trained machine learning model, and provide the information in a form that is intuitively easy for a doctor to understand.

Means for Solving the Problems

[0012] The label generation method according to the first aspect of the present disclosure includes steps in which one or more first processors acquire one or more candidate positions of a disease from a first medical image in a first segmentation unit, acquire diagnostic information for the first medical image, where the position of the disease is indeterminate or the position of the disease is specified in a second segmentation unit, convert the diagnostic information into a confidence level label according to the severity of the disease, associate the confidence level of the disease corresponding to the confidence level label with the candidate position of the disease acquired from the first medical image, and acquire a correct label of the position and confidence level of the disease for the first medical image generated by the association.

[0013] According to the first aspect, by combining the candidate position of the disease obtained from the first medical image and the severity of the disease grasped from the diagnostic information, converting the severity into a confidence level label, and associating the confidence level corresponding to the severity of the disease with the candidate position of the disease, a correct label of the position and confidence level of the disease for the first medical image can be generated.

[0014] The first division unit and the second division unit that define the granularity (resolution) of the information indicating the position may be different division units from each other. The term "division unit" means a unit for dividing a region to distinguish positions. According to the first aspect, it is possible to efficiently generate a correct label using diagnostic information with an indeterminate disease position or diagnostic information in which the division unit of the disease position is different from the desired division unit.

[0015] In the label generation method according to the second aspect, in the label generation method according to the first aspect, in the step of obtaining the correct label, one or more first processors may be configured to obtain the correct label of the disease position and confidence level in the first division unit or the second division unit.

[0016] In the label generation method according to the third aspect, in the label generation method according to the first aspect or the second aspect, one or more first processors further execute a step of obtaining anatomical structure information from the first medical image, and in the step of associating the confidence level of the disease with the candidate position of the disease, the position of the disease may be restricted to be located within a desired anatomical structure specified from the anatomical structure information.

[0017] In the label generation method according to the fourth aspect, in the label generation method according to any one of the first to third aspects, the diagnostic information is a three-dimensional examination image, and the step of converting the diagnostic information into a confidence level label includes a step of recognizing the anatomical structure from the three-dimensional examination image, a step of recognizing the position of the disease from the three-dimensional examination image, and a step of calculating the confidence level label of the disease for each anatomical structure from the recognized anatomical structure and the position of the disease.

[0018] In the label generation method according to the fifth aspect, in the label generation method according to any one of the first to third aspects, the diagnostic information is sputum test information including the test result of the sputum test, and the step of converting the diagnostic information into the confidence level label of the disease may include a step of calculating the confidence level label of the disease based on the amount of bacteria collected in the sputum test.

[0019] In the label generation method according to the sixth aspect, in the label generation method according to any one of the first to fifth aspects, in the step of obtaining candidate positions of one or more diseases, a pre-trained first machine learning model may be used to calculate a saliency map of the disease.

[0020] In the label generation method according to the seventh aspect, in the step of associating a confidence level of a disease with a candidate position of the disease in the label generation method according to the sixth aspect, the confidence level label may be weighted according to the value of the saliency map.

[0021] In the label generation method according to the eighth aspect, in the label generation method according to any one of the first to seventh aspects, the diagnostic information is information in which the position of the disease is specified in the second division unit, and one or more first processors perform a step of obtaining anatomical structure information from the first medical image in the third division unit, a step of converting the candidate position of the disease in the first division unit to the candidate position of the disease in the third division unit, and a step of converting the confidence level label of the second division unit converted from the diagnostic information to the confidence level label of the third division unit. In the step of associating the confidence level of the disease with the candidate position of the disease, the confidence level of the disease corresponding to the candidate position of the disease in the third division unit is associated according to the confidence level label of the third division unit, and in the step of obtaining the correct label, the correct label of the position and confidence level of the disease in the third division unit may be obtained.

[0022] In the label generation method according to the ninth aspect, in the label generation method according to any one of the first to eighth aspects, the first medical image may be a chest X-ray image, a computed tomography image, or a magnetic resonance image.

[0023] In the label generation method according to the tenth aspect, in the label generation method according to any one of the first to ninth aspects, at least one of pleural effusion, pneumothorax, and pulmonary tuberculosis may be targeted as the disease.

[0024] The method for generating a learned model according to the 11th aspect of the present disclosure includes steps in which one or more second processors cause a second machine learning model to be learned by machine learning using training data including correct labels generated by the label generation method according to any one of the 1st to 10th aspects, and generate a learned second machine learning model trained to receive an input of a second medical image and output the position and confidence of a disease with respect to the second medical image.

[0025] The method for generating a learned model according to the 12th aspect is the method for generating a learned model according to the 11th aspect, wherein the confidence label of the disease is expressed as a continuous value, and in the step of causing the second machine learning model to be learned, the second machine learning model may be configured to perform regression prediction of the confidence of the disease from the first medical image.

[0026] The method for generating a learned model according to the 13th aspect is the method for generating a learned model according to the 11th aspect, wherein the confidence label of the disease is expressed as a discrete value, and in the step of causing the second machine learning model to be learned, the second machine learning model may be configured to perform classification prediction of the confidence of the disease from the first medical image.

[0027] The image processing method according to the 14th aspect of the present disclosure includes steps in which one or more third processors calculate the position and confidence of a disease with respect to a second medical image using the learned second machine learning model generated by the method for generating a learned model according to any one of the 11th to 13th aspects.

[0028] The image processing method according to the 15th aspect may be configured such that in the image processing method according to the 14th aspect, one or more third processors further perform a step of changing the display form of the disease according to the value of the confidence of the disease with respect to the second medical image.

[0029] The label generation device according to the 16th aspect of the present disclosure is a label generation device including one or more first processors, and the one or more first processors perform a process of obtaining candidate positions of one or more diseases from a first medical image in a first segmentation unit, a process of obtaining diagnostic information for the first medical image, where the position of the disease is uncertain or the position of the disease is specified in a second segmentation unit, a process of converting the diagnostic information into a confidence label according to the severity of the disease, a process of associating a confidence level of the disease corresponding to the confidence label with the candidate position of the disease obtained from the first medical image, and a process of obtaining a correct label of the position and confidence level of the disease for the first medical image generated by the associating process.

[0030] The machine learning device according to the 17th aspect of the present disclosure is a machine learning device including one or more second processors, and the one or more second processors perform a process of training a second machine learning model by machine learning using training data including correct labels generated by the label generation method according to any one of the 1st to 10th aspects of the present disclosure, and training the second machine learning model to receive an input of a second medical image and output the position and confidence level of the disease in the second medical image from the second machine learning model.

[0031] The image processing device according to the 18th aspect of the present disclosure is an image processing device including one or more third processors, and the one or more third processors perform a process of calculating the position and confidence level of the disease for the second medical image using the learned second machine learning model generated by the learned model generation method according to any one of the 11th to 13th aspects of the present disclosure.

[0032] The program according to the 19th aspect of the present disclosure is a program that causes a computer to execute the label generation method described in any one of the 1st to 10th aspects of the present disclosure.

[0033] The program according to the 20th aspect of the present disclosure is a program that causes a computer to execute the learned model generation method according to any one of the 11th to 13th aspects of the present disclosure.

[0034] The program according to the 21st aspect of the present disclosure is a program that causes a computer to execute the image processing method according to any one of the 14th aspect or the 15th aspect.

Effects of the Invention

[0035] According to the label generation method, label generation device, and program of the present disclosure, a correct label that can contribute to the generation of a machine learning model for estimating the confidence level according to the position and severity of a disease from a medical image can be efficiently generated. Further, according to the learned model generation method, machine learning device, and program of the present disclosure, a learned machine learning model for estimating the position of a disease and the confidence level according to the severity of the disease from a medical image can be generated by machine learning using the generated correct label. Furthermore, according to the image processing method, image processing device, and program of the present disclosure, by using the generated learned machine learning model, information indicating the position and confidence level of a disease in a medical image can be provided as information in a form that is intuitively easy for a doctor to understand.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

DETAILED DESCRIPTION OF THE INVENTION

[0037] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0038] 〔Overall configuration example of the system according to the embodiment〕 FIG. 1 is a block diagram schematically showing an overall configuration example of a system 1 according to an embodiment of the present disclosure. This system 1 includes an inspection information management device 4, a label generation device 10, a machine learning device 20, and an image processing device 30. The processing functions of these devices can be realized by a combination of computer hardware and software.

[0039] The inspection information management device 4 is an information processing device that stores and manages information including the inspection results of various inspections performed in a medical facility. The inspection information management device 4 includes a large-capacity storage device 6 and a database management program. Various data including a medical image IM taken using a modality device are stored in the storage device 6. The modality device can be various inspection devices such as, for example, an X-ray imaging device, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, an ultrasonic diagnostic device, a positron emission tomography (PET) device, a mammography device, an X-ray fluoroscopy diagnostic device, and an endoscope device.

[0040] The inspection information management device 4 may function as a picture archiving and communication system (PACS). The inspection information management device 4 may be configured to include, for example, a DICOM server operating according to the specifications of DICOM (Digital Imaging and Communications in Medicine).

[0041] In the memory device 6, medical images IM and definitive diagnosis examination information DD for a plurality of patients are stored in association with patient information PI. "Association" is synonymous with "correlation". The medical image IM may be, for example, a simple chest X-ray image. The definitive diagnosis examination may be, for example, a CT examination. When the target disease is pleural effusion, the definitive diagnosis examination information DD may be, for example, the amount of pleural effusion stored in each of the left and right lung fields measured from a CT image, or may be a CT image which is an examination image, or may be an MRI image. Alternatively, the definitive diagnosis examination may be a sputum examination, and in this case, the definitive diagnosis examination information DD may be the amount of bacteria indicating the examination result of the sputum examination. Sputum examination information is an example of data with an indefinite disease location.

[0042] The label generation device 10 is an information processing device that acquires the medical image IM stored in the memory device 6 and the corresponding definitive diagnosis examination information DD, and generates a correct label of confidence according to the disease location and severity for the medical image IM based on these pairs of data PD.

[0043] The label generation device 10 includes a pre-trained first machine learning model 12, a confidence label conversion unit 14, and an association unit 16. The first machine learning model 12 is a disease detection model trained (learned) by machine learning to estimate candidate disease positions from the input medical image IM.

[0044] The first machine learning model 12 may be, for example, a model that performs a segmentation task of recognizing a disease from a medical image IM and labeling it in pixel units. The first machine learning model 12 is, for example, configured using a neural network. The first machine learning model 12 may be configured using a convolutional neural network. Note that the first machine learning model 12 is substantially a program.

[0045] The label generation device 10 can obtain a saliency map SM indicating candidate positions of diseases for the medical image IM from the output of the first machine learning model 12 for the input of the medical image IM. This saliency map SM may be a binary image or a heatmap image visualizing the candidate positions of diseases. The granularity of the candidate positions of diseases shown in the saliency map SM may be, for example, in units of pixels of the medical image IM.

[0046] The "granularity" of the information indicating the position means the fineness of the unit for dividing the target area to specify the position, and means the region division granularity. The fact that the granularity is fine (small) means that the region as one unit of region division is small. In this specification, the region as the unit for dividing the region is referred to as the "division unit". The term region division granularity can be understood by replacing it with the term "division unit".

[0047] The definitive diagnosis examination information DD may be data with an indeterminate disease position or data with a recorded disease position. When the definitive diagnosis examination information DD includes information on the disease position, the granularity of the information indicating the disease position, that is, the region division granularity, may be coarser than the region division granularity of the saliency map SM. For example, the region division granularity of the disease position recorded in the definitive diagnosis examination information DD may be a division unit of an anatomical structure such as whether it is the right lung field or the left lung field. The term anatomical structure means an anatomical structure.

[0048] The confidence label conversion unit 14 performs a process of converting the severity of the disease grasped from the definitive diagnosis examination information DD into a confidence label. "Severity" may be paraphrased as the grading of the disease. The confidence label may be defined by discrete values or continuous values.

[0049] The association unit 16 is a processing unit that associates the position of a disease with a confidence level. The association unit 16 performs an association of a confidence level indicating severity for each candidate position of the disease based on the candidate position of the disease specified by the saliency map SM and the confidence label calculated by the confidence label conversion unit 14. Through the processing of this association unit 16, the correct label of the position and confidence level of the disease for the medical image IM is generated.

[0050] The label generation device 10 generates correct data GT, which is label data to which (associated with) a correct label is assigned for each position specified at a predetermined region division granularity in the medical image IM.

[0051] The label generation device 10 generates corresponding correct data GT for each of a plurality of medical images IM, and generates a data set DS including data of pairs of the plurality of sets of medical images IM and the correct data GT. A part or all of the data set DS thus generated is used as a training data set TDS for machine learning.

[0052] The machine learning device 20 is a computer system that performs machine learning using the training data set TDS and trains the second machine learning model 22. The second machine learning model 22 is trained to receive an input of the medical image IM included in the training data set TDS and output the position and confidence level of the disease in the medical image IM.

[0053] The machine learning device 20 updates the parameters of the second machine learning model 22 so that the output from the second machine learning model 22 for the input of the medical image IM approaches the correct data GT. The second machine learning model 22 is configured using, for example, a neural network. The second machine learning model 22 may be configured using a convolutional neural network. The machine learning device 20 may be configured to optimize the parameters of the second machine learning model 22 by, for example, a deep learning algorithm.

[0054] The machine learning device 20 executes machine learning using the training data set TDS, thereby generating a trained (learned) second machine learning model 22 with desired inference performance. The third machine learning model 32, which is the learned model thus generated, is implemented in the image processing device 30. Note that the second machine learning model 22 and the third machine learning model 32 are substantially programs.

[0055] The image processing device 30 is an information processing device (computer system) that includes the third machine learning model 32, accepts the input of an unknown medical image IMu, infers the disease location and confidence level for the unknown medical image IMu using the third machine learning model 32, and outputs the inference result. The image processing device 30 may be incorporated, for example, as part of a diagnostic support system.

[0056] The inspection information management device 4, the label generation device 10, the machine learning device 20, and the image processing device 30 may be communicably connected to each other via the telecommunication line 40, or some or all of these devices may be configured as stand-alone devices. The transfer of data between the devices is not limited to via the network, and for example, a portable information recording medium may be used. The telecommunication line 40 may be a wide area communication line, a local area communication line, or a combination thereof.

[0057] Also, in FIG. 1, the inspection information management device 4, the label generation device 10, the machine learning device 20, and the image processing device 30 are shown as separate devices, but it is also possible to integrate the processing functions of a plurality of these devices into one device.

[0058] For example, the storage device 6 of the inspection information management device 4 may be included in the label generation device 10. Further, for example, the label generation device 10 and the machine learning device 20 may be integrated and configured as one device. The respective processing functions of the inspection information management device 4, the label generation device 10, the machine learning device 20, and the image processing device 30 can be realized by a computer system including one or a plurality of computers. Furthermore, some or all of the processing functions of these devices may be realized by cloud computing.

[0059] 〔Example of Hardware Configuration of Label Generation Device〕 FIG. 2 is a block diagram showing an example of the hardware configuration of the label generation device 10 according to the embodiment. The label generation device 10 includes a processor 102, a computer-readable medium 104 which is a non-transitory tangible object, a communication interface 106, an input / output interface 108, and a bus 110. The processor 102 is connected to the computer-readable medium 104, the communication interface 106, and the input / output interface 108 via the bus 110.

[0060] The form of the label generation device 10 is not particularly limited, and it may be a server, or a workstation, a personal computer, or the like.

[0061] The processor 102 includes a CPU (Central Processing Unit). The processor 102 may include a GPU (Graphics Processing Unit). The processor 102 is an example of the "first processor" in the present disclosure. The computer-readable medium 104 includes a memory 112 which is a main storage device and a storage 114 which is an auxiliary storage device. The computer-readable medium 104 may be, for example, a semiconductor memory, a hard disk drive (HDD) device, a solid state drive (SSD) device, or a combination of a plurality of these. The computer-readable medium 104 is an example of a storage device that stores instructions executed by the processor 102.

[0062] The label generation device 10 further includes an input device 122 and a display device 124. The input device 122 is configured by, for example, a keyboard, a mouse, a multi-touch panel, or other pointing devices, or a voice input device, or an appropriate combination thereof.

[0063] The display device 124 is configured by, for example, a liquid crystal display, an organic EL (organic electro-luminescence: OEL) display, a projector, or an appropriate combination thereof. The input device 122 and the display device 124 are connected to the processor 102 via the input / output interface 108. The label generation device 10 may be connected to the telecommunication line 40 via the communication interface 106.

[0064] 〔Example 1 of label generation method〕 FIG. 3 is an explanatory diagram showing Example 1 of the label generation method executed by the label generation device 10. Here, a workflow is exemplified when the medical image IM1 as the input image is a chest plain X-ray image, the definitive diagnosis examination is a CT examination, and the target disease is pleural effusion.

[0065] The processor 102 inputs the medical image IM1 to the first machine learning model 12 and obtains a saliency map SM1 indicating the candidate position of the disease from the output of the first machine learning model 12. The medical image IM1 is an example of the "first medical image" and the "chest X-ray image" in the present disclosure. The saliency map SM1 may be a binary image or a heatmap image reflecting the probability of being a disease (a score indicating the likelihood of being a disease).

[0066] The region FP1a and the region FP1b shown on the saliency map SM1 in FIG. 3 respectively indicate the positions of the findings (candidate positions of the disease) estimated by the first machine learning model 12. The region division granularity of the saliency map SM1, that is, the region division granularity of the candidate positions of the disease, may be in units of pixels of the medical image IM1. The unit of pixels, which is the region division granularity of the saliency map SM1, is an example of the "first division unit" in the present disclosure.

[0067] Also, the processor 102 acquires the pleural effusion storage amounts for each of the left and right lung fields as the definitive diagnosis examination information DD1. That is, the definitive diagnosis examination information DD1 includes the pleural effusion storage amount in the left lung field and the pleural effusion storage amount in the right lung field. Note that the pleural effusion storage amounts for each of the left and right lung fields can be measured, for example, from the CT images of the definitive diagnosis examination. The definitive diagnosis examination information DD1 may be data in a text format such as a sentence or data in a table format. The definitive diagnosis examination information DD1 is an example of the "diagnostic information" in the present disclosure.

[0068] This definitive diagnosis examination information DD1 includes information indicating positions such as "left lung field" and "right lung field". The region division granularity of the position information included in the definitive diagnosis examination information DD1 has the anatomical structure of "left lung field" or "right lung field" as the division unit, which is coarser than the region division granularity of the saliency map SM1. The division unit of the position information included in the definitive diagnosis examination information DD1 is an example of the "second division unit" in the present disclosure.

[0069] The processor 102 calculates a confidence label (a value indicating confidence) according to the pleural effusion storage amount from the acquired definitive diagnosis examination information DD1. The pleural effusion storage amount is related to the severity, and the larger the value of the pleural effusion storage amount, the higher the degree (grade) of severity. That is, the larger the value of the pleural effusion storage amount, the larger the value is calculated as the confidence value. As a method for calculating the confidence from the pleural effusion storage amount, for example, a look-up table may be used or a calculation formula may be used.

[0070] Here, for example, if the amount of pleural effusion PEV_L in the left lung field is greater than the amount of pleural effusion PEV_R in the right lung field, the confidence label CL_L for the left lung field calculated from the amount of pleural effusion PEV_L in the left lung field will be a value greater than the confidence label CL_R for the right lung field calculated from the amount of pleural effusion PEV_R in the right lung field.

[0071] Subsequently, the processor 102 combines the information on the candidate positions of the disease identified from the saliency map SM1 and the confidence label calculated from the definitive diagnosis examination information DD1 to associate the candidate positions of the disease with the confidence levels. In the example shown in FIG. 3, the confidence label CL_L calculated from the amount of pleural effusion PEV_L in the left lung field is associated with the position within the region FP1a belonging to the left lung field. Also, the confidence label CL_R calculated from the amount of pleural effusion PEV_R in the right lung field is associated with the position within the region FP1b belonging to the right lung field.

[0072] The processor 102 may directly assign the confidence label converted from the amount of pleural effusion in the definitive diagnosis examination information DD1 as the confidence level of the disease for each candidate position of the disease, or, for example, weight the confidence label according to the value of the candidate position in the saliency map SM1 (the score value indicating the likelihood of the candidate position of the disease) and assign the confidence level to each position.

[0073] In this way, for each candidate position of the disease identified from the saliency map SM1, the confidence level calculated based on the definitive diagnosis examination information DD1 is assigned, and the correct label of the position and confidence level of the disease in the medical image IM1 is generated.

[0074] That is, in Example 1 of the label generation method according to the present embodiment, the saliency map SM1 of the disease obtained from the medical image IM1 is used as prior knowledge to identify the disease position in the definitive diagnosis examination information DD1, and the confidence level calculated from the definitive diagnosis examination information DD1 is used as the correct label.

[0075] The ground truth data GT1, which is the data of the ground truth labels generated for the medical image IM1, may be, for example, a grayscale image based on the grayscale representation of the ground truth labels assigned to each pixel of the medical image IM1. The ground truth data GT1 becomes data indicating the correct output for the input of the medical image IM1 in supervised learning.

[0076] FIG. 4 is a block diagram schematically showing the functional configuration of a label generation device 10 that executes the label generation method shown in FIG. 3. The label generation device 10 includes a data acquisition unit 130, a disease detection unit 140, a confidence label conversion unit 14, an association unit 16, and a data storage unit 150.

[0077] The data acquisition unit 130 includes a medical image acquisition unit 132 and a definitive diagnosis examination information acquisition unit 134. The medical image acquisition unit 132 acquires the medical image IM1 to be processed. The definitive diagnosis examination information acquisition unit 134 acquires the data of the definitive diagnosis examination information DD1 associated with the medical image IM1. In FIG. 4, an example is shown in which the amount of pleural effusion in each of the left and right lung fields is acquired as the definitive diagnosis examination information DD1.

[0078] The disease detection unit 140 includes a first machine learning model 12 and detects the position of the disease from the input medical image IM1. A saliency map SM1 indicating the candidate position of the disease for the medical image IM1 is obtained by the processing of the disease detection unit 140.

[0079] The confidence label conversion unit 14 converts the definitive diagnosis examination information DD1 into a confidence label along the severity of the disease.

[0080] The association unit 16 associates the candidate position of the disease specified from the saliency map SM1 with the confidence level calculated from the definitive diagnosis examination information DD1, and generates the correct labels of the disease position and the confidence level.

[0081] The association unit 16 may use, as the correct label of the confidence level for each position, the confidence level label directly converted from the definitive diagnosis examination information DD1 for the candidate position of the disease, or may determine the confidence level for each position by weighting the confidence level label according to the value (score value indicating likelihood) of the candidate position in the saliency map SM1.

[0082] The correct data GT1 generated by the association process of the association unit 16 is associated with the medical image IM1 and stored in the data storage unit 150.

[0083] FIG. 5 is a flowchart showing Example 1 of the label generation method according to the embodiment. FIG. 5 is a flowchart of the explanatory diagram of FIG. 3.

[0084] In step S10, the processor 102 acquires the medical image IM1.

[0085] In step S12, the processor 102 detects candidate positions of the disease using the first machine learning model 12 for the acquired medical image IM1. By this step S12, the processor 102 acquires the candidate positions of the disease as the detection result.

[0086] In step S14, the processor 102 acquires the definitive diagnosis examination information DD1 corresponding to the medical image IM1.

[0087] In step S16, the processor 102 converts the acquired definitive diagnosis examination information DD1 into a confidence level label.

[0088] Note that the order of the processes from step S10 to step S16 is not limited to the example shown in FIG. 5. For example, step S14 may be executed before step S10, or may be executed in parallel with step S10.

[0089] In step S18, the processor 102 associates the candidate positions of the disease with the confidence levels. By this association, the processor 102 generates the correct data GT1 of the disease positions and confidence levels for the medical image IM1 (step S20).

[0090] In step S22, the processor 102 associates the medical image IM1 with the generated correct data GT1 and stores them in the data storage unit 150.

[0091] After step S22, the processor 102 ends the flowchart of FIG. 5.

[0092] For the data of the pairs of the medical images IMi and the definitive diagnosis examination information DDi regarding a plurality of patients, when the processor 102 executes the processing of the flowchart of FIG. 5, correct data GTi is generated for each of the plurality of medical images IMi, and a data set DS including a plurality of pairs of the medical image IMi and the correct data GTi is obtained. Note that the subscript i is an index number for identifying the pair of data.

[0093] 〔Example 2 of label generation method〕 In FIG. 3, an example of generating the correct labels of the disease positions and confidence levels at the region division granularity of the saliency map SM1 (hereinafter referred to as the first granularity) was described. However, the present invention is not limited to this example, and the correct labels of the disease positions and confidence levels may be generated at the region division granularity of the disease positions in the definitive diagnosis examination information DD1 (hereinafter referred to as the second granularity).

[0094] FIG. 6 is an explanatory diagram showing Example 2 of the label generation method executed by the label generation device 10. Differences between FIG. 6 and FIG. 3 will be described.

[0095] In FIG. 6, the processor 102 performs a process of extracting a region of an anatomical structure from the input medical image IM1 and obtains anatomical structure information AS1. When the target disease is pleural effusion, the processor 102 may extract the respective regions (parts) of the left lung field and the right lung field in the medical image IM1 and obtain the anatomical structure information AS1 in which the regions of the left and right lung fields are specified. The region division granularity of the anatomical structure information AS1 shown in the example of FIG. 5 may be the same granularity (second granularity) as the definitive diagnosis examination information DD1.

[0096] The processor 102 further combines the saliency map SM1 and the anatomical structure information AS1 to generate disease position data DP1 indicating the position of the disease at the second region division granularity. The disease position data DP1 shown in FIG. 6 may be table data indicating that there is pleural effusion in each of the right lung and the left lung.

[0097] The processor 102 associates the confidence labels converted from the definitive diagnosis examination information DD1 for each position (here, part) specified by the disease position data DP1, and generates ground truth data GT1_2 of the position and confidence of the disease.

[0098] In the case of FIG. 6, a high-confidence label is assigned to the position of the right lung, and a low-confidence label is assigned to the position of the left lung. This is an example of generating confidence labels for each position at the second granularity. The ground truth data GT1_2 may be data in tabular form.

[0099] 〔Example 3 of label generation method〕 In FIG. 6, an example of generating the ground truth labels of the position and confidence of the disease at the region division granularity (second granularity) of the definitive diagnosis examination information DD1 has been described. However, it is not limited to this example, and the ground truth labels of the position and confidence of the disease may be generated at a third region division granularity (hereinafter referred to as the third granularity) that is different from both the region division granularity (first granularity) of the saliency map SM1 and the region division granularity (second granularity) of the position of the disease in the definitive diagnosis examination information DD1. An example thereof is shown in FIG. 7.

[0100] FIG. 7 is an explanatory diagram showing Example 3 of the label generation method executed by the label generation device 10. Differences between FIG. 7 and FIG. 6 will be described.

[0101] In FIG. 7, instead of the combination of the medical image IM1 and the definitive diagnosis examination information DD1 in FIG. 6, a combination of the medical image IM2 and the definitive diagnosis examination information DD2_1 is used.

[0102] The medical image IM2 may be a simple X-ray image similar to the medical image IM1. The definitive diagnosis examination information DD2_1 may be a CT image (computed tomography image), which is a three-dimensional examination image obtained by a CT examination as a definitive diagnosis examination.

[0103] The processor 102 generates a saliency map SM2 from the input medical image IM2 by the first machine learning model 12. Further, the processor 102 extracts an anatomical structure from the medical image IM2 and acquires anatomical structure information AS2.

[0104] The processor 102 may use a machine learning model that has been learned (trained) by machine learning to perform a segmentation task of recognizing an anatomical structure in pixel units from the input medical image IM2 and labeling according to the classification of the region of the anatomical structure to acquire the anatomical structure information AS2.

[0105] In the example shown in FIG. 7, region extraction in pixel units is performed from a simple chest X-ray image, and anatomical structures such as the clavicle, trachea, right lung field, left lung field, superior vena cava, right atrium, aortic arch, descending aorta, and left atrium are recognized, and a segmentation image in which the types of anatomical structures are labeled in pixel units is obtained. There can be various aspects regarding the types of anatomical structures to be extracted. For example, regarding the lungs, it may be classified into units of regions such as the upper right lobe, middle right lobe, lower right lobe, upper left lobe, and lower left lobe, which are further subdivisions of the classifications of the right lung field and the left lung field.

[0106] Here, in order to show an example of a region division granularity different from the region division granularity (second granularity) of the disease position specified by the definitive diagnosis examination, it is explained that the anatomical structure information AS2 includes position information for the lungs in units of five classifications that are finer than the specification of the positions of the two classifications of the left lung field and the right lung field. Note that the anatomical structure information AS2 may specify the positions of the region division granularity based on the anatomical structures of the left lung field and the right lung field.

[0107] The processor 102 generates disease position data DP2 indicating candidate positions of a disease with a region division granularity different from that of the saliency map SM2 by combining the saliency map SM2 and the anatomical structure information AS2. This disease position data DP2 may be, for example, tabular data as shown in FIG. 7. The division unit (region division granularity) of the position information included in the disease position data DP2 is an example of the "third division unit" in the present disclosure.

[0108] Also, the processor 102 analyzes the CT image using the analysis model 13, and extracts the anatomical structure and detects the disease from the CT image. The analysis model 13 may be a trained machine learning model that performs labeling of the anatomical structure in voxel units and detection of the disease from the input CT image by machine learning. The analysis model 13 may be a combination of a model for extracting the anatomical structure and a model for detecting the disease. As a machine learning model that performs three-dimensional segmentation, for example, a neural network model using the architecture of three-dimensional U-Net can be applied.

[0109] The analysis model 13 of this example extracts, for example, the left lung field and the right lung field as anatomical structures from the CT image. Also, the analysis model 13 detects the region of pleural effusion from the CT image.

[0110] The processor 102 calculates the amount of pleural effusion for each of the left and right lung fields based on the analysis result by the analysis model 13. The values of the amount of pleural effusion for each of the left and right lung fields thus calculated correspond to the definitive diagnosis examination information DD1 described with reference to FIG. 6.

[0111] The amount of pleural effusion in each of the left and right lung fields calculated based on the definitive diagnosis examination information DD2_1 shown in FIG. 7 is understood as the definitive diagnosis examination information DD2_2 potentially inherent in the definitive diagnosis examination information DD2_1. In other words, the definitive diagnosis examination information DD2_1, which is a three-dimensional examination image, is understood to contain information for specifying the position of the pleural effusion in voxel units and information for specifying in units of anatomical structures.

[0112] The processor 102 converts the amount of pleural effusion in each of the left and right lung fields calculated from the CT image into a confidence label respectively. Thereby, data of confidence labels reflecting the severity of pleural effusion in the left and right lung fields is obtained. The granularity of the position information for specifying the position of the pleural effusion in this confidence label data is different from the granularity of the position information in the disease position data DP2.

[0113] In the example of FIG. 7, the magnitude relationship among the first granularity which is the region division granularity of the saliency map SM2, the second granularity which is the region division granularity of the position of the disease (here, the position of the pleural effusion) in the confidence label data for each of the left and right lung fields, and the third granularity which is the region division granularity of the disease position data DP2 is the first granularity < the third granularity < the second granularity.

[0114] The processor 102 converts the confidence label data into data of the third granularity in order to align the granularity of the position information of both the disease position data DP2 and the confidence label data.

[0115] Then, the processor 102 combines the confidence label data converted to the third granularity and the disease position data DP2 to associate each candidate position of the disease position data DP2 with the confidence level, and generates the correct data GT2 of the position and confidence level of the disease for the medical image IM2. In this way, the correct data GT in which the position and confidence level of the disease are specified at the third granularity is obtained.

[0116] Note that the "high" confidence level in the correct data GT2 shown in FIG. 7 indicates that a numerically high confidence level is assigned, and the "low" confidence level indicates that a numerically low confidence level is assigned. For example, when expressing the confidence level according to the severity of the disease as a numerical value in the range of 0 to 1, "high" may be "1", "low" may be "0.2", etc. Also, the "-" for the confidence level of the non-disease position in the correct data GT2 shown in FIG. 7 may be "0".

[0117] FIG. 8 is a functional block diagram of a label generation device 10 that executes the label generation method shown in FIG. 7. Regarding the configuration shown in FIG. 8, the differences from FIG. 4 will be described. The label generation device 10 shown in FIG. 8 further includes an anatomical structure extraction unit 141, a disease position conversion unit 142, a 3D image analysis unit 143, and a label data conversion unit 145 in addition to the configuration of FIG. 4. Note that the notation "3D" means "three-dimensional".

[0118] The anatomical structure extraction unit 141 extracts the anatomical structure from the medical image IM2 acquired via the medical image acquisition unit 132 and obtains the anatomical structure information AS2.

[0119] The disease position conversion unit 142 converts the information on the candidate positions of the disease shown in the saliency map SM2 into the information on the candidate positions with different region division granularities. The disease position conversion unit 142 generates disease position data DP2 by, for example, converting the information on the candidate positions in pixel units of the medical image IM2 into the information on the candidate positions with the region division granularity of the anatomical structure shown in the anatomical structure information AS2.

[0120] The 3D image analysis unit 143 includes an analysis model 13 that analyzes a CT image, which is a three-dimensional examination image acquired via the definitive diagnosis examination information acquisition unit 134. The analysis model 13 functions as an anatomical structure extraction unit 147 that extracts the anatomical structure from the CT image and a disease detection unit 148 that detects the disease from the CT image. The disease detection unit 148 detects, for example, the region of pleural effusion from the CT image.

[0121] In addition, the 3D image analysis unit 143 includes a pleural effusion volume calculation unit 149. The pleural effusion volume calculation unit 149 counts the voxels in the pleural effusion region in the CT image based on the detection result of the disease detection unit 148, and calculates the pleural effusion volume for each left and right lung field. The pleural effusion volume calculation unit 149 may calculate the pleural effusion volume based on the information of the pleural effusion region specified on the CT image via the user interface.

[0122] The confidence label conversion unit 14 converts the pleural effusion volume for each left and right lung field obtained by the analysis of the 3D image analysis unit 143 into a confidence label.

[0123] The label data conversion unit 145 converts the confidence label for each left and right lung field into label data with the same region division granularity as the disease position data DP2. Here, the label data of the second granularity is converted into the label data of the third granularity.

[0124] The association unit 16 combines the disease position data DP2 and the confidence label obtained from the definitive diagnosis examination information DD2 by the label data conversion unit 145, associates the confidence label with each candidate position of the disease in the disease position data DP2, and generates the correct label of the disease position and confidence. The correct data GT2 generated by the association unit 16 is associated with the medical image IM2 and stored in the data storage unit 150.

[0125] FIG. 9 is a flowchart showing Example 3 of the label generation method according to the embodiment. FIG. 9 is a flowchart of the explanatory diagram of FIG. 8.

[0126] In step S30, the processor 102 acquires the medical image IM2.

[0127] In step S32, the processor 102 detects the candidate position of the disease using the first machine learning model 12 for the acquired medical image IM2. By this step S12, the processor 102 acquires the candidate position of the disease as the detection result. That is, the processor 102 acquires the saliency map SM2 indicating the candidate position of the disease for the medical image IM2.

[0128] In step S34, the processor 102 extracts an anatomical structure from the medical image IM2 and obtains anatomical structure information AS2.

[0129] In step S36, the processor 102 converts the information on the candidate positions of the disease shown in the saliency map SM2 into disease position data DP2 in anatomical structure units.

[0130] Further, the processor 102 may use the anatomical structure information AS2 to restrict the candidate positions of the disease in the saliency map SM2 so that they are located within a desired anatomical structure. For example, the processor 102 may exclude from the candidate positions of the pleural effusion estimated by the first machine learning model 12 those located outside the lung region including the left and right lung fields, and use only the candidate positions located within the lung region as appropriate candidate positions. Depending on the inference performance of the first machine learning model 12, it is also assumed that incorrect candidate positions may be output from the first machine learning model 12. Therefore, it is desirable to use the anatomical structure information AS2 in combination to restrict the disease position so that it is located within a desired anatomical structure.

[0131] In step S40, the processor 102 obtains a three-dimensional examination image of a definitive diagnosis examination for the medical image IM2. The three-dimensional examination image is, for example, a CT image.

[0132] In step S42, the processor 102 extracts an anatomical structure from the three-dimensional examination image and obtains anatomical structure information.

[0133] In step S43, the processor 102 detects a disease from the three-dimensional examination image. The disease to be detected is, for example, pleural effusion, and the processor 102 extracts the pleural effusion region from the three-dimensional examination image.

[0134] In step S44, the processor 102 calculates the pleural effusion volume based on the detection result of step S43 and obtains the pleural effusion volume for each of the left and right lung fields.

[0135] In step S45, the processor 102 converts the amount of pleural effusion stored in each of the left and right lung fields into a confidence level.

[0136] In step S46, the confidence level labels for each of the left and right lung fields obtained in step S45 are converted into label data with the same region division granularity as the disease position data DP2.

[0137] Note that the order of the processes from step S30 to step S46 is not limited to the example in FIG. 9, and the order can be changed as long as the processes do not conflict. For example, step S40 may be executed before step S30, or may be executed in parallel with step S30.

[0138] In step S48, the processor 102 associates the candidate positions of the disease with the confidence levels. By this association, the processor 102 generates the correct answer data GT2 for the position and confidence level of the disease in the medical image IM2 (step S49).

[0139] In step S50, the processor 102 associates the medical image IM2 with the generated correct answer data GT2 and stores them in the data storage unit 150.

[0140] After step S50, the processor 102 ends the flowchart of FIG. 9.

[0141] The flowchart shown in FIG. 9 is repeatedly executed for the data of pairs of medical images of a plurality of patients and three-dimensional examination images of definitive diagnosis examinations. By the processor 102 executing the processes of the flowchart of FIG. 9 for the data of a plurality of pairs, correct answer data GTi is generated for each of the plurality of medical images IMi, and a data set DS including a plurality of pairs of the medical image IMi and the correct answer data GTi is obtained.

[0142] FIG. 10 is a block diagram showing an example of a program and data stored in the memory 112 of the label generation device 10 that executes the label generation method of the flowchart shown in FIG. 9.

[0143] The memory 112 stores a plurality of programs and data including a medical image acquisition program 162, a definitive diagnosis examination information acquisition program 164, a disease detection program 170, an anatomical structure extraction program 171, a disease position restriction program 172, a disease position conversion program 173, a 3D image analysis program 183, a confidence label conversion program 184, a label data conversion program 185, an association program 186, a correct answer data storage processing program 187, and a display control program 188. The term "program" includes the concept of program modules. The processor 102 (see FIG. 2) functions as various processing units by executing the instructions of the programs stored in the memory 112.

[0144] The medical image acquisition program 162 includes instructions for executing a process of acquiring a medical image, and realizes the function as the medical image acquisition unit 132.

[0145] The definitive diagnosis examination information acquisition program 164 includes instructions for executing a process of acquiring definitive diagnosis examination information, and realizes the function as the definitive diagnosis examination information acquisition unit 134. The disease detection program 170 includes the first machine learning model 12. The disease detection program 170 includes instructions for executing a process of detecting a disease from the input medical image, and realizes the function as the disease detection unit 140.

[0146] The anatomical structure extraction program 171 includes instructions for executing a process of recognizing an anatomical structure from a medical image and generating anatomical structure information, and realizes the function as the anatomical structure extraction unit 141.

[0147] The disease position restriction program 172 includes instructions for executing a process of restricting the position of a disease within the region of an anatomical structure by using the candidate position of the disease detected by the disease detection program 170 and the anatomical structure information generated by the anatomical structure extraction program 171.

[0148] The disease position conversion program 173 includes instructions for executing a process of converting information on candidate positions of a disease detected by the disease detection program 170 into position information with a desired region division granularity. For example, the disease position conversion program 173 realizes a processing function of converting information on candidate positions of a disease specified in pixel units of a medical image into disease position data DP2 representing candidate positions of the disease in units of regions of anatomical structures.

[0149] The 3D image analysis program 183 is a program for executing a process of analyzing a three-dimensional inspection image, and includes an anatomical structure extraction program 190, a disease detection program 191, and a pleural effusion volume calculation program 192. The anatomical structure extraction program 190 includes instructions for executing a process of extracting an anatomical structure from a three-dimensional inspection image and performing labeling according to the classification of the anatomical structure. The anatomical structure extraction program 190 extracts, for example, regions of the left lung field and the right lung field from a CT image.

[0150] The disease detection program 191 includes instructions for executing a process of detecting a disease from a three-dimensional inspection image. The disease detection program 191 includes instructions for executing a process of detecting a pleural effusion region from a CT image, for example. Note that the anatomical structure extraction program 190 and the disease detection program 191 may be configured as an analysis model 13.

[0151] The pleural effusion volume calculation program 192 includes instructions for executing a process of calculating the pleural effusion volume from the pleural effusion region in a three-dimensional inspection image.

[0152] The confidence label conversion program 184 includes instructions for executing a process of converting the pleural effusion volume into a confidence label. The confidence label conversion program 184 is configured to perform conversion processing using, for example, a lookup table 194 that describes the correspondence between the pleural effusion volume and the confidence level.

[0153] The label data conversion program 185 includes instructions for causing a process to be executed that converts data of confidence level labels for each of the left and right lung fields (at the second granularity) obtained from the definitive diagnosis examination information into label data at the same granularity (the third granularity) as the disease location data generated by the disease location conversion program 173.

[0154] The association program 186 includes instructions for causing a process to be executed that combines the disease location data DP2 obtained from the medical image and the confidence level label data obtained from the definitive diagnosis examination information DD2 to perform an association between the disease location and the confidence level, and generate the correct label for the disease location and the confidence level. The association program 186 realizes the function of the association unit 16.

[0155] The correct data storage processing program 187 includes instructions for causing a process to be executed that stores the correct data generated by the association program 186 in association with the medical image in the data storage unit 150. The storage area as the data storage unit 150 may be provided in the storage 114.

[0156] The display control program 188 includes instructions for generating a display signal necessary for display output to the display device 124 and causing the display control of the display device 124 to be executed.

[0157] 〔Example of using sputum examination information as definitive diagnosis examination information〕 When using sputum examination information as the definitive diagnosis examination information, when the processor 102 converts the sputum examination information into a confidence level label, the processor 102 calculates the confidence level label of the disease based on the amount of bacteria collected in the sputum examination. Other processes may be the same as the processes in the label generation device 10 described above.

[0158] 〔Example of a machine learning device〕 FIG. 11 is a block diagram showing an example of the hardware configuration of the machine learning device 20 according to the embodiment. The machine learning device 20 includes a processor 202, a computer-readable medium 204 which is a non-transitory tangible substance, a communication interface 206, an input / output interface 208, and a bus 210. The computer-readable medium 204 includes a memory 212 and a storage 214. The processor 202 is connected to the computer-readable medium 204, the communication interface 206, and the input / output interface 208 via the bus 210. Further, the machine learning device 20 may further include an input device 222 and a display device 224. The hardware configuration of the machine learning device 20 may be the same as the corresponding elements of the label generation device 10 shown in FIG. 2. The processor 202 is an example of the "second processor" in the present disclosure.

[0159] The form of the machine learning device 20 is not particularly limited, and it may be a server, or a workstation, a personal computer, or the like.

[0160] The machine learning device 20 is communicably connected to an external device such as a training data storage unit 250 via the communication interface 206. The training data storage unit 250 includes a storage in which a training data set including a plurality of training data is stored. Note that the training data storage unit 250 may be constructed in the storage 214 within the machine learning device 20.

[0161] Various programs including a machine learning program 230 and a display control program 240 and data are stored in the computer-readable medium 204.

[0162] The machine learning program 230 includes instructions for acquiring training data and executing the learning process of the second machine learning model 22. That is, the machine learning program 230 includes a data acquisition program 232, the second machine learning model 22, a loss calculation program 236, and an optimizer 238.

[0163] The data acquisition program 232 includes instructions for causing the training data storage unit 250 to execute a process of acquiring training data in which the medical image IMj and the correct answer data GTj are associated with each other.

[0164] The second machine learning model 22 receives an input of the medical image IMj, estimates the position and confidence level of the disease from the input medical image IMj, and outputs an estimation result. The medical image IMj is an example of the "second medical image" in the present disclosure.

[0165] The loss calculation program 236 includes instructions for causing a process of calculating a loss indicating an error between the output data of the second machine learning model 22 and the correct answer data GTj. The optimizer 238 includes instructions for causing a process of calculating an update amount of the parameters of the second machine learning model 22 from the calculated loss and updating the parameters of the second machine learning model 22 based on the calculated update amount.

[0166] The display control program 240 includes instructions for generating a display signal necessary for display output to the display device 224 and causing the display control of the display device 224 to be executed.

[0167] FIG. 12 is a block diagram schematically showing the functional configuration of the machine learning device 20. The machine learning device 20 includes a second machine learning model 22 and a learning processing unit 24. The learning processing unit 24 includes a loss calculation unit 26 and a parameter update unit 28.

[0168] The loss calculation unit 26 calculates a loss indicating an error between the output data PRj indicating the position and confidence level of the disease output from the second machine learning model 22 and the correct answer data GTj associated with the medical image IMj.

[0169] The parameter update unit 28 calculates the update amount of the parameters of the second machine learning model 22 based on the loss calculated by the loss calculation unit 26 so that the loss is reduced, that is, so that the output data PRj approaches the correct answer data GTj, and updates the parameters of the second machine learning model 22 according to the calculated update amount. The parameters of the second machine learning model 22 include, for example, the filter coefficients (weights of connections between nodes) of the filters used for the processing of each layer of the neural network and the biases of the nodes. The parameter update unit 28 optimizes the parameters of the model by a method such as Stochastic Gradient Descent (SGD).

[0170] By performing the learning process using a plurality of training data and repeating the update of the parameters of the second machine learning model 22, the parameters of the second machine learning model 22 are optimized, and the second machine learning model 22 is trained to output an estimation result similar to the correct answer data GTj for the input of the medical image IMj.

[0171] When the confidence level of the disease is represented by a continuous value, the second machine learning model 22 may be configured as a regression model that regression-predicts the confidence level of the disease from the input medical image IMj.

[0172] Also, when the confidence level of the disease is represented by a discrete value, the second machine learning model 22 may be configured as a classification model that classification-predicts the confidence level of the disease from the input medical image IMj.

[0173] 〔Example of learned model generation method〕 FIG. 13 is a flowchart showing an example of the machine learning method executed by the machine learning apparatus 20.

[0174] In step S60, the processor 202 acquires training data, which is a data set in which the medical image IMj and the correct answer data GTj are associated, from the training data set.

[0175] In step S62, the processor 202 inputs the acquired medical image IMj into the second machine learning model 22, and obtains output data PRj indicating the estimated results of the disease position and confidence level in the medical image IMj from the second machine learning model 22. For example, the second machine learning model 22 performs regression prediction on the confidence level of the disease from the medical image IMj and outputs the prediction result (estimated result). Alternatively, the second machine learning model 22 performs classification prediction on the confidence level of the disease from the medical image IMj and outputs the prediction result.

[0176] In step S64, the processor 202 calculates a loss indicating the error between the output data PRj of the second machine learning model 22 and the correct data GTj.

[0177] In step S65, the processor 202 calculates the parameter update amount of the second machine learning model 22 so that the loss calculated in step S64 becomes smaller.

[0178] In step S66, the processor 202 updates the parameters of the second machine learning model 22 according to the parameter update amount calculated in step S65. The operations from step S60 to step S66 described above may be performed in units of mini-batches.

[0179] After step S66, in step S68, the processor 202 determines whether to end the learning. The end condition of the learning may be determined based on the value of the loss or based on the number of parameter updates. As a method based on the value of the loss, for example, it may be set that the learning ends when the loss converges within a specified range. Also, as a method based on the number of updates, for example, it may be set that the learning ends when the number of updates reaches a specified number. Alternatively, a data set for evaluating the performance of the model may be prepared separately from the training data set, and whether to end the learning may be determined based on the evaluation value using the evaluation data.

[0180] If the determination result in step S68 is a No determination, the processor 202 returns to step S60 and continues the learning process. On the other hand, if the determination result in step S68 is a Yes determination, the processor 202 ends the flowchart in FIG. 13.

[0181] In this way, the generated learned (trained) second machine learning model 22 becomes a disease detection model that receives the input of the unknown medical image IMu and outputs the position and confidence level of the disease for the medical image IMu. The machine learning method executed by the machine learning device 20 can be understood as a method for generating the learned second machine learning model 22, and is an example of the "learned model generation method" in the present disclosure.

[0182] 〔Example of Image Processing Apparatus〕 FIG. 14 is a block diagram showing an example of the hardware configuration of the image processing apparatus 30 according to the embodiment. The image processing apparatus 30 includes a processor 302, a computer-readable medium 304 which is a non-tangible physical entity, a communication interface 306, an input / output interface 308, and a bus 310. The computer-readable medium 304 includes a memory 312 and a storage 314. The processor 302 is connected to the computer-readable medium 304, the communication interface 306, and the input / output interface 308 via the bus 310. Further, the image processing apparatus 30 may further include an input device 322 and a display device 324. The hardware configuration of the image processing apparatus 30 may be the same as the corresponding elements of the label generation apparatus 10 shown in FIG. 2. The processor 302 is an example of the "third processor" in the present disclosure.

[0183] The form of the image processing apparatus 30 is not particularly limited, and it may be a server, or a workstation, a personal computer, or the like.

[0184] The computer-readable medium 304 stores various programs and data including a medical image acquisition program 332, a disease detection program 334, a display form control program 336, a heatmap image generation program 338, an overlay information generation program 340, a synthesis program 342, and a display control program 344.

[0185] The medical image acquisition program 332 includes instructions for executing a process of acquiring a medical image IMu to be processed.

[0186] The disease detection program 334 includes a pre-trained third machine learning model 32. The disease detection program 334 includes instructions for executing a process of inferring the position and confidence level of a disease from the medical image IMu.

[0187] The display form control program 336 includes instructions for executing a process of controlling the display form when displaying the disease detection result obtained from the disease detection program 334.

[0188] The heatmap image generation program 338 includes instructions for executing a process of generating a heatmap image indicating the position and confidence level of a disease based on the disease detection result obtained from the disease detection program 334. The heatmap image represents the distribution of the position and confidence level of the disease. The heatmap image is displayed with colors changed according to the confidence level value. For example, the heatmap image changes colors in the order of red, orange, yellow, green, blue, indigo, and violet from the highest confidence level to represent the distribution of the confidence level values.

[0189] The overlay information generation program 340 includes instructions for executing a process of generating overlay information for the medical image IMu based on the detection result by the disease detection program 334. The overlay information is information about the disease detected from the medical image IMu and may be, for example, characters, symbols, or figures, or a combination thereof. The overlay information may include, for example, a character string specifying the type (disease name) of the disease, a symbol or character string indicating the grade classification of the severity of the disease, a rectangular frame indicating the position of the disease, or a numerical value indicating the size of the disease area.

[0190] The synthesis program 342 includes an instruction for causing the medical image IMu to execute a process of generating a synthesized image in which a heat map image and superimposed information are superimposed.

[0191] The display control program 344 includes an instruction for generating a display signal necessary for display output to the display device 324 and causing the display device 324 to execute display control.

[0192] FIG. 15 is a block diagram schematically showing the functional configuration of the image processing apparatus 30. The image processing apparatus 30 includes a medical image acquisition unit 31, a disease detection unit 34, a display information generation unit 36, and a display control unit 38. The medical image acquisition unit 31 acquires a medical image IMu. The disease detection unit 34 includes a third machine learning model 32, receives an input of the medical image IMu, and outputs an estimation result of the position and confidence of the disease from the medical image IMu.

[0193] The display information generation unit 36 is a processing unit that generates display information for visualizing and displaying the result obtained by inference using the third machine learning model 32. The display information generation unit 36 includes a display form control unit 362, a heat map image generation unit 364, a superimposed information generation unit 366, and a synthesis unit 368.

[0194] The display form control unit 362 controls the display form of information when presenting the detection result of the disease detection unit 34 according to the confidence of the disease output from the third machine learning model 32. The display form control unit 362 may control the display form by changing the visualization process of at least one of the heat map image and the superimposed information. For example, the display form control unit 362 may change the color of the heat map image according to the confidence and display it. Further, the display form control unit 362 may display an alert when a disease with high confidence is detected.

[0195] The heat map image generation unit 364 generates a heat map image showing the position and confidence of the disease based on the output of the third machine learning model 32 and the control from the display form control unit 362.

[0196] The superimposed information generation unit 366 generates superimposed information based on the output of the third machine learning model 32 and the control from the display form control unit 362.

[0197] The synthesizing unit 368 superimposes a heatmap image on the input medical image to generate a synthesized image for display. The synthesizing unit 368 may further generate a synthesized image in which the superimposed information is superimposed on the input medical image.

[0198] The display control unit 38 generates a display signal necessary for display output to the display device 324 and controls the display of the display device 324. The synthesized image generated by the synthesizing unit 368 is displayed on the display device 324 via the display control unit 38.

[0199] FIG. 16 is an explanatory diagram showing an example of an image processing method executed using the third machine learning model 32 mounted on the image processing apparatus 30. Note that the third machine learning model 32 shown in FIG. 16 is a learned model learned using the correct data generated by the example 1 of the label generation method described in FIG. 3.

[0200] The processor 302 of the image processing apparatus 30 inputs an unknown medical image IMu to be processed into the third machine learning model 32 and executes a process of calculating the position and confidence level of a disease with respect to the unknown medical image IMu using the third machine learning model 32.

[0201] Further, the processor 302 executes a process of changing the display form of the disease according to the value of the confidence level of the disease with respect to the medical image IMu obtained using the third machine learning model 32 and causing the display device 324 to display the processing result.

[0202] 〔Regarding the type of disease to be detected〕 The disease to be detected from the chest plain X-ray image is not limited to pleural effusion, and may be, for example, pneumothorax, pulmonary tuberculosis, or an appropriate combination thereof.

[0203] 〔Example 1 of control of display form according to confidence level〕 Figures 17 and 18 are examples of synthetic images displayed on the display device 324 as processing results by the third machine learning model 32. Figures 17 and 18 are examples of a form of displaying information by changing color according to the confidence level calculated by the third machine learning model 32. In Figures 17 and 18, examples of synthetic images in which a heat map image visualizing the position and confidence level of a disease estimated by the third machine learning model 32 is superimposed on a medical image are shown.

[0204] Figure 17 shows an example of an image displayed when a severe disease is detected from a medical image, and Figure 18 shows an example of an image displayed when a mild disease is detected from a medical image. For the heat map image superimposed on the medical image, for example, pixels with relatively high confidence levels use colors on the red side, and pixels with relatively low confidence levels use colors on the purple side. The larger the confidence level value, the more preferably it is displayed in a color that appeals more strongly to the visual sense.

[0205] For the area of a severe disease, it may be displayed in, for example, "red", and for the area of a mild disease, it may be displayed in "blue". As shown in Figure 17, when a severe disease is detected, that is, when a disease with a high confidence level is detected, the heat map image indicating the area of the detected disease is displayed in red.

[0206] On the other hand, as shown in Figure 18, when a mild disease is detected, that is, when a disease with a low confidence level is detected, the heat map image indicating the area of the detected disease is displayed in blue (or purple).

[0207] Note that the correspondence between the confidence level value and the color in the heat map image is not limited to this example, and various definitions are possible.

[0208] 〔Example 2 of control of display form according to confidence level〕 FIG. 19 is an explanatory diagram showing another example of changing the display form according to the confidence level of a disease. FIG. 19 shows a display example of an examination list that provides information on the severity of a disease indicated by the confidence level calculated by the third machine learning model 32. As shown in FIG. 19, an alert 372 is displayed for the data of a patient in whom a disease with high severity is recognized in an examination list in which the examination results of a plurality of patients are listed. In the example of FIG. 19, an alert 372 is attached to the row (record) of the data of patient B. By displaying this alert 372, a doctor can easily identify a patient who needs emergency treatment.

[0209] 〔Types of medical images〕 In the above-described embodiment, the case where a chest plain X-ray image is used as an example of the medical image IM has been described. However, the target medical image is not limited to the chest plain X-ray image, and various medical images taken by various medical devices (modalities) such as CT images, MR images taken using an MRI device, ultrasonic images, PET images, or endoscopic images can be targeted. The image targeted by the technology of the present disclosure is not limited to a 2D image, and may be a 3D image.

[0210] 〔Regarding the hardware configuration of each processing unit〕 The hardware structure of the processing unit that executes various processes such as the data acquisition unit 130, disease detection unit 140, anatomical structure extraction unit 141, disease position conversion unit 142, 3D image analysis unit 143, confidence label conversion unit 14, label data conversion unit 145, association unit 16, learning processing unit 24, loss calculation unit 26, parameter update unit 28 in the label generation device 10, medical image acquisition unit 31, disease detection unit 34, display information generation unit 36, display form control unit 362, heat map image generation unit 364, superimposed information generation unit 366, synthesis unit 368, and display control unit 38 described in the above-described embodiment is, for example, various processors as shown below.

[0211] Various processors include a general-purpose processor such as a CPU, a GPU, and an FPGA (Field Programmable Gate Array) that executes a program and functions as various processing units, a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacturing, and an application-specific integrated circuit (ASIC), and a dedicated electric circuit that is a processor having a circuit configuration designed specifically to execute specific processing.

[0212] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Also, a plurality of processing units may be composed of one processor. As an example of composing a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, one processor is composed of a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of an entire system including a plurality of processing units with one integrated circuit (IC) chip is used. Thus, various processing units are configured by using one or more of the above various processors as a hardware structure.

[0213] Furthermore, the hardware structure of these various processors is more specifically an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0214] 〔Regarding the program for operating the computer〕 A program that causes a computer to implement some or all of the processing functions in each of the label generation device 10, the machine learning device 20, and the image processing device 30 described in the above embodiments can be recorded on a computer-readable medium, which is a non-transitory tangible information storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, and the program can be provided through this information storage medium.

[0215] Alternatively, instead of storing and providing the program in such a non-transitory tangible computer-readable medium, it is also possible to provide the program signal as a download service using a telecommunication line such as the Internet.

[0216] Furthermore, some or all of the processing functions in each of the above devices may be implemented by cloud computing, and it is also possible to provide them as SaaS (Software as a Service).

[0217] 〔Advantages of Embodiments of the Present Disclosure〕 According to the embodiments of the present disclosure described above, the following effects can be obtained.

[0218] (1) The label generation device 10 can efficiently generate correct labels that can contribute to the generation of the second machine learning model 22 by using definitive diagnostic examination information in which the disease position is indefinite or the disease position is recorded at a granularity different from the desired region division granularity.

[0219] (2) The machine learning device 20 can generate a learned second machine learning model 22 that estimates the position of a disease and the confidence level according to the severity of the disease from a medical image by machine learning using the correct data GT generated by the label generation device 10.

[0220] (3) By using the third machine learning model 32, which is a learned model generated by the machine learning device 20, the image processing device 30 can provide information indicating the position and confidence level of a disease in an unknown medical image IMu as information in a form that is intuitively easy for a doctor to understand.

[0221] 〔Others〕 The present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the technical idea of the present disclosure.

Explanation of Reference Numerals

[0222] 1 System 4 Inspection Information Management Device 6 Storage Device 10 Label Generation Device 12 First Machine Learning Model 13 Analysis Model 14 Confidence Level Label Conversion Unit 16 Association Unit 20 Machine Learning Device 22 Second Machine Learning Model 24 Learning Processing Unit 26 Loss Calculation Unit 28 Parameter Update Unit 30 Image Processing Device 31 Medical Image Acquisition Unit 32 Third Machine Learning Model 34 Disease Detection Unit 36 Display Information Generation Unit 38 Display Control Unit 40 Telecommunication Line 102 Processor 104 Computer-Readable Medium 106 Communication Interface 108 Input / Output Interface 110 Bus 112 Memory 114 Storage 122 Input Device 124 Display Device 130 Data Acquisition Unit 132 Medical Image Acquisition Unit 134 Definitive Diagnosis Examination Information Acquisition Unit 140 Disease Detection Unit 141 Anatomical Structure Extraction Unit 142 Disease Location Conversion Unit 143 3D Image Analysis Unit 145 Label Data Conversion Unit 147 Anatomical Structure Extraction Unit 148 Disease Detection Unit 149 Pleural Effusion Volume Calculation Unit 150 Data Storage Unit 162 Medical Image Acquisition Program 164 Definitive Diagnosis Examination Information Acquisition Program 170 Disease Detection Program 171 Anatomical Structure Extraction Program 172 Disease Location Constraint Program 173 Disease Location Conversion Program 183 3D Image Analysis Program 184 Confidence Label Conversion Program 185 Label Data Conversion Program 186 Association Program 187 Correct Answer Data Storage Processing Program 188 Display Control Program 190 Anatomical Structure Extraction Program 191 Disease Detection Program 192 Pleural Effusion Volume Calculation Program 194 Lookup Table 202 Processor 204 Computer Readable Medium 206 Communication Interface 208 Input / Output Interface 210 Bus 212 Memory 214 Storage 222 Input Device 224 Display Device 230 Machine Learning Program 232 Data Acquisition Program 236 Loss Calculation Program 238 Optimizer 240 represents a control program 250 Training data storage unit 302 Processor 304 Computer-readable medium 306 Communication interface 308 Input / output interface 310 Bus 312 Memory 314 Storage 322 Input device 324 Display device 332 Medical image acquisition program 334 Disease detection program 336 Display form control program 338 Heatmap image generation program 340 Overlay information generation program 342 Synthesis program 344 Display control program 362 Display form control unit 364 Heatmap image generation unit 366 Overlay information generation unit 368 Synthesis unit 372 Alert AS1, AS2 Anatomical structure information DD, DD1, DD2, DD2_1, DD2_2 Definitive diagnosis examination information DP1, DP2 Disease location data DS Dataset FP1a, FP1b Region GT, GT1, GT1_2, GT2, GTj Correct answer data IM, IM1, IM2, IMj, IMu Medical image PD Pair data PRj Output data SM, SM1, SM2 Salience map TDS Training dataset S10~S22 Steps of Example 1 of label generation method S30~S50 Steps of Example 3 of label generation method S60~S68 Steps of machine learning method

Claims

1. One or more first processors perform the steps of: obtaining candidate positions of one or more diseases in a first medical image in a first segmentation unit; obtaining diagnostic information for the first medical image, where the position of the disease is uncertain or the position of the disease is specified in a second segmentation unit; converting the diagnostic information into a confidence label according to the severity of the disease; associating the confidence of the disease with respect to the candidate position of the disease obtained from the first medical image according to the confidence label; obtaining a ground truth label of the position and confidence of the disease for the first medical image generated by the association; A label generation method that executes the above steps.

2. In the step of obtaining the ground truth label, the one or more first processors obtain the ground truth label of the position and confidence of the disease in the first segmentation unit or the second segmentation unit. The label generation method according to claim 1.

3. The one or more first processors further perform the step of obtaining anatomical structure information from the first medical image, and in the step of associating the confidence of the disease with respect to the candidate position of the disease, constrain the position of the disease to be located within a desired anatomical structure specified from the anatomical structure information. The label generation method according to claim 1.

4. The diagnostic information is a three-dimensional examination image, and the step of converting the diagnostic information into the confidence label includes recognizing an anatomical structure from the three-dimensional examination image, recognizing the position of the disease from the three-dimensional examination image, and calculating the confidence label of the disease for each anatomical structure from the recognized anatomical structure and the position of the disease. The label generation method according to claim 1.

5. The diagnostic information is sputum test information including the test results of a sputum test, and the step of converting the diagnostic information into the confidence label of the disease includes calculating the confidence label of the disease based on the amount of bacteria collected in the sputum test. The label generation method according to claim 1.

6. In the step of obtaining the candidate positions of the one or more diseases, calculate a saliency map of the disease using a pre-trained first machine learning model. The label generation method according to claim 1.

7. In the step of associating the confidence of the disease with respect to the candidate position of the disease, Weight the confidence label according to the value of the saliency map. The label generation method according to claim 6. **Claim 8** The diagnostic information is information in which the position of the disease is specified in the second division unit. The one or more first processors Obtaining anatomical structure information from the first medical image in a third division unit; Converting the candidate position of the disease in the first division unit to the candidate position of the disease in the third division unit; Further executing the step of converting the confidence label of the second division unit converted from the diagnostic information into the confidence label of the third division unit; In the step of associating the confidence of the disease with the candidate position of the disease, the confidence of the disease corresponding to the confidence label of the third division unit is associated with the candidate position of the disease in the third division unit. In the step of obtaining the correct label, the correct label of the position and confidence of the disease is obtained in the third division unit. The label generation method according to claim 1. **Claim 9** The first medical image is a chest X-ray image, a computed tomography image, or a magnetic resonance image. The label generation method according to claim 1. **Claim 10** The disease targets at least one of pleural effusion, pneumothorax, and pulmonary tuberculosis. The label generation method according to claim 1. **Claim 11** One or more second processors Executing a step of training a second machine learning model by machine learning using training data including a correct label generated by the label generation method according to any one of claims 1 to 10; Generating the learned second machine learning model trained to receive an input of a second medical image and output the position and confidence of a disease for the second medical image. Learned model generation method. **Claim 12** The confidence label of the disease is represented by a continuous value. In the step of training the second machine learning model, the confidence of the disease is regressively predicted from the first medical image by the second machine learning model. The learned model generation method according to claim 11. **Claim 13** The confidence label of the disease is represented by a discrete value. In the step of training the second machine learning model, the confidence of the disease is classified and predicted from the first medical image by the second machine learning model. The learned model generation method according to claim 11. **Claim 14** One or more third processors Performing a step of calculating the position and confidence level of a disease for the second medical image using the learned second machine learning model generated by the learned model generation method according to claim 11 An image processing method

15. The one or more third processors Further performing a step of changing a display form of the disease according to a value of the confidence level of the disease for the second medical image The image processing method according to claim 14

16. A label generation device including one or more first processors, The one or more first processors Performing a process of obtaining one or more candidate positions of diseases from a first medical image in a first division unit, Performing a process of obtaining diagnostic information for the first medical image, where the position of the disease is uncertain or the position of the disease is specified in a second division unit, Performing a process of converting the diagnostic information into a confidence label according to the severity of the disease, Performing a process of associating the confidence level of the disease corresponding to the confidence label with the candidate position of the disease obtained from the first medical image, Performing a process of obtaining a correct label of the position and confidence level of the disease for the first medical image generated by the associating process, A label generation device that performs the above

17. A machine learning device including one or more second processors, The one or more second processors Performing a process of training a second machine learning model by machine learning using training data including a correct label generated by the label generation method according to any one of claims 1 to 10, Training the second machine learning model to receive an input of a second medical image and output the position and confidence level of a disease in the second medical image from the second machine learning model, A machine learning device

18. An image processing device including one or more third processors, The one or more third processors Performing a process of calculating the position and confidence level of a disease for the second medical image using the learned second machine learning model generated by the learned model generation method according to claim 11, An image processing device

19. A program for causing a computer to execute the label generation method according to any one of claims 1 to 10

20. A program for causing a computer to execute the learned model generation method according to claim 11

21. A program for causing a computer to execute the image processing method according to claim 14.

Images