Learning device, learning method, recording medium, and program

Abstract

Provided are a learning device, a learning method, a trained model, and a program which can perform learning for detecting a disease at reduced cost, efficiently, and highly accurately. A learning device (100) comprises: a processor (129); memory (114) which stores a data set of a medical image and lesion information included in the medical image; and an attention mechanism (128)-attached learning model (126) which estimates a disease from the input medical image. The processor performs a process for specifying a position in organ labeling information (206) about an area of interest indicated in an attention map (208) and outputting the specified result (210), a process for comparing the estimation result (212) and the lesion information (204) and calculating an error, a process for resetting the error on the basis of the specified result (210), and a process for training the learning model (126) by using the set error.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, a trained model, and a program, and more particularly to a learning device, a learning method, a trained model, and a program for performing training on medical images. [Background technology]

[0002] In recent years, a technology has been proposed for detecting diseases from medical images by performing machine learning using multi-layer neural networks.

[0003] For example, Non-Patent Document 1 proposes a semi-supervised deep learning method that simultaneously classifies input medical images as nodules or normal and detects the location of nodules. Non-Patent Document 2 proposes a deep learning method that calculates attention for a model that learns disease classification and detects the location of nodules. Non-Patent Document 3 proposes a method for correcting the attention of a disease classification model using radiology reports. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Pesce, Emanuele, et al. "Learning to detect chest radiographs containing pulmonary lesions using visual attention networks.", Medical image analysis, 2019. [Non-patent document 2] Guan, Qingji, et al. "Diagnose like a radiologist: Attention guided convolutional neural network for thorax disease classification." arXiv preprint arXiv:1801.09927, 2018. [Non-patent document 3] Kashyap, Satyananda, et al. "Looking in the Right Place for Anomalies: Explainable Ai Through Automatic Location Learning.", IEEE ISBI, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Methods for detecting diseases from medical images can be broadly divided into semi-supervised learning methods for disease detection using class labels and location labels, and methods that calculate attention by learning using only class labels. The former requires annotation of the location of the disease, which is costly (time and effort). The latter is relatively inexpensive, but does not provide high accuracy in disease detection.

[0006] The present invention has been made in consideration of these circumstances, and its purpose is to provide a learning device, a learning method, a trained model, and a program that can efficiently learn to detect diseases with high accuracy while reducing costs. [Means for solving the problem]

[0007] A learning device, which is one aspect of the present invention for achieving the above-mentioned object, is a learning device comprising a processor, a memory for storing a dataset of medical images and lesion information contained in the medical images, and a learning model with an attention mechanism for inferring diseases from input medical images, wherein the learning model performs the following processes: outputting an attention map indicating a region of interest in the medical image using the attention mechanism from the input medical image; and outputting an estimation result of the disease estimated from the input medical image; and the processor performs the following processes: identifying the position in the organ labeling information of the region of interest indicated in the attention map based on organ labeling information indicating the type and region of the organ appearing in the medical image and the attention map, and outputting the identification result; comparing the estimation result with the lesion information to calculate an error; setting an error based on the identification result; and having the learning model perform learning using the set error.

[0008] According to this aspect, based on the organ labeling information and the attention map, an identification result is output that identifies the position in the organ labeling information of the region of interest indicated in the attention map. Then, this aspect uses this identification result to set an error obtained by comparing the estimation result with the lesion information, and performs learning using the set error. This allows this aspect to efficiently perform highly accurate disease detection learning.

[0009] Another aspect of the present invention is a learning device comprising a processor, a memory for storing a dataset of medical images and lesion information contained in the medical images, and a learning model with an attention mechanism for inferring diseases from input medical images. The learning model performs the following processes: outputting an attention map indicating a region of interest in the medical image using the attention mechanism from the input medical image; and outputting an estimation result of the disease estimated from the input medical image. The processor performs the following processes: identifying the position in the organ labeling information of the region of interest indicated in the attention map based on organ labeling information indicating the type and region of the organ appearing in the medical image and the attention map, and outputting the identification result; imposing constraints on the medical image based on the identification result; inputting the constrained medical image into the learning model; comparing the estimation result output by the learning model based on the constrained medical image with the lesion information to calculate an error; and causing the learning model to learn using the error.

[0010] According to this aspect, based on the organ labeling information and the attention map, an identification result is output that identifies the position in the organ labeling information of the region of interest indicated in the attention map. Then, this aspect uses this identification result to impose constraints on the medical image, and the constrained medical image is input into a learning model for learning. This allows this aspect to efficiently perform highly accurate disease detection learning.

[0011] Preferably, when the position of the region of interest in the identification result differs from the position corresponding to the lesion information, the processor increases the error when setting the region of interest.

[0012] Preferably, if the position of the region of interest in the identification result differs from the position corresponding to the lesion information, the processor imposes a constraint on a location corresponding to the region of interest in the medical image.

[0013] Preferably, the system includes a segmentation trained model that acquires organ labeling information from medical images.

[0014] Preferably, the attention map is configured as a heat map indicating the degree of contribution to the estimation of the learning model.

[0015] Preferably, the medical image is a chest X-ray image.

[0016] Preferably, the organ labeling information includes at least information indicating the region of the aorta in the medical image.

[0017] Preferably, the organ labeling information includes at least information indicating a lung region in the medical image.

[0018] Preferably, the lesion information includes information regarding pulmonary nodules.

[0019] Preferably, the organ labeling information includes at least information indicating the region of the heart in the medical image.

[0020] Preferably, the lesion information includes information regarding pleural effusion.

[0021] Another aspect of the present invention is a learning method in which a processor uses a dataset of medical images and lesion information contained in the medical images stored in a memory to train a learning model with an attention mechanism that infers diseases from input medical images.The learning model performs the following processes: using the attention mechanism to output an attention map indicating a region of interest in the medical image from the input medical image; and outputting an estimation result of the disease inferred from the input medical image.The learning model includes the steps of identifying the position of the region of interest indicated in the attention map in the organ labeling information based on organ labeling information and the attention map indicating the type and region of the organ appearing in the medical image, and outputting the identification result; comparing the estimation result with the lesion information to calculate an error; setting an error based on the identification result; and having the learning model perform learning using the set error.

[0022] Another aspect of the present invention is a learning method in which a processor uses a dataset of medical images and lesion information contained in the medical images stored in a memory to cause a learning model with an attention mechanism to perform learning, which estimates a disease from an input medical image. The learning model performs the following processes: outputting an attention map indicating a region of interest in the medical image using the attention mechanism from the input medical image; and outputting an estimation result of a disease estimated from the input medical image. The learning method includes the steps of: identifying the position of the region of interest indicated in the attention map in the organ labeling information based on organ labeling information indicating the type and region of the organ appearing in the medical image and the attention map, and outputting the identification result;

[0023] Another aspect of the present invention is a program that causes a processor to execute each step of the above-described learning method.

[0024] The trained model, which is another aspect of the present invention, is trained using the above-mentioned training method. [Effects of the Invention]

[0025] According to the present invention, based on organ labeling information, attention map, and lesion information, an identification result is output that identifies the position in the organ labeling information of the region of interest indicated in the attention map, and this identification result is used to learn to detect diseases from medical images, thereby enabling efficient and highly accurate learning to detect diseases. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram showing an embodiment of the hardware configuration of a learning device. [Figure 2]FIG. 2 is a block diagram illustrating the main functions of the learning device. [Figure 3] FIG. 3 is a diagram illustrating a chest X-ray image and lesion information, which are examples of a data set. [Figure 4] FIG. 4 is a diagram illustrating a learning model with an attention mechanism. [Figure 5] FIG. 5 is a diagram illustrating an attention map output from the attention mechanism of the learning model. [Figure 6] FIG. 6 is a diagram showing an example of organ labeling information acquired from a chest X-ray image. [Figure 7] FIG. 7 is a diagram conceptually illustrating the output of the identification result in the identification unit. [Figure 8] FIG. 8 is a diagram for explaining the calculation and setting of the error in the error acquisition unit. [Figure 9] FIG. 9 is a flow diagram illustrating a learning method using the learning device and each step executed by a processor according to a program. [Figure 10] FIG. 10 is a block diagram illustrating the main functions of the learning device. [Figure 11] FIG. 11 is a diagram for explaining the generation of a constraint image in the constraint image generating unit. [Figure 12] FIG. 12 explains a learning method using the learning device and a program that causes a processor to execute each step. [Figure 13] FIG. 13 is a diagram illustrating a pseudo X-ray image obtained from a CT image. [Figure 14] FIG. 14 is a diagram illustrating generation of a separated image. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, preferred embodiments of a learning device, a learning method, a trained model, and a program according to the present invention will be described with reference to the accompanying drawings.

[0028] FIG. 1 is a block diagram showing an embodiment of the hardware configuration of a learning device.

[0029] The learning device 100 shown in FIG. 1 is configured as a computer. The computer may be a personal computer, a workstation, or a server computer. The learning device 100 includes a communication unit 112, a memory (storage unit) 114, a learning model 126 with an attention mechanism 128, an operation unit 116, a CPU (Central Processing Unit) 118, a GPU (Graphics Processing Unit) 119, a RAM (Random Access Memory) 120, a ROM (Read Only Memory) 122, and a display unit 124. The CPU 118 and the GPU 119 constitute a processor 129. The GPU 119 may be omitted.

[0030] The communication unit 112 is an interface that performs communication processing with an external device via wire or wirelessly, and exchanges information with the external device.

[0031] The memory 114 includes a storage device configured using, for example, a hard disk drive, an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination of these. The memory 114 stores various programs and data required for image processing such as learning processing and / or image generation processing. The programs stored in the memory 114 are loaded into the RAM 120 and executed by the processor 129, causing the computer to function as a means for performing various processes defined by the programs.

[0032] Operation unit 116 is an input interface that accepts various operational inputs to study device 100. Operation unit 116 may be, for example, a keyboard, a mouse, a touch panel, operation buttons, or a voice input device, or an appropriate combination of these.

[0033] The processor 129 reads out various programs stored in the ROM 122 or the memory 114, etc., and executes various processes. The RAM 120 is used as a working area for the processor 129. The RAM 120 is also used as a storage unit that temporarily stores the read programs and various data.

[0034] The display unit 124 is an output interface that displays various types of information. The display unit 124 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination of these.

[0035] The learning model 126 is configured as a CNN (Convolutional Neural Network) and detects diseases from chest X-ray images. The learning model 126 is preferably configured as a DenseNet (Densely connected convolutional networks). The learning model 126 also has an attention mechanism 128. When a medical image is input to the learning model 126, it outputs an attention map 208 indicating a region of interest in the input medical image along with an estimation result 212 (see FIG. 2). The learning model 126 in the learning device 100 is an untrained model, and the learning device 100 according to the present invention trains the learning model 126 by machine learning. A detailed description of the learning model 126 will be provided later.

[0036] First Embodiment A description will be given of a first embodiment of the learning device 100. In the following description, an example will be described in which learning is performed to detect pulmonary nodules from chest X-ray images, which are an example of medical images.

[0037] FIG. 2 is a block diagram illustrating the main functions of the learning device 100 of this embodiment.

[0038] The learning device 100 is mainly composed of a memory 114, a processor 129, and a learning model 126. The processor 129 realizes the functions of a learning data acquisition unit 130, an organ labeling information generation unit 132, an identification unit 134, an error acquisition unit 136, and a learning control unit 138.

[0039] The learning data acquisition unit 130 acquires a data set to be used for learning stored in the memory 114. For example, the data set is made up of chest X-ray images and lesion information included in the chest X-ray images.

[0040] FIG. 3 is a diagram illustrating a chest X-ray image and lesion information, which are examples of a data set.

[0041] The dataset 200 is composed of a set of chest X-ray images 202 and lesion information 204. The memory 114 stores a plurality of datasets 200, and the learning model 126 is trained using these plurality of datasets 200.

[0042] The chest X-ray image 202 is a so-called real X-ray image or plain X-ray image, and is an X-ray image actually taken of a patient who is the subject. Note that, as will be explained later, images other than real X-ray images can be used as the chest X-ray image 202. Furthermore, the chest X-ray image 202 is an example of a medical image, and medical images taken of other body parts can also be used in this embodiment.

[0043] The lesion information 204 corresponds to the chest X-ray image 202 and includes information (disease label) about the disease in the chest X-ray image 202. For example, the lesion information 204 includes an X-ray image ID "ZXXXX" associated with the chest X-ray image 202 and a disease label "pulmonary nodule" obtained by interpreting the chest X-ray image 202. The disease label is obtained by a doctor interpreting the chest X-ray image 202. In the illustrated example, the doctor detected a pulmonary nodule by interpreting the chest X-ray image 202, so the lesion information 204 is assigned the disease label "pulmonary nodule." This disease label serves as a correct label for disease detection performed by the learning model 126. A plurality of datasets 200 are stored in the memory 114, and the learning device 100 uses the plurality of datasets 200 to train the learning model 126.

[0044] The learning data acquisition unit 130 sends the chest X-ray image 202 acquired from the memory 114 to the learning model 126 and the organ labeling information generation unit 132 , and sends the lesion information 204 to the error acquisition unit 136 .

[0045] The learning model 126 has an attention mechanism 128, and when a chest X-ray image 202 is input, it outputs an inference result 212 and an attention map 208.

[0046] FIG. 4 is a diagram illustrating the learning model 126 with the attention mechanism 128.

[0047] The illustrated learning model 126 is configured with DenseNet, an example of a CNN. As illustrated, DenseNet includes multiple dense blocks D1-D3 and multiple transition layers T1-T4 before and after the dense blocks D1-D3, and has a network structure that exhibits high performance in classification tasks (e.g., disease detection). Within the dense blocks D1-D3, skip connections are imposed on all layers to reduce gradient vanishing. The transition layers T1-T4 include convolutional layers and / or pooling layers. The learning model 126 extracts features from an input chest X-ray image 202 and outputs the presence or absence of a disease (pulmonary nodule) as an inference result 212. The learning model 126 is trained to classify the chest X-ray image 202 into two categories, "pulmonary nodule present" and "pulmonary nodule absent," and therefore outputs the inference result 212 as a score corresponding to "pulmonary nodule present" and "pulmonary nodule absent." In the illustrated case, the score "pulmonary nodule present [0.8]" is output as the estimation result 212.

[0048] The learning model 126 also has an attention mechanism 128, which outputs an attention map 208 from the input chest X-ray image 202. The attention mechanism 128 generates the attention map 208 based on a feature map extracted when the chest X-ray image 202 is input to DenseNet. The attention map 208 can be generated using a known method, such as the method described in the publication "Wang, Fei, et al. "Residual attention network for image classification.", CVPR, 2017." Another method for generating the attention map 208 is a method called GradCAM, which calculates class activation mapping for any trained network using the weight gradient. An example of this method is described in the publication "Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization.", ICCV, 2017."

[0049] FIG. 5 is a diagram illustrating the attention map 208 output from the attention mechanism 128 of the learning model 126.

[0050] The attention map 208 visualizes the regions of interest in the chest X-ray image 202 input to the learning model 126, and has a heat map HM that indicates the degree of contribution to the estimation of the learning model 126. The heat map HM shows three regions with different degrees of interest. The heat map HM has, in descending order of interest level, regions of interest R3, R2, and R1. In the attention map 208, regions of interest R3, R2, and R3 are located in the aortic region (see FIG. 6).

[0051] The organ labeling information generation unit 132 receives the chest X-ray image 202 as input from the training data acquisition unit 130. Based on the chest X-ray image 202, the organ labeling information generation unit 132 generates organ labeling information 206 of the organs appearing in the chest X-ray image 202. Here, the organ labeling information 206 is information that identifies the type and area of ​​each organ appearing in the chest X-ray image 202. The organ labeling information generation unit 132 can generate the organ labeling information 206 by various known methods. For example, the organ labeling information generation unit 132 generates the organ labeling information 206 from the chest X-ray image 202 by using a segmentation trained model.

[0052] FIG. 6 is a diagram showing an example of organ labeling information 206 acquired from a chest X-ray image 202.

[0053] The organ labeling information 206 has labeling information for each organ shown in the chest X-ray image 202. Specifically, the organ labeling information 206 has left lung labeling information 206A, right lung labeling information 206B, heart labeling information 206C, and aorta labeling information 206D. Here, the left lung labeling information 206A indicates the left lung region in the chest X-ray image 202, the right lung labeling information 206B indicates the right lung region in the chest X-ray image 202, the heart labeling information 206C indicates the heart region in the chest X-ray image 202, and the aorta labeling information 206D indicates the aorta region in the chest X-ray image 202. Note that the above-described example of organ labeling information 206 is just an example, and other forms of organ labeling information may be generated. For example, as another example of organ labeling information, organ labeling information composed of lung labeling information and labeling information of organs other than the lungs may be generated.

[0054] The identification unit 134 receives the attention map 208 and the organ labeling information 206. Based on the received attention map 208 and organ labeling information 206, the identification unit 134 identifies the positions of the regions of interest R1 to R3 indicated in the attention map 208 in the organ labeling information 206, and outputs the identification result 210.

[0055] FIG. 7 is a diagram conceptually illustrating the output of the identification result 210 in the identification unit 134. As shown in FIG.

[0056] The identification unit 134 receives the organ labeling information 206 and the attention map 208. The identification unit 134 compares the attention map 208 with the organ labeling information 206 to identify the positions of the regions of interest in the organ labeling information 206. In the illustrated example, the identification unit 134 identifies that the regions of interest R1, R2, and R3 are located on the aorta labeling information 206D. The identification unit 134 then outputs the fact that the regions of interest R1, R2, and R3 are located on the aorta labeling information 206D as an identification result 210.

[0057] The error acquiring unit 136 receives input of the lesion information 204, the estimation result 212, and the identification result 210. The error acquiring unit 136 calculates the error (calculated error 214: see FIG. 8) between the lesion information 204 and the estimation result 212, and sets the calculated error 214 based on the identification result 210 (set error 216: see FIG. 8). Specifically, the error acquiring unit 136 acquires the difference between the score of the lesion information 204 (disease label) and the score of the estimation result 212, and sets the difference as the calculated error 214. Furthermore, if the position corresponding to the lesion information 204 does not match the region of interest, the error acquiring unit 136 increases the calculated error 214 (adds the error) and sets the set error 216. Here, the position corresponding to the lesion information 204 is the position where the disease in the lesion information 204 is detected. For example, if the lesion information 204 is a pulmonary nodule, the position corresponding to the lesion information 204 is the lung region. The error acquiring unit 136 has a list in which disease names are associated with the locations where the diseases are detected, and acquires the position corresponding to the lesion information 204 from the list. Alternatively, the lesion information 204 may include the position corresponding to the lesion information 204, and the error acquiring unit 136 may acquire the position corresponding to the lesion information 204. The error acquiring unit 136 also switches the setting of the calculated error 214 depending on the identification result 210. Specifically, if the position corresponding to the lesion information 204 differs from the position of the region of interest, the error acquiring unit 136 sets the error, and if the position corresponding to the lesion information 204 matches the position of the region of interest, the error acquiring unit 136 does not set the error. If the position corresponding to the lesion information 204 matches the position of the region of interest, the learning model 126 outputs the estimation result 212 using the feature amount of the appropriate region, and therefore appropriate learning can be performed without setting the calculated error 214. On the other hand, if the position corresponding to the lesion information 204 does not match the position of the region of interest, the learning model 126 does not output the estimation result 212 using the feature amount of the appropriate region, and therefore it is necessary to perform learning using the set error 216 obtained by setting the calculation error 214 and increasing the error. The acquisition of the calculation error 214 and the set error 216 by the error acquisition unit 136 will be specifically described below.

[0058] Fig. 8 is a diagram for explaining the calculation and setting of the error by the error acquiring unit 136. Note that the case explained in Fig. 8 is an example in which the position corresponding to the lesion information 204 is different from the region of interest, and an error is set.

[0059] The error acquiring unit 136 acquires a calculation error 214 based on the lesion information 204 and the estimation result 212. The lesion information 204 is a correct label (correct data) in machine learning. In the illustrated example, the lesion information 204 is a pulmonary nodule, so the score for the lesion information 204 is "pulmonary nodule present [1.0]." The estimation result 212 is the result of estimating the disease output by the learning model 126, and the score for the estimation result 212 is "pulmonary nodule [0.8]." The error acquiring unit 136 then calculates the error between the score of the lesion information 204 and the estimation result 212, and acquires the calculation error 214 of "pulmonary nodule present [0.2]."

[0060] Next, the error acquiring unit 136 sets a set error 216 based on the identification result 210. Here, since the lesion information 204 is a pulmonary nodule, the position corresponding to the lesion information 204 is "lung." The error acquiring unit 136 then compares the position corresponding to the lesion information 204 ("lung") with the position of the region of interest in the identification result 210 ("aorta"). In the illustrated example, the position corresponding to the lesion information 204 does not match the position of the region of interest in the identification result 210. In such a case, for example, the error acquiring unit 136 adds a score of 4.8 to the score of the calculated error 214 (0.2), to set the set error 216 (score 5.0). As a result, the set error 216 becomes [5.0], which means that the region of interest does not match the position corresponding to the lesion information. Note that the setting of the set error 216 described above is merely an example and is not limited thereto. If the position corresponding to the lesion information 204 does not match the position of the region of interest in the identification result 210 , an error is added so that at least the set error 216 has a larger score than the calculated error 214 .

[0061] In this way, in this embodiment, even if the estimation result 212 is close to the lesion information 204 (correct label) (the error is small), if the region of interest identification result 210 is not in the region corresponding to the lesion information 204, the error will be set to be large.

[0062] The learning control unit 138 adjusts the coefficients of the filters applied to the convolutional layer in the learning model 126, the offset values, and the weights of the connections between the previous and next layers by the error backpropagation method, based on the errors acquired by the error acquisition unit 136. Here, the learning control unit 138 uses the set error 216 when the set error 216 has been set by the error acquisition unit 136, and uses the calculated error 214 when the set error 216 has not been set, to train the learning model 126. Then, the learning control unit 138 repeatedly adjusts the coefficients and the like in the learning model 126, and causes the learning model 126 to train so that the difference between the estimation result 212 of the learning model 126 and the lesion information 204 becomes smaller.

[0063] The learning control unit 138 sequentially learns the learning model 126 in mini-batches of one set of data sets 200 stored in the memory 114 or around 10 to 100 sets, and performs learning to optimize coefficients, etc. in the learning model 126, thereby converting the unlearned learning model 126 into a trained learning model 126.

[0064] Next, a learning method using the learning device 100 of the first embodiment and a program that causes the processor 129 to execute each step will be described.

[0065] FIG. 9 is a flow diagram illustrating a learning method using the learning device 100 of the first embodiment and each step executed by a processor according to a program.

[0066] First, the learning data acquisition unit 130 of the learning device 100 reads the chest X-ray image 202 from the memory 114 and inputs it to the learning model 126 (step S10). Then, the attention mechanism 128 provided in the learning model 126 outputs an attention map 208 based on the input chest X-ray image 202 (step S11). The learning model 126 also performs disease detection based on the input chest X-ray image 202 and outputs the estimation result 212 (step S12). Then, the error acquisition unit 136 acquires an error (calculated error 214) based on the lesion information 204 and the estimation result 212 (step S13). Next, the identification unit 134 identifies the position of the region of interest of the attention map 208 in the organ labeling information 206 based on the organ labeling information 206 and the attention map 208 generated by the organ labeling information generation unit 132, and outputs the identification result 210 (step S14). Then, the error acquiring unit 136 determines whether or not the position of the region of interest matches the position corresponding to the lesion information 204 based on the identification result 210 (step S15).

[0067] If the position of the region of interest does not match the position corresponding to the lesion information 204, the error acquisition unit 136 sets an error calculated based on the identification result 210 (step S16). Next, the learning control unit 138 causes the learning model 126 to perform learning using the set error (set error 216) (step S17). On the other hand, if the position of the region of interest matches the position corresponding to the lesion information 204, the learning control unit 138 causes the learning model 126 to perform learning using the calculated error (calculated error 214) (step S18). The learning model 126 trained by the above-mentioned learning method can perform disease detection with high accuracy.

[0068] As described above, the learning device 100 of this embodiment outputs an identification result 210 that identifies the position in the organ labeling information 206 of the region of interest indicated by the attention map 208, based on the organ labeling information 206 and the attention map 208. Then, based on this identification result 210, the learning device 100 sets an error obtained by comparing the estimation result 212 with the lesion information 204, and performs learning using the set error 216. This allows the learning device 100 to efficiently perform highly accurate disease detection learning without requiring tasks such as annotation by the user.

[0069] <Second embodiment> Next, a second embodiment of the learning device 100 will be described. In this embodiment, a constraint image is generated based on the identification result 210, and the generated constraint image is input to the learning model 126 for learning. In the following description, an example of detecting pulmonary nodules from a chest X-ray image 202 will be described, and differences from the first embodiment will be mainly described.

[0070] 10 is a block diagram illustrating the main functions of the learning device 100 of the second embodiment. Note that the same reference numerals are used to denote parts that have already been explained in FIG. 2, and explanations thereof will be omitted.

[0071] The learning device 100 is mainly composed of a memory 114, a processor 129, and a learning model 126. The processor 129 constitutes a learning data acquisition unit 130, an organ labeling information generation unit 132, an identification unit 134, a constraint image generation unit 142, an error acquisition unit 136, and a learning control unit 138. In this embodiment, instead of setting the setting error 216 in the first embodiment, a constraint image 232 is generated and input to the learning model 126.

[0072] The constraint image generation unit 142 receives the identification result 210, the chest X-ray image 202, and the lesion information 204. If the position of the region of interest does not match the position corresponding to the lesion information 204, the constraint image generation unit 142 imposes a constraint on the chest X-ray image 202. On the other hand, if the position of the region of interest matches the position corresponding to the lesion information 204, the constraint image generation unit 142 does not generate the constraint image 232.

[0073] FIG. 11 is a diagram for explaining the generation of the constraint image 232 in the constraint image generating unit 142. As shown in FIG.

[0074] The constraint image generation unit 142 receives the identification result 210, lesion information 204, and organ labeling information 206. The lesion information 204 indicates the presence of a pulmonary nodule, and the region corresponding to the lesion information 204 is a lung region. In this embodiment, the region corresponding to the lesion information 204 is acquired by the constraint image generation unit 142. In the identification result 210, regions of interest R1, R2, and R3 are located in the aortic region, so the positions of the regions of interest do not match the positions corresponding to the lesion information 204. In this case, in this embodiment, a constraint image 232 is generated by constraining the aorta. Here, the constraint image generation unit 142 generates the constraint image 232 by imposing constraints on the aortic region using various methods. In the illustrated example, the constraint image generation unit 142 generates the constraint image 232 by imposing constraints by masking the aortic region. Here, masking the aortic region means cutting out an image of the region corresponding to the aortic labeling information from the chest X-ray image 202.

[0075] The constrained image 232 generated by the constrained image generation unit 142 is input to the learning model 126. The learning model 126 then outputs an inference result 212 and an attention map 208 based on the constrained medical image. When the constrained image 232 is input, the learning model 126 does not generate a feature map in the constrained region, and the region of interest in the attention map 208 is no longer located in the constrained region. This allows the learning model 126 to learn to output an inference result 212 based on features in an appropriate region.

[0076] Next, a learning method using the learning device 100 of the second embodiment and a program that causes the processor 129 to execute each step will be described.

[0077] FIG. 12 is a flow diagram illustrating the steps executed by a processor according to a learning method and program using the learning device 100 of the second embodiment.

[0078] The learning data acquisition unit 130 of the learning device 100 reads out the chest X-ray image 202 from the memory 114 and inputs it to the learning model 126 (step S30). Then, the attention mechanism 128 provided in the learning model 126 outputs an attention map 208 based on the input chest X-ray image 202 (step S31). The learning model 126 also performs disease detection based on the input chest X-ray image 202 and outputs the estimation result 212 (step S32). Next, the organ labeling information 206 and the attention map 208 generated by the organ labeling information generation unit 132 are input to the identification unit 134, and the identification unit 134 identifies the position of the region of interest of the attention map 208 in the organ labeling information 206 and outputs the identification result 210 (step S33). Then, the error acquiring unit 136 determines whether or not the position of the region of interest matches the position corresponding to the lesion information 204 based on the identification result 210 (step S34).

[0079] If the position of the region of interest does not match the position corresponding to the lesion information 204, the constraint image generation unit 142 generates a constrained chest X-ray image 202 based on the identification result 210 (step S35). The generated constraint image 232 is then input to the learning model 126 (step S36). The error acquisition unit 136 then calculates the error between the estimation result 212 and the lesion information 204 when the constraint image 232 is input, and acquires the calculation error 214 (step S37). The learning control unit 138 then causes the learning model 126 to perform learning using the calculation error 214 (step S38). On the other hand, if the position of the region of interest matches the position corresponding to the lesion information 204, the error acquisition unit 136 calculates the error between the estimation result 212 and the lesion information 204 when the unconstrained chest X-ray image 202 is input, and acquires the calculation error 214 (step S37). Next, the learning control unit 138 causes the learning model 126 to perform learning using the calculation error 214 (step S38). The learning model 126 that has been trained by the above-mentioned learning method can perform disease detection with high accuracy.

[0080] As described above, the learning device 100 of this embodiment outputs an identification result 210 that identifies the position in the organ labeling information 206 of the region of interest indicated by the attention map 208, based on the organ labeling information 206 and the attention map 208. Then, in this aspect, based on this identification result 210, a constrained image 232, in which a constraint is imposed on the chest X-ray image 202, is input to the learning model 126, and learning is performed. This allows the learning device 100 to efficiently perform highly accurate disease detection learning without requiring work such as annotation by the user.

[0081] <Examples of other diseases> In the above example, an example of a combination of the lungs (region corresponding to the lesion information 204), nodules (lesion information 204), and aorta (region excluded or constrained) has been described. However, the present invention may be applied to combinations with other target organs, abnormal (disease) regions, and regions with constraints. For example, in a plain X-ray image, the present invention can be similarly applied to the heart (region with constraints) for pleural effusion (lesion information 204) that has the characteristic of accumulating below the lungs (region corresponding to the lesion information 204).

[0082] <Use of pseudo-X-ray images> In the above description, a chest X-ray image 202 is used as an example of a medical image. However, other images can also be used as medical images in the present invention. For example, a pseudo X-ray image 304 obtained by projecting a CT (computed tomography) image is another example of a medical image.

[0083] FIG. 13 is a diagram illustrating a pseudo X-ray image 304 obtained from a CT image 302.

[0084] The CT image 302 has three-dimensional information. By projecting this CT image 302, a pseudo X-ray image 304 having two-dimensional information can be generated. For example, the pseudo X-ray image 304 may be generated from the CT image 302 using a trained model that has been trained to generate the pseudo X-ray image 304 based on the CT image 302. Note that in the X-ray CT image 302, cross sections 600S, 600C, and 600A are cross sections in the sagittal direction, coronal direction, and axial direction, respectively.

[0085] In this way, the pseudo plain X-ray image generated from the CT image 302 can be used as the medical image applied to the present invention.

[0086] <Generation and use of separated images> In the above example, the organ labeling information generation unit 132 generates the organ labeling information 206 based on the chest X-ray image 202, and the organ labeling information 206 is input to the identification unit 134. In the present invention, instead of the organ labeling information 206, separated images, which are images separated for each organ, may be input to the identification unit 134.

[0087] FIG. 14 is a diagram illustrating generation of a separated image.

[0088] The generator 306 is a trained model that generates separated images for each organ from an X-ray image. In the illustrated example, a chest X-ray image 202 is input to the generator 306, and separated images 308A to 308D separated for each organ are generated. Specifically, the generator 306 generates a right lung separated image 308A, a left lung separated image 308B, a heart separated image 308C, and an aorta separated image 308D from the chest X-ray image 202. The separated images 308A to 308D are then input to the identification unit 134 instead of the organ labeling information 206. The identification unit 134 outputs an identification result 210 based on the separated images 308A to 308D and the attention map 208.

[0089] <Other> In the above embodiment, the hardware structure of the processing unit that executes various processes is the following various processors: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0090] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0091] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0092] The above-described configurations and functions can be realized by any hardware, software, or a combination of both. For example, the present invention can be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) on which such a program is recorded, or a computer on which such a program can be installed.

[0093] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0094] 100: Learning device 112: Communications Department 114: Memory 116:Operation unit 118:CPU 120:RAM 122:ROM 124: Display section 126: Learning model 128: Attention mechanism 129: Processor 130: Learning data acquisition unit 132: Organ labeling information generation unit 134: Specific part 136:Error acquisition section 138: Learning control unit 142: Constraint image generation unit

Claims

1. A learning device comprising: a processor; a memory that stores a data set of medical images and lesion information included in the medical images; and a learning model with an attention mechanism that estimates a disease from the input medical images, The learning model is a process of outputting an attention map indicating a region of interest in the input medical image by the attention mechanism from the input medical image; and outputting a result of the disease estimation based on the input medical image. The processor: a process of identifying the position of the region of interest indicated by the attention map in the organ labeling information based on organ labeling information indicating the type and region of the organ shown in the medical image and the attention map, and outputting the identification result; A process of comparing the estimation result with the lesion information to calculate an error; a process of setting the error based on the identification result and an organ corresponding to the lesion information; A process of causing the learning model to learn using the set error; A learning device that performs the following:

2. A learning device comprising: a processor; a memory that stores a data set of medical images and lesion information included in the medical images; and a learning model with an attention mechanism that estimates a disease from the input medical images, The learning model is a process of outputting an attention map indicating a region of interest in the input medical image by the attention mechanism from the input medical image; and outputting a result of the disease estimation based on the input medical image. The processor: a process of identifying the position of the region of interest indicated by the attention map in the organ labeling information based on organ labeling information indicating the type and region of the organ shown in the medical image and the attention map, and outputting the identification result; a process of imposing constraints on the medical image based on the identification result and the organ corresponding to the lesion information; inputting the constrained medical image into the learning model; a process of calculating an error by comparing an estimation result output by the learning model based on the medical image to which the constraints are applied with the lesion information; a process of training the learning model using the error; A learning device that performs the following:

3. The processor: The learning device according to claim 1 , wherein when the position of the region of interest in the identification result differs from the position corresponding to the lesion information, the error is increased and set.

4. The processor: The learning device according to claim 2 , wherein when the position of the region of interest in the identification result differs from the position corresponding to the lesion information, a constraint is imposed on a location of the medical image corresponding to the region of interest.

5. The learning device according to claim 1 , further comprising a segmentation trained model that acquires the organ labeling information from the medical image.

6. The learning device according to claim 1 , wherein the attention map is configured as a heat map indicating a degree of contribution to the estimation of the learning model.

7. The learning device according to claim 1 , wherein the medical image is a chest X-ray image.

8. The learning device according to claim 7 , wherein the organ labeling information includes information indicating at least an aorta region in the medical image.

9. The learning device according to claim 7 or 8, wherein the organ labeling information includes information indicating at least a lung region in the medical image.

10. The learning device according to claim 7 , wherein the lesion information includes information about pulmonary nodules.

11. The learning device according to claim 7 , wherein the organ labeling information includes information indicating at least a heart region in the medical image.

12. The learning device according to claim 7 or 11, wherein the lesion information includes information related to pleural effusion.

13. A learning device described in any one of claims 1 to 12, wherein the organ corresponding to the lesion information is obtained based on the disease name included in the lesion information, or is included in the lesion information.

14. A learning method in which a processor uses a data set of medical images stored in a memory and lesion information included in the medical images to train a learning model with an attention mechanism that estimates a disease from an input medical image, the learning method comprising: The learning model performs a process of outputting an attention map indicating a region of interest in the input medical image using the attention mechanism from the input medical image, and a process of outputting an estimation result of the disease estimated from the input medical image, a step of identifying a position of the region of interest indicated by the attention map in the organ labeling information based on organ labeling information indicating the type and region of the organ shown in the medical image and the attention map, and outputting the identification result; a step of comparing the estimation result with the lesion information to calculate an error; setting the error based on the identification result and an organ corresponding to the lesion information; a step of causing the learning model to perform learning using the set error; Learning methods including.

15. A learning method in which a processor causes a learning model with an attention mechanism to perform learning using a data set of medical images stored in a memory and lesion information included in the medical images, the learning model inferring a disease from an input medical image, the learning method comprising: The learning model performs a process of outputting an attention map indicating a region of interest in the input medical image using the attention mechanism from the input medical image, and a process of outputting an estimation result of the disease estimated from the input medical image, a step of identifying a position of the region of interest indicated by the attention map in the organ labeling information based on organ labeling information indicating the type and region of the organ shown in the medical image and the attention map, and outputting the identification result; a step of imposing constraints on the medical image based on the identification result and the organ corresponding to the lesion information; inputting the constrained medical image into the learning model; a step of comparing an estimation result output by the learning model based on the medical image to which the constraints are applied with the lesion information to calculate an error; training the learning model using the error; Learning methods including.

16. A program that causes the processor to execute each step of the learning method according to claim 14 or 15.

17. A non-transitory computer-readable recording medium having the program according to claim 16 recorded thereon.

18. A program that constructs a trained model trained using the learning method described in claim 14 or 15, and that outputs an estimation result that estimates the presence or absence of a disease and, if said disease is present, the name of the disease when a medical image is input.

Images