Medical image diagnostic support device, medical image diagnostic support method, and program

Abstract

To provide a medical image diagnosis support device, a medical image diagnosis support method, and a program which make it possible for a convolutional neural network not to pay attention to areas unnecessary for disease presence / absence classification in a medical image.SOLUTION: The medical image diagnosis support device for supporting diagnosis of a medical image comprises: a classification model which classifies at least presence / absence of diseases from the medical image; a prediction unit which performs prediction using the classification model; and a learning unit which performs learning with teacher of the classification model. Medical images for learning in which at least presence / absence of diseases is known is used as teacher data in the learning with teacher, and the classification model is constituted of a convolutional neural network and an attention branch network which visualizes areas of interest of the convolutional neural network. In the stage where the learning with teacher is performed, prior information showing areas for classification being areas necessary for classification of presence / absence of diseases in the medical images for learning is given to the attention branch network.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a medical image diagnosis support device, a medical image diagnosis support method, and a program.

Background Art

[0002] Conventionally, an image diagnosis support device for assisting a doctor in diagnosing an image has been known (see, for example, Patent Document 1). In the technique described in Patent Document 1, the diagnostic ability of an endoscopic image possessed by a convolutional neural network (CNN) is used. Specifically, in the technique described in Patent Document 1, a lesion estimation unit uses a convolutional neural network to estimate the lesion name and lesion position of a lesion existing in an endoscopic image represented by endoscopic image data, and the accuracy of the lesion name and lesion position. Further, in the technique described in Patent Document 1, learning processing of a convolutional neural network is performed using reference data (teacher data) marked in advance by an experienced endoscopist. By the way, in the medical field, it is necessary to consider the black box problem of AI (artificial intelligence). That is, it is necessary to consider the black box problem of AI for medical image diagnosis support such as the image diagnosis support device described in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, in their diligent research, the inventors used chest X-ray images as medical images and constructed a classification model to classify the presence or absence of disease from chest X-ray images using a convolutional neural network and an attention branch network (ABN) that visualizes the region of focus of the convolutional neural network, and verified the region of focus of the convolutional neural network. As a result, the inventors found that the convolutional neural network may focus on regions in the chest X-ray image that are unnecessary for classifying the presence or absence of disease. Thus, the inventors believed that medical AI that focuses on areas in chest X-ray images that are unnecessary for classifying the presence or absence of disease would not be able to gain the trust of medical professionals, such as the trust of specialists, and therefore furthered their research. [Means for solving the problem]

[0005] Specifically, the inventors, through diligent research, provided an attention branch network with prior information indicating the regions in chest X-ray images necessary for classifying the presence or absence of disease. As a result, they were able to prevent the convolutional neural network from paying attention to regions in chest X-ray images that are unnecessary for classifying the presence or absence of disease.

[0006] In other words, the present invention aims to provide a medical image diagnostic support device, a medical image diagnostic support method, and a program that enable a convolutional neural network to avoid focusing on areas in medical images that are unnecessary for classifying the presence or absence of disease.

[0007] One aspect of the present invention is a medical image diagnostic support device for assisting in the diagnosis of medical images, comprising: a classification model for classifying at least the presence or absence of disease from the medical image; a prediction unit for making predictions using the classification model; and a learning unit for performing supervised learning of the classification model before the classification model is used by the prediction unit, wherein in the supervised learning performed by the learning unit, training medical images with known presence or absence of disease are used as training data; the classification model is composed of a convolutional neural network and an attention branch network for visualizing the region of interest of the convolutional neural network; and at the stage in which the learning unit performs supervised learning of the classification model, the attention branch network is provided with prior information indicating a classification region, which is a region of the training medical image necessary for classifying the presence or absence of disease.

[0008] In a medical image diagnostic support device according to one aspect of the present invention, the attention branch network comprises a feature extractor that extracts features necessary for classifying the medical image and generates a feature map, an attention branch that generates an attention map by class activation mapping, and a perception branch. During the stage in which the learning unit performs supervised learning of the classification model, the attention map generated by the attention branch is reflected in the feature map generated by the feature extractor, the perception branch outputs the feature map weighted by the attention map as the classification result of the medical image for learning, and the loss function of the attention branch network is the sum of the learning error of the attention branch, the learning error of the perception branch, and a regularization term, the regularization term being the Frobenius norm of the matrix obtained by the Hadamard product of the attention map and the weight map, and the weight map may correspond to the classification region.

[0009] In a medical image diagnostic support device according to one aspect of the present invention, during the stage in which the learning unit performs supervised learning of the classification model, the attention branch network may receive the weight map created by performing convex hull processing on the segmentation image of the first portion which is part of the classification region.

[0010] In a medical image diagnostic support device according to one aspect of the present invention, during the stage in which the learning unit performs supervised learning of the classification model, the attention branch network may receive the weight map created by combining a segmentation image of a first part which is part of the classification region and a segmentation image of a second part which is another part of the classification region.

[0011] In a medical image diagnostic support device according to one aspect of the present invention, the segmentation image of the first portion and / or a combination of the segmentation image of the first portion and the segmentation image of the second portion which is another part of the classification region may be generated by using U-Net.

[0012] In a medical image diagnostic support device according to one aspect of the present invention, any of VGG16, ResNet50, and DenseNet121 may be used as the convolutional neural network.

[0013] In a medical image diagnostic support device according to one aspect of the present invention, the attention branch network comprises a feature extractor that extracts features necessary for classifying the medical image and generates a feature map, an attention branch that generates an attention map by class activation mapping, and a perception branch, wherein the attention map generated by the attention branch is reflected in the feature map generated by the feature extractor, the perception branch outputs the feature map weighted by the attention map as the classification result of the medical image, the loss function of the attention branch network is the sum of the learning error of the attention branch, the learning error of the perception branch, and a regularization term, the regularization term is the Frobenius norm of the matrix obtained by the Hadamard product of the attention map and the weight map, and the weight map may correspond to the prediction region.

[0014] One aspect of the present invention is a medical image diagnostic support method for assisting the diagnosis of medical images, comprising: a prediction step of making a prediction using a classification model that classifies at least the presence or absence of disease from the medical image; and a learning step of performing supervised learning of the classification model before the prediction step is performed, wherein in the supervised learning performed in the learning step, training medical images in which the presence or absence of disease is at least known are used as training data, the classification model is composed of a convolutional neural network and an attention branch network that visualizes the region of interest of the convolutional neural network, and in the learning step, the attention branch network is provided with prior information indicating a classification region which is a region in the training medical image necessary for classifying the presence or absence of disease.

[0015] One aspect of the present invention is a program for causing a computer to execute a prediction step of performing a prediction using a classification model that classifies at least the presence or absence of a disease from a medical image, and a learning step of performing supervised learning of the classification model before the prediction step is executed. In the supervised learning performed in the learning step, learning medical images in which at least the presence or absence of a disease is known are used as teacher data. The classification model is composed of a convolutional neural network and an attention branch network that visualizes an attention area of the convolutional neural network. In the learning step, prior information indicating a classification area, which is an area necessary for classifying the presence or absence of a disease in the learning medical images, is given to the attention branch network.

Advantages of the Invention

[0016] According to the present invention, it is possible to provide a medical image diagnosis support device, a medical image diagnosis support method, and a program that can prevent a convolutional neural network from paying attention to areas unnecessary for classifying the presence or absence of a disease in a medical image.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram showing an example of the configuration of the medical image diagnosis support device of the first embodiment. [Figure 2] It is a diagram showing an example of the configuration of the attention branch network shown in FIG. 1. [Figure 3] It is a diagram for explaining a method of creating a weight map in the first example of the medical image diagnosis support device of the first embodiment. [Figure 4] It is a diagram for explaining a method of creating a weight map in the second example of the medical image diagnosis support device of the first embodiment. [Figure 5] It is a diagram for explaining a calculation process of a regularization term in the medical image diagnosis support device of the first embodiment. [Figure 6]It is a diagram for explaining the attention area etc. of the convolutional neural network of the medical image diagnosis support device according to the first embodiment. [Figure 7] It is a flowchart showing an example of the processing executed in the medical image diagnosis support device according to the first embodiment. [Figure 8] It is a diagram for explaining the attention area etc. of the convolutional neural network of the medical image diagnosis support device according to the second embodiment.

Mode for Carrying Out the Invention

[0018] Hereinafter, embodiments of the medical image diagnosis support device, medical image diagnosis support method, and program of the present invention will be described with reference to the drawings.

[0019] [First Embodiment] FIG. 1 is a diagram showing an example of the configuration of the medical image diagnosis support device 1 according to the first embodiment. In the example shown in FIG. 1, the medical image diagnosis support device 1 according to the first embodiment supports the diagnosis of medical images by, for example, a specialist. The medical images to be diagnosed and supported by the medical image diagnosis support device 1 include, for example, X-ray images such as chest X-ray images, echo images, MRI (Magnetic Resonance Imaging) images, CT (computed tomography) images, fundus images, and the like. The medical image diagnosis support device 1 includes a prediction unit 11, a learning unit 12, and a classification model 1A. The classification model 1A classifies the presence or absence of a disease from a medical image. That is, the classification model 1A classifies a medical image that requires a diagnosis of whether or not there is a disease into a medical image having a disease and a medical image not having a disease. The prediction unit 11 makes a prediction using the classification model 1A. The learning unit 12 performs supervised learning of the classification model 1A before the classification model 1A is used by the prediction unit 11. In the supervised learning performed by the learning unit 12, learning medical images with known presence or absence of a disease are used as teacher data. In the example shown in Figure 1, during the learning phase in which the learning unit 12 performs supervised learning of the classification model 1A, training medical images with known presence or absence of disease are input to the medical image diagnostic support device 1 as training data. In other words, the training data used for supervised learning of the classification model 1A is a pair of training medical images and information indicating whether or not the subject of the training medical image has a disease in that training medical image. In the prediction stage, where the prediction unit 11 predicts the presence or absence of disease from medical images using the classification model 1A, the medical images to be classified by the classification model 1A are input to the medical image diagnostic support device 1, and the classification model 1A classifies the input medical images into medical images with disease and medical images without disease. Furthermore, the classification result by the classification model 1A (that is, information indicating whether each medical image is an image with disease or an image without disease) is output from the medical image diagnostic support device 1.

[0020] As mentioned above, in the medical field, the black box problem of AI must be considered. Specifically, unless medical specialists who use medical AI can understand which areas of medical images the AI ​​is focusing on, there is a risk that they will not trust and use medical AI. In the example shown in Figure 1, the classification model 1A consists of a convolutional neural network 1A1 and an attention branch network 1A2 that visualizes the region of interest of the convolutional neural network 1A1. The attention branch network 1A2 is a classification model that simultaneously achieves visualization of the region of interest of the convolutional neural network 1A1 through visual explanation and improvement of prediction accuracy. For details on the attention branch network 1A2, see, for example, the following literature. H. Fukui, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Attention branch network: Learning of attention mechanism for visual explanation,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2019-June, pp. 10697-10706, 2019, doi: 10.1109 / CVPR.2019.01096.

[0021] Figure 2 shows an example of the configuration of the attention branch network 1A2 shown in Figure 1. In the example shown in Figure 2, the attention branch network 1A2 comprises a feature extractor 1A21, an attention branch 1A22, and a perception branch 1A23. For example, during the stage when the learning unit 12 performs supervised learning of the classification model 1A, the feature extractor 1A21 extracts the features necessary for classifying the training medical images from the training medical images input to the attention branch network 1A2 and generates a feature map. Attention branch 1A22 generates an attention map using class activation mapping (CAM) from the feature map generated by feature extractor 1A21. In attention branch network 1A2, the attention map generated by attention branch 1A22 is reflected in the feature map generated by feature extractor 1A21 (i.e., the feature map is weighted by the attention map). The perception branch 1A23 performs classification of feature maps weighted by the attention map and outputs the classification result of the training medical images input to the attention branch network 1A2.

[0022] In a typical attention branch network, the loss function Loss is expressed by the following equation (1). In equation (1), CrossEntropyLoss att This shows the learning error of the attention branch, and CrossEntropyLoss per This shows the learning error of the perception branch. In other words, in a typical attention branch network, the loss function of the attention branch network, Loss, is equal to the learning error of the attention branch, CrossEntropyLoss. att and the learning error of the perception branch CrossEntropyLoss per It is the sum of the above.

[0023]

number

[0024] Through diligent research, the inventors have discovered that when the loss function of the attention branch network 1A2 is the loss function shown in equation (1), the convolutional neural network 1A1 focuses on regions in the chest X-ray image that are unnecessary for classifying the presence or absence of disease. However, by using the loss function of the attention branch network 1A2 shown in equation (2) below, the convolutional neural network 1A1 stops focusing on regions in the chest X-ray image that are unnecessary for classifying the presence or absence of disease. In detail, the inventors noted in their research that the attention map output from attention branch 1A22 contains information corresponding to the position within the training medical image. Furthermore, the inventors added a regularization term (the third term on the right-hand side of equation (2)) to the loss function Loss of the attention branch network 1A2 so that the convolutional neural network 1A1 would focus on the lung and cardiac regions of the chest X-ray image, and a high penalty would be imposed if the convolutional neural network 1A1 focused on regions other than the lung and cardiac regions of the chest X-ray image. As a result, the convolutional neural network 1A1 no longer focused on regions of the chest X-ray image that were unnecessary for classifying the presence or absence of disease.

[0025]

number

[0026] In other words, the loss function Loss of the attention branch network 1A2 shown in equation (2) is the learning error CrossEntropyLoss of the attention branch 1A22. att And the learning error of perception branch 1A23, CrossEntropyLoss per This is the sum of the regularization term and the regularization term. More specifically, the regularization term is the Frobenius norm of the matrix obtained by the Hadamard product (component-wise product of attention map M and weight map W) of the attention map M output from attention branch 1A22 and the weight map W. In equation (2), the regularization parameter λ is a hyperparameter that needs to be tuned each time training is performed. The weight map W corresponds to the region (classification region) that the convolutional neural network 1A1 should focus on. In other words, the weight map W corresponds to the classification region, which is the region necessary for classifying the presence or absence of disease in the training medical images input to the medical image diagnostic support device 1.

[0027] In other words, in the example shown in Figure 2, the attention branch network 1A2 is provided with prior information indicating the classification region, which is the region necessary for classifying the presence or absence of disease among the learning medical images input to the medical image diagnostic support device 1. As a result, in the examples shown in Figures 1 and 2, the convolutional neural network 1A1 can avoid focusing on regions in the training medical images that are unnecessary for classifying the presence or absence of disease. In other words, the risk that the medical image diagnostic support device 1 may lose the trust of specialists and others due to the convolutional neural network 1A1 focusing on regions in the training medical images that are unnecessary for classifying the presence or absence of disease can be suppressed. In the examples shown in Figures 1 and 2, classification model 1A does not perform prognosis classification (i.e., classifying medical images into those with a good prognosis and those with a poor prognosis), but in other examples, classification model 1A may perform prognosis classification. In these examples, training data is used, which consists of training medical images in which the presence or absence of disease is known and information about the prognosis is known.

[0028] Figure 3 is a diagram illustrating the method for creating a weight map W in a first example of the medical image diagnostic support device 1 of the first embodiment. Specifically, Figure 3(A) shows an example of a training medical image (input image) input to the medical image diagnostic support device 1, Figure 3(B) shows a segmentation image of a part of the region (lung field) necessary for classifying the presence or absence of disease (classification region) from the training medical image (chest X-ray image) shown in Figure 3(A), and Figure 3(C) shows the weight map W created from the segmentation image shown in Figure 3(B). In the example shown in Figure 3, a weight map W shown in Figure 3(C) is created by performing a convex hull operation on the segmentation image (see Figure 3(B)) of the first part (specifically, the lung field), which is part of the classification region (specifically, the region in the chest X-ray image shown in Figure 3(A) that is necessary for classifying the presence or absence of disease). The weight map W shown in Figure 3(C) is created, for example, by the administrator of the medical image diagnostic support device 1 (e.g., a specialist). During the stage in which the classification model 1A is supervised learning is performed by the learning unit 12, the attention branch network 1A2 of the medical image diagnostic support device 1 receives the input of the weight map W and uses the weight map W to obtain the regularization term described above (that is, the weight map W is input to the attention branch network 1A2 and used to obtain the regularization term). In another example, the medical image diagnostic support device 1 may have a function for creating the weight map W shown in Figure 3(C).

[0029] Figure 4 is a diagram illustrating the method for creating a weight map W in a second example of the medical image diagnostic support device 1 of the first embodiment. Specifically, Figure 4(A) shows an example of a learning medical image (input image) input to the medical image diagnostic support device 1, Figure 4(B) shows a combination of a segmentation image of a part (lung field) and another part (heart) of the region necessary for classifying the presence or absence of disease (classification region) from the learning medical image (chest X-ray image) shown in Figure 4(A), and Figure 4(C) shows a weight map W created from a combination of the lung field segmentation image and the heart segmentation image shown in Figure 4(B). In the example shown in Figure 4, the weight map W shown in Figure 4(C) is created by combining a segmentation image of the first part (specifically, the lung field), which is part of the classification region (specifically, the region of the chest X-ray image shown in Figure 4(A) that is necessary for classifying the presence or absence of disease), with a segmentation image of the second part (specifically, the heart), which is another part of the classification region (see Figure 4(B)). During the supervised learning stage of the classification model 1A by the learning unit 12, the attention branch network 1A2 of the medical image diagnostic support device 1 receives the weight map W as input and uses it to obtain the regularization term described above (that is, the weight map W is input to the attention branch network 1A2 and used to obtain the regularization term). In another example, the medical image diagnostic support device 1 may have a function for creating the weight map W shown in Figure 4(C).

[0030] In detail, the inventors generated the segmentation image shown in Figure 3(B) (segmentation image of the lung field) and the segmentation image shown in Figure 4(B) (a combination of the segmentation image of the lung field and the segmentation image of the heart) by using U-Net in their research. U-Net is described in the following document. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9351, pp. 234-241, 2015, doi: 10.1007 / 978-3-319-24574-4_28.

[0031] Furthermore, in their research, the inventors used 704 chest X-ray images from the Montgomery Country-Chest X-ray database as training data for the lung fields. The 704 chest X-ray images from the Montgomery Country-Chest X-ray database are described in the following two documents. S. Candemir et al., “Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration,” IEEE Trans. Med. Imaging, vol. 33, no. 2, pp. 577-590, 2014, doi: 10.1109 / TMI.2013.2290491. S. Jaeger et al., “Automatic Tuberculosis Screening Using Chest Radiographs,” IEEE Trans. Med. Imaging, vol. 33, no. 2, pp. 233-245, Feb. 2014, doi: 10.1109 / TMI.2013.2284099.

[0032] Furthermore, in their research, the inventors used 247 chest X-ray images from JSRT as training data for the heart. The 247 chest X-ray images from JSRT are described in the following two documents. P. (Project leader) TU of C. Junji Shiraishi et al., “Standard Digital Image Database:Chest Lung Nodules and Non-Nodules Created by the Japanese Society of Radiological Technology (JSRT) In cooperation with the Japanese Radiological Society (JRS),” 1997. B. van Ginneken, MB Stegmann, and M. Loog, “Segmentation of anatomical structures in chest radiographs using supervised methods: A comparative study on a public database,” Med. Image Anal., vol. 10, no. 1, pp. 19-40, 2006, doi: 10.1016 / j.media.2005.02.002.

[0033] Figure 5 is a diagram illustrating the calculation process of the regularization term in the medical image diagnostic support device 1 of the first embodiment. In the medical image diagnostic support device 1 of the first embodiment, as shown in equation (2) and Figure 5 above, the regularization term is calculated as the Frobenius norm of the matrix obtained by the Hadamard product (component-wise product of attention map M and weight map W) of attention map M and weight map W.

[0034] <Data used> In a study verifying the medical image diagnostic support device 1 of the first embodiment, the inventors used 3032 chest X-ray images taken at Teikyo University Hospital. The data consisted of 2002 normal cases (i.e., the subject of the chest X-ray image did not have any disease in the chest X-ray image) and 1030 abnormal cases (i.e., the subject of the chest X-ray image had a disease in the chest X-ray image). The abnormal cases included images taken in a sitting or lying position. The chest X-ray images were resized to 224 x 224 pixels to match the input size recommended by the classification model.

[0035] <Classification performance> In a study verifying the medical image diagnostic support device 1 of the first embodiment, the inventors evaluated the learning and accuracy of the Teikyo University chest X-ray dataset using stratified sampling 10-fold cross-validation. The learning and accuracy of the medical image diagnostic support device 1 of the first embodiment were evaluated by constructing an attention branch network based on three typical CNNs (VGG16, ResNet50, DenseNet121) as classification models. The mean and standard deviation of Accuracy (Balanced Accuracy), Sensitivity, Specificity, and AUC (Area Under the Curve) are shown in Table 1 below. In Table 1, "Weightmap1" shows a weight map W (see Figure 3(C)) obtained by applying convex hull processing to lung field segmentation images, and "weightmap2" shows a weight map W (see Figure 4(C)) obtained by combining lung field segmentation images and cardiac segmentation images.

[0036] [Table 1]

[0037] In detail, the row labeled "VGG16" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which VGG16 is used as the CNN and ABN is not applied. The row labeled "VGG16+ABN" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which VGG16 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row labeled "VGG16+ABN+weightmap1" in Table 1 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which VGG16 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "VGG16+ABN+weightmap2" in Table 1 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which VGG16 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2. The row labeled "Resnet50" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which ResNet50 is used as the CNN and ABN is not applied. The row labeled "Resnet50+ABN" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which ResNet50 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row labeled "Resnet50+ABN+weightmap1" in Table 1 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which ResNet50 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "Resnet50+ABN+weightmap2" in Table 1 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which ResNet50 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2. The row for "Densenet121" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which DenseNet121 is used as the CNN and ABN is not applied. The row for "Densenet121+ABN" in Table 1 shows the evaluation results of a comparative example of a medical image diagnostic support device in which DenseNet121 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row for "Densenet121+ABN+weightmap1" in Table 1 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which DenseNet121 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "Densenet121+ABN+weightmap2" in Table 1 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which DenseNet121 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2.

[0038] Balanced Accuracy was used as the evaluation metric to appropriately assess the bias in the number of data points between classes. Balanced Accuracy is expressed by the following equation (3). In equation (3), TP is True Positive, TN is True Negative, FP is False Positive, and FN is False Negative.

[0039]

number

[0040] In all of the examples shown in the row for "VGG16+ABN+weightmap1" in Table 1, the row for "VGG16+ABN+weightmap2" in Table 1, the row for "Resnet50+ABN+weightmap1" in Table 1, the row for "Resnet50+ABN+weightmap2" in Table 1, the row for "Densenet121+ABN+weightmap1" in Table 1, and the row for "Densenet121+ABN+weightmap2" in Table 1, no performance degradation was observed as a result of adding the regularization term to the medical image diagnostic support device 1 of the first embodiment.

[0041] Furthermore, in a study to evaluate the learning and accuracy of the medical image diagnostic support device 1 of the first embodiment, the inventors used the large-scale chest X-ray dataset NIH14 published by the National Institute of Health. Similar to the case in Table 1, the learning and accuracy of the medical image diagnostic support device 1 of the first embodiment were evaluated by constructing an attention branch network based on three typical CNNs (VGG16, ResNet50, DenseNet121) as classification models. The mean and standard deviation of Accuracy, Sensitivity, Specificity, and AUC are shown in Table 2 below. In Table 2, "Weightmap1" shows a weight map W (see Figure 3(C)) obtained by applying convex hull processing to lung field segmentation images, and "weightmap2" shows a weight map W (see Figure 4(C)) obtained by combining lung field segmentation images and cardiac segmentation images. In NIH14, each image is assigned a [0,1] to indicate the presence or absence of one of 14 diseases (Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion, Emphysema, Fibrosis, Hernia, Infiltration, Mass, Nodule, Pleural Thickening, Pneumonia, Pneumothorax). The results shown in Table 2 are the mean values ​​for the 14 diseases.

[0042] [Table 2]

[0043] In detail, the row labeled "VGG16" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which VGG16 is used as the CNN and ABN is not applied. The row labeled "VGG16+ABN" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which VGG16 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row labeled "VGG16+ABN+weightmap1" in Table 2 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which VGG16 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "VGG16+ABN+weightmap2" in Table 2 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which VGG16 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2. The row labeled "Resnet50" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which ResNet50 is used as the CNN and ABN is not applied. The row labeled "Resnet50+ABN" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which ResNet50 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row labeled "Resnet50+ABN+weightmap1" in Table 2 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which ResNet50 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "Resnet50+ABN+weightmap2" in Table 2 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which ResNet50 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2. The row for "Densenet121" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which DenseNet121 is used as the CNN and ABN is not applied. The row for "Densenet121+ABN" in Table 2 shows the evaluation results of a comparative example of a medical image diagnostic support device in which DenseNet121 is used as the CNN and ABN is applied, but prior information (weight map W) is not provided to the ABN. The row for "Densenet121+ABN+weightmap1" in Table 2 shows the evaluation results of a first example of the medical image diagnostic support device 1 of the first embodiment in which DenseNet121 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 3(C) is provided to the attention branch network 1A2. The row for "Densenet121+ABN+weightmap2" in Table 2 shows the evaluation results of a second example of the medical image diagnostic support device 1 of the first embodiment, in which DenseNet121 is used as the convolutional neural network 1A1 and the weight map W (prior information) shown in Figure 4(C) is provided to the attention branch network 1A2.

[0044] In all of the examples shown in the row for "VGG16+ABN+weightmap1" in Table 2, the row for "VGG16+ABN+weightmap2" in Table 2, the row for "Resnet50+ABN+weightmap1" in Table 2, the row for "Resnet50+ABN+weightmap2" in Table 2, the row for "Densenet121+ABN+weightmap1" in Table 2, and the row for "Densenet121+ABN+weightmap2" in Table 2, no performance degradation was observed when the regularization term was added to the medical image diagnostic support device 1 of the first embodiment.

[0045] <Visualization of the basis for judgment> Figure 6 is a diagram illustrating the region of interest of the convolutional neural network 1A1 of the medical image diagnostic support device 1 of the first embodiment. Specifically, Figure 6(A) shows the medical image (input image) input to the medical image diagnostic support device 1 of the first embodiment and a comparative example medical image diagnostic support device in which no prior information (weight map) is provided to the ABN; Figure 6(B) shows the region of interest of the CNN in the comparative example medical image diagnostic support device in which no prior information (weight map) is provided to the ABN; and Figure 6(C) shows the region of interest of the convolutional neural network 1A1 of the medical image diagnostic support device 1 of the first embodiment, in which the attention branch network 1A2 is provided with prior information (weight map W (specifically, a weight map W combining segmentation images of the lung field and segmentation images of the heart)). In the example shown in Figure 6, the Teikyo University chest X-ray dataset was used as the training medical image (input image) (see Figure 6(A)) input to the medical image diagnostic support device 1 of the first embodiment and the comparative example medical image diagnostic support device in which no prior information (weight map) was provided to the ABN.

[0046] In the comparative example of a medical image diagnostic support system where the ABN was not given prior information (weight map), the ABN's attention map, as shown in Figure 6(B), indicated that the CNN was paying attention to the entire input image. More specifically, in the uppermost example in Figure 6(B), the ABN's attention map showed that the CNN was paying more attention to the lower portion (the portion not containing the lungs and heart) of the input image than to the upper portion (the portion containing the lungs and heart). In the second-to-last example in Figure 6(B), the ABN's attention map showed that the CNN was paying attention to the less important portion (the portion not containing the lungs and heart) of the input image as much as the important portion (the portion containing the lungs and heart). In the lowermost example in Figure 6(B), the ABN's attention map showed that the CNN was not paying much attention to the portion containing the heart of the input image. In contrast, in the medical image diagnostic support device 1 of the first embodiment, in which the attention branch network 1A2 is given prior information (weight map W), as shown in Figure 6(C), the attention map of the attention branch network 1A2 indicates that the convolutional neural network 1A1 is focusing on the lung field within the input image.

[0047] Figure 7 is a flowchart showing an example of a process performed in the medical image diagnostic support device 1 of the first embodiment. In the example shown in Figure 7, in step S1, the learning unit 12 performs supervised learning of a classification model 1A, which consists of a convolutional neural network 1A1 and an attention branch network 1A2 that visualizes the region of interest of the convolutional neural network 1A1. In the supervised learning performed in step S1, training medical images with known presence or absence of disease are used as training data. In step S1, information indicating the classification region (weight map W), which is the region of the training medical image necessary for classifying the presence or absence of disease, is provided to the attention branch network 1A2. Next, in step S2, the prediction unit 11 uses the trained classification model 1A to classify the presence or absence of disease from the medical image input to the medical image diagnostic support device 1.

[0048] In the example shown in Figure 7, prior information indicating the weight map W is provided to the attention branch network 1A2 in step S1. However, in other examples, prior information indicating the weight map W may be provided to the attention branch network 1A2 not only in step S1 but also in step S2.

[0049] [Second Embodiment] A second embodiment of the medical image diagnostic support device, medical image diagnostic support method, and program of the present invention will be described below. The medical image diagnostic support device 1 of the second embodiment is configured in the same way as the medical image diagnostic support device 1 of the first embodiment described above, except for the points described later. Therefore, the medical image diagnostic support device 1 of the second embodiment can achieve the same effects as the medical image diagnostic support device 1 of the first embodiment described above, except for the points described later.

[0050] The medical image diagnostic support device 1 of the second embodiment is configured similarly to the medical image diagnostic support device 1 of the first embodiment shown in Figure 1. In other words, the medical image diagnostic support device 1 of the second embodiment comprises a prediction unit 11, a learning unit 12, and a classification model 1A. The classification model 1A consists of a convolutional neural network 1A1 and an attention branch network 1A2 that visualizes the region of interest of the convolutional neural network 1A1.

[0051] The attention branch network 1A2 of the medical image diagnostic support device 1 in the second embodiment is configured in the same way as the attention branch network 1A2 of the medical image diagnostic support device 1 in the first embodiment shown in Figure 2. In other words, the attention branch network 1A2 of the medical image diagnostic support device 1 of the second embodiment includes a feature extractor 1A21 that extracts features necessary for classifying training medical images and generates a feature map during the stage when supervised learning of the classification model 1A is performed by the learning unit 12, for example, an attention branch 1A22, and a perception branch 1A23. In detail, in the medical image diagnostic support device 1 of the second embodiment, the output from perception branch 1A23 is visualized by applying Grad-CAM to the output from perception branch 1A23.

[0052] In the medical image diagnostic support device 1 of the second embodiment, similar to the medical image diagnostic support device 1 of the first embodiment, the loss function Loss of the attention branch network 1A2 is the learning error CrossEntropyLoss of the attention branch 1A22. att And the learning error of perception branch 1A23, CrossEntropyLossper This is the sum of the terms including the regularization term. In the medical image diagnostic support device 1 of the second embodiment, similar to the medical image diagnostic support device 1 of the first embodiment, the regularization term is the Frobenius norm of the matrix obtained by the Hadamard product of the attention map M and the weight map W output from the attention branch 1A22. In the medical image diagnostic support device 1 of the second embodiment, similar to the medical image diagnostic support device 1 of the first embodiment, the weight map W corresponds to the region (classification region) that the convolutional neural network 1A1 should focus on. In other words, the weight map W corresponds to the classification region, which is the region in the training medical image input to the medical image diagnostic support device 1 that is necessary for classifying whether or not there is a disease.

[0053] Figure 8 is a diagram (Grad-CAM) illustrating the region of interest of the convolutional neural network 1A1 of the medical image diagnostic support device 1 of the second embodiment. Specifically, Figure 8(A) shows the medical image (input image) input to the medical image diagnostic support device 1 of the second embodiment, the comparative example medical image diagnostic support device without ABN, and the comparative example medical image diagnostic support device with ABN applied but without prior information (weight map) provided to the ABN; Figure 8(B) shows the region of interest of the CNN in the comparative example medical image diagnostic support device without ABN; Figure 8(C) shows the region of interest of the CNN in the comparative example medical image diagnostic support device with ABN applied but without prior information (weight map) provided to the ABN; and Figure 8(D) shows the region of interest of the convolutional neural network 1A1 of the medical image diagnostic support device 1 of the second embodiment, where prior information (weight map W) is provided to the attention branch network 1A2. In the example shown in Figure 8, the Teikyo University chest X-ray dataset was used as the medical image (input image) (see Figure 8(A)) input to the medical image diagnostic support device 1 of the second embodiment, the comparative example medical image diagnostic support device in which ABN is not applied, and the comparative example medical image diagnostic support device in which ABN is applied but prior information (weight map) is not provided to ABN.

[0054] In the medical image diagnostic support device 1 of the second embodiment, where the attention branch network 1A2 is given prior information (weight map W) during the supervised learning of the classification model 1A by the learning unit 12, as shown in Figure 8(D) (similar to the medical image diagnostic support device 1 of the first embodiment shown in Figure 6(C)), the attention map of the attention branch network 1A2 indicates that the convolutional neural network 1A1 is focusing on the lung field within the input image.

[0055] <Examples of application> In the examples described above, the medical image diagnostic support device 1 of the first and second embodiments is applied to chest X-ray images, but in other examples, the medical image diagnostic support device 1 of the first or second embodiment may be applied to X-ray images other than chest X-ray images. In other examples, the medical image diagnostic support device 1 of the first or second embodiment may be applied to echocardiograms. Specifically, by having the convolutional neural network 1A1 focus on the wall of the heart, the classification model 1A can classify whether or not myocardial infarction is present. Alternatively, by having the convolutional neural network 1A1 focus on the mammary gland, the classification model 1A can classify whether or not breast cancer is present. In further examples, the medical image diagnostic support device 1 of the first or second embodiment may be applied to MRI images. Specifically, by having the convolutional neural network 1A1 focus only on intracranial regions, the classification model 1A can classify the presence or absence of cerebral infarction. Alternatively, by having the convolutional neural network 1A1 focus only on prostate regions, the classification model 1A can classify the presence or absence of prostate cancer. In further examples, the medical image diagnostic support device 1 of the first or second embodiment may be applied to CT images. Specifically, by having the convolutional neural network 1A1 focus only on the lung field, the classification model 1A can classify the presence or absence of pneumonia, nodules, tumors, etc. Also, by having the convolutional neural network 1A1 focus only on the intracranial region, the classification model 1A can classify the presence or absence of cerebral hemorrhage. In further examples, the medical image diagnostic support device 1 of the first or second embodiment may be applied to fundus images. Specifically, by having the convolutional neural network 1A1 focus on the optic nerve head, the classification model 1A can classify whether or not glaucoma is present.

[0056] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and modifications can be made as appropriate without departing from the spirit of the present invention. The configurations described in each of the embodiments and examples above may be combined.

[0057] In addition, all or part of the medical image diagnostic support device 1 in the above embodiment may be implemented by dedicated hardware, or it may be implemented by memory and a microprocessor. Furthermore, the medical image diagnostic support device 1 may consist of all or part of memory and a CPU (central processing unit), and its functions may be realized by loading and executing programs into memory to realize the functions of each part of the system. Furthermore, a program to implement all or part of the functions of the medical image diagnostic support device 1 may be recorded on a computer-readable recording medium, and the processing of each part may be performed by loading the program recorded on this recording medium into a computer system and executing it. The term "computer system" here includes hardware such as the operating system and peripheral devices. Also, if a WWW system is used, the "computer system" includes the homepage provisioning environment (or display environment). Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. In addition, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system. [Explanation of symbols]

[0058] 1…Medical image diagnostic support device, 11…Prediction unit, 12…Learning unit, 1A…Classification model, 1A1…Convolutional neural network, 1A2…Attention branch network, 1A21…Feature extractor, 1A22…Attention branch, 1A23…Perception branch

Claims

1. A medical image diagnostic support device that assists in the diagnosis of medical images, A classification model that classifies the presence or absence of disease from the aforementioned medical images, A prediction unit that performs predictions using the aforementioned classification model, The system comprises a learning unit that performs supervised learning of the classification model before the classification model is used by the prediction unit, In the supervised learning performed by the aforementioned learning unit, training medical images with at least known presence or absence of disease are used as training data. The classification model is composed of a convolutional neural network and an attention branch network that visualizes the region of interest of the convolutional neural network. During the stage in which the learning unit performs supervised learning of the classification model, the attention branch network is provided with prior information in the form of a weight map that indicates classification regions, which are regions in the medical images used for learning that are necessary for classifying whether or not a disease is present. Medical image diagnostic support device.

2. The aforementioned attention branch network is A feature extractor that extracts the features necessary for classifying the aforementioned medical images and generates a feature map, An attention branch that generates an attention map using class activation mapping, With a Perception Branch, During the stage in which the learning unit performs supervised learning of the classification model, The attention map generated by the attention branch is reflected in the feature map generated by the feature extractor. The perception branch outputs the feature map, weighted by the attention map, as the classification result of the training medical image. The loss function of the aforementioned attention branch network is: The learning error of the aforementioned attention branch and The learning error of the aforementioned perception branch and It is the sum with the regularization term, The regularization term is the Frobenius norm of the matrix obtained by the Hadamard product of the attention map and the weight map. The medical image diagnostic support device according to claim 1.

3. During the stage in which the learning unit performs supervised learning of the classification model, The aforementioned attention branch network is The system accepts the weight map created by performing a convex hull operation on the segmentation image of the first portion, which is part of the classification region. The medical image diagnostic support device according to claim 2.

4. During the stage in which the learning unit performs supervised learning of the classification model, The aforementioned attention branch network is The system accepts the weight map created by combining a segmentation image of a first portion, which is part of the classification region, and a segmentation image of a second portion, which is another part of the classification region. The medical image diagnostic support device according to claim 2.

5. The segmentation image of the first portion and / or the combination of the segmentation image of the first portion and the segmentation image of the second portion, which is another part of the classification region, are generated by using U-Net. A medical image diagnostic support device according to claim 3 or claim 4.

6. As the aforementioned convolutional neural network, one of VGG16, ResNet50, and DenseNet121 is used. The medical image diagnostic support device according to claim 1.

7. By applying Grad-CAM to the output from the perception branch, the output from the perception branch is visualized. The medical image diagnostic support device according to claim 2.

8. A medical image diagnostic support method that assists in the diagnosis of medical images, A prediction step involves performing a prediction using a classification model that classifies at least the presence or absence of disease from the aforementioned medical images, The system includes a learning step in which supervised learning of the classification model is performed before the prediction step is executed, In the supervised learning performed in the aforementioned learning step, training medical images with at least known presence or absence of disease are used as training data. The classification model is composed of a convolutional neural network and an attention branch network that visualizes the region of interest of the convolutional neural network. In the learning step, the attention branch network is given prior information in the form of a weight map that indicates classification regions, which are regions in the medical images used for learning that are necessary for classifying whether or not a disease is present. Medical image diagnostic support methods.

9. On the computer, A prediction step that uses a classification model to make predictions based on medical images, which at least classifies the presence or absence of disease, A program for causing a learning step to perform supervised learning of the classification model before the prediction step is performed, In the supervised learning performed in the aforementioned learning step, training medical images with at least known presence or absence of disease are used as training data. The classification model is composed of a convolutional neural network and an attention branch network that visualizes the region of interest of the convolutional neural network. In the learning step, the attention branch network is given prior information in the form of a weight map that indicates classification regions, which are regions in the medical images used for learning that are necessary for classifying whether or not a disease is present. program.

Images