Diagnostic support device, diagnostic support method, and program

Abstract

To provide a diagnosis support device, a diagnosis support method and a program which can realize highly accurate classification of a chest X-ray image due to the presence / absence of the pulmonary hypertension while improving the convergence property of learning of a classification model that classifies the chest X-ray image depending on the presence / absence of the pulmonary hypertension.SOLUTION: A diagnosis support device for supporting diagnosis of the presence / absence of the pulmonary hypertension based on a chest X-ray image comprises: a classification model which classifies the chest X-ray image depending on the presence / absence of the pulmonary hypertension; a prediction unit which performs prediction using the classification model; and a learning unit which performs supervised learning of the classification model before the classification model is used by the prediction unit. In the supervised learning performed by the learning unit, a chest X-ray image for learning in which the presence / absence of the pulmonary hypertension is already known is used as teacher data. The classification model includes: a capsule network which can learn a positional relation of feature amounts; and a skip connection which suppresses the gradient disappearance due to deepening of a layer.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , , ,

[0005] , , , , , ,

[0001] The present invention relates to a diagnostic support device, a diagnostic support method, and a program.

Background Art

[0002] Conventionally, there has been known an image processing device that performs image analysis of a chest X-ray image and diagnoses whether the chest X-ray image corresponds to any of a plurality of types of lesion patterns (see, for example, Patent Document 1). In the technique described in Patent Document 1, image analysis of a chest X-ray image is performed using a learned discriminator. Patent Document 1 describes that a convolutional neural network (CNN) or the like is used as the discriminator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in image recognition using CNN, it has been pointed out that in the pooling process, the mutual positional relationship of a plurality of features in the image is lost, and thus in image recognition using CNN, it is necessary to learn an enormous number of images.

Means for Solving the Problems

[0005] Therefore, the present inventors diligently studied and applied a capsule network to a classification model that classifies chest X-ray images according to the presence or absence of pulmonary hypertension, and verified the classification model. As a result, the present inventors confirmed that when a capsule network is applied to the classification model, it takes a longer time to learn the classification model than when CNN is applied to the classification model. The inventors believed it was necessary to improve the convergence of learning in classification models and therefore furthered their research. Specifically, the inventors have diligently researched and applied a capsule network and skip connections, which suppress gradient vanishing due to increasing layer depth, to a classification model that classifies chest X-ray images based on the presence or absence of pulmonary hypertension. As a result, they were able to improve the convergence of the classification model's learning, and by using the trained classification model, they were able to achieve highly accurate classification of chest X-ray images based on the presence or absence of pulmonary hypertension.

[0006] In other words, the present invention aims to provide a diagnostic support device, a diagnostic support method, and a program that can improve the convergence of learning of a classification model that classifies chest X-ray images based on the presence or absence of pulmonary hypertension, while achieving highly accurate classification of chest X-ray images based on the presence or absence of pulmonary hypertension.

[0007] One aspect of the present invention is a diagnostic support device for assisting in the diagnosis of the presence or absence of pulmonary hypertension based on chest X-ray images, comprising: a classification model for classifying the chest X-ray images according to the presence or absence of pulmonary hypertension; a prediction unit for making predictions using the classification model; and a learning unit for performing supervised learning of the classification model before the classification model is used by the prediction unit, wherein in the supervised learning performed by the learning unit, training chest X-ray images with known presence or absence of pulmonary hypertension are used as training data, and the classification model comprises a capsule network capable of learning the positional relationships of feature quantities and skip connections for suppressing gradient vanishing due to increasing layer depth. The classification model comprises a plurality of residual blocks that suppress gradient vanishing due to increasing layer depth, each residual block comprising: a first convolutional layer; a first batch normalization unit connected to the output side of the first convolutional layer; a first activation function connected to the output side of the first batch normalization unit; a second convolutional layer connected to the output side of the first activation function; a second batch normalization unit connected to the output side of the second convolutional layer; a second activation function connected to the output side of the second batch normalization unit; the input side of the first convolutional layer; and the skip connection connecting the output side of the second batch normalization unit and the input side of the second activation function. It is a diagnostic support device.

[0009] In one aspect of the present invention, the diagnostic support device may include, as the plurality of residual blocks, a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, and a sixth residual block.

[0010] In one embodiment of the present invention, the output side of an activation function is connected to the input side of each residual block, the output side of a batch normalization unit is connected to the input side of the activation function connected to the input side of each residual block, and the output side of a convolutional layer may be connected to the input side of the batch normalization unit connected to the input side of each residual block via an activation function.

[0011] One aspect of the present invention is a diagnostic support method that assists in the diagnosis of the presence or absence of pulmonary hypertension based on chest X-ray images, Supervised learning was conducted. A classification model that classifies the aforementioned chest X-ray images based on the presence or absence of pulmonary hypertension. When the chest X-ray image is input, the classification model outputs information indicating whether or not the patient has pulmonary hypertension as a classification result. A prediction step and a learning step in which supervised learning of the classification model is performed before the classification model is used in the prediction step. Yes In the supervised learning performed in the learning step, training chest X-ray images with and without pulmonary hypertension are used as training data, and the classification model comprises a capsule network capable of learning the positional relationships of features and skip connections that suppress gradient vanishing due to the depth of the layers. The classification model comprises a plurality of residual blocks that suppress gradient vanishing due to increasing layer depth, each residual block comprising: a first convolutional layer; a first batch normalization unit connected to the output side of the first convolutional layer; a first activation function connected to the output side of the first batch normalization unit; a second convolutional layer connected to the output side of the first activation function; a second batch normalization unit connected to the output side of the second convolutional layer; a second activation function connected to the output side of the second batch normalization unit; the input side of the first convolutional layer; and the skip connection connecting the output side of the second batch normalization unit and the input side of the second activation function. This is a diagnostic support method.

[0012] One aspect of the present invention is a program for causing a computer to perform a prediction step of classifying chest X-ray images according to the presence or absence of pulmonary hypertension using a classification model, and a learning step of performing supervised learning of the classification model before the classification model is used in the prediction step, wherein in the supervised learning performed in the learning step, training chest X-ray images with known presence or absence of pulmonary hypertension are used as training data, and the classification model comprises a capsule network capable of learning the positional relationships of feature quantities and skip connections that suppress gradient vanishing due to the depth of layers. The classification model comprises a plurality of residual blocks that suppress gradient vanishing due to increasing layer depth, each residual block comprising: a first convolutional layer; a first batch normalization unit connected to the output side of the first convolutional layer; a first activation function connected to the output side of the first batch normalization unit; a second convolutional layer connected to the output side of the first activation function; a second batch normalization unit connected to the output side of the second convolutional layer; a second activation function connected to the output side of the second batch normalization unit; the input side of the first convolutional layer; and the skip connection connecting the output side of the second batch normalization unit and the input side of the second activation function. It is a program. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide a diagnostic support device, a diagnostic support method, and a program that can realize highly accurate classification of chest X-ray images based on the presence or absence of pulmonary hypertension while improving the convergence of learning of a classification model that classifies chest X-ray images based on the presence or absence of pulmonary hypertension.

Brief Description of Drawings

[0014] [Figure 1] It is a diagram showing an example of the configuration of the diagnostic support device of the first embodiment. [Figure 2] It is a diagram showing an example of the configuration of the classification model shown in FIG. 1. [Figure 3] It is a diagram showing an example of a capsule network to which skip connections are not applied, which is a classification model of a comparative example. [Figure 4] It is a flowchart showing an example of the processing executed in the diagnostic support device of the first embodiment. [Figure 5] It is a diagram for explaining the classification model (neural network) of the first embodiment. [Figure 6] It is a diagram showing an evaluation set, a conventional method for measuring anatomical structures, and a ROC curve of an expert doctor in the first embodiment. [Figure 7] It is a diagram showing the visualization of the attention area of AI (classification model) in chest X-ray images. [Figure 8] It is a diagram for explaining the accuracy, AUC, etc. of the classification model of the second embodiment when performing 10-fold cross-validation.

Modes for Carrying Out the Invention

[0015] Hereinafter, embodiments of the diagnostic support device, diagnostic support method, and program of the present invention will be described with reference to the drawings.

[0016] [First Embodiment] FIG. 1 is a diagram showing an example of the configuration of the diagnostic support device 1 of the first embodiment. In the example shown in FIG. 1, the diagnostic support device 1 of the first embodiment supports the diagnosis of the presence or absence of pulmonary hypertension based on a chest X-ray image by, for example, a specialist. The diagnostic support device 1 includes a classification model 1A, a prediction unit 11, and a learning unit 12. The classification model 1A classifies chest X-ray images according to the presence or absence of pulmonary hypertension. The prediction unit 11 makes a prediction using the classification model 1A. The learning unit 12 performs supervised learning of the classification model 1A before the classification model 1A is used by the prediction unit 11. In the supervised learning performed by the learning unit 12, learning chest X-ray images whose presence or absence of pulmonary hypertension is known are used as teacher data. In the example shown in FIG. 1, in the learning stage where the learning unit 12 performs supervised learning of the classification model 1A, learning chest X-ray images whose presence or absence of pulmonary hypertension is known are input into the diagnostic support device 1 as teacher data. That is, the teacher data used for the supervised learning of the classification model 1A is a pair of a learning chest X-ray image and information indicating whether the imaging subject of the learning chest X-ray image has pulmonary hypertension. In the prediction stage by the prediction unit 11, a chest X-ray image to be predicted is input into the diagnostic support device 1, and the prediction unit 11 predicts whether the imaging subject of the input chest X-ray image has pulmonary hypertension based on the input chest X-ray image by using the classification model 1A. Further, the prediction result by the prediction unit 11 (that is, information indicating whether the imaging subject of the input chest X-ray image has pulmonary hypertension) is output from the diagnostic support device 1.

[0017] FIG. 2 is a diagram showing an example of the configuration of the classification model 1A shown in FIG. 1. Specifically, FIG. 2(A) shows an example of the whole of the classification model 1A, and FIG. 2(B) shows an example of the residual block 1A24 shown in FIG. 2(A). FIG. 3 is a diagram showing an example of a capsule network 1AN of a comparative example where skip connections are not applied, which is a classification model. The capsule network 1AN can learn the positional relationship of feature amounts. In the example shown in Figure 2, the classification model 1A comprises an input layer 1A1, a convolutional layer 1A21, a batch normalization unit 1A22, an activation function 1A23, and a residual block 1A24. The output side of the input layer 1A1 is connected to the input side of the convolutional layer 1A21. The output side of the convolutional layer 1A21 is connected to the input side of the batch normalization unit 1A22. The output side of the batch normalization unit 1A22 is connected to the input side of the activation function 1A23. The output side of the activation function 1A23 is connected to the input side of the residual block 1A24. The residual block 1A24 suppresses gradient vanishing due to increasing layer depth. Specifically, the residual block 1A24 comprises a first convolutional layer 1AA, a first batch normalization unit 1AB, a first activation function 1AC, a second convolutional layer 1AD, a second batch normalization unit 1AE, a second activation function 1AF, and a skip connection 1AG.

[0018] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, comprises an input layer 1A1, a convolutional layer 1A21, and an activation function 1A23. The output side of the input layer 1A1 is connected to the input side of the convolutional layer 1A21. The output side of the convolutional layer 1A21 is connected to the input side of the activation function 1A23.

[0019] In the example shown in Figure 2, the input side of the first convolutional layer 1AA is connected to the output side of the activation function 1A23. The output side of the first convolutional layer 1AA is connected to the input side of the first batch normalization unit 1AB. The output side of the first batch normalization unit 1AB is connected to the input side of the first activation function 1AC. The output side of the first activation function 1AC is connected to the input side of the second convolutional layer 1AD. The output side of the second convolutional layer 1AD is connected to the input side of the second batch normalization unit 1AE. The output side of the second batch normalization unit 1AE is connected to the input side of the second activation function 1AF. The skip connection 1AG connects the input side of the first convolutional layer 1AA to the output side of the second batch normalization unit 1AE and the input side of the second activation function 1AF. In other words, the output of the activation function 1A23 is branched and input to the first convolutional layer 1AA and the skip connection 1AG. Also, the output of the second batch normalization unit 1AE and the output of the skip connection 1AG are added together and input to the second activation function 1AF. Therefore, the skip connection 1AG can suppress gradient vanishing due to the depth of the layer (that is, the residual block 1A24 having the skip connection 1AG can suppress gradient vanishing due to the depth of the layer).

[0020] In the example shown in Figure 2, the classification model 1A further comprises a convolutional layer 1A31, a batch normalization unit 1A32, an activation function 1A33, and a residual block 1A34. The input side of the convolutional layer 1A31 is connected to the output side of the residual block 1A24 (more specifically, to the output side of the second activation function 1AF of the residual block 1A24). The output side of the convolutional layer 1A31 is connected to the input side of the batch normalization unit 1A32. The output side of the batch normalization unit 1A32 is connected to the input side of the activation function 1A33. The output side of the activation function 1A33 is connected to the input side of the residual block 1A34. The residual block 1A34 is configured similarly to the residual block 1A24 shown in Figure 2(B).

[0021] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, further comprises a convolutional layer 1A31 and an activation function 1A33. The input side of convolutional layer 1A31 is connected to the output side of activation function 1A23. The output side of convolutional layer 1A31 is connected to the input side of activation function 1A33.

[0022] In the example shown in Figure 2, the classification model 1A further comprises a convolutional layer 1A41, a batch normalization unit 1A42, an activation function 1A43, and a residual block 1A44. The input side of the convolutional layer 1A41 is connected to the output side of the residual block 1A34 (more specifically, to the output side of the second activation function 1AF of the residual block 1A34). The output side of the convolutional layer 1A41 is connected to the input side of the batch normalization unit 1A42. The output side of the batch normalization unit 1A42 is connected to the input side of the activation function 1A43. The output side of the activation function 1A43 is connected to the input side of the residual block 1A44. The residual block 1A44 is configured similarly to the residual block 1A24 shown in Figure 2(B).

[0023] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, further comprises a convolutional layer 1A41 and an activation function 1A43. The input side of the convolutional layer 1A41 is connected to the output side of the activation function 1A33. The output side of the convolutional layer 1A41 is connected to the input side of the activation function 1A43.

[0024] In the example shown in Figure 2, the classification model 1A further comprises a convolutional layer 1A51, a batch normalization unit 1A52, an activation function 1A53, and a residual block 1A54. The input side of the convolutional layer 1A51 is connected to the output side of the residual block 1A44 (more specifically, to the output side of the second activation function 1AF of the residual block 1A44). The output side of the convolutional layer 1A51 is connected to the input side of the batch normalization unit 1A52. The output side of the batch normalization unit 1A52 is connected to the input side of the activation function 1A53. The output side of the activation function 1A53 is connected to the input side of the residual block 1A54. The residual block 1A54 is configured similarly to the residual block 1A24 shown in Figure 2(B).

[0025] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, further comprises a convolutional layer 1A51 and an activation function 1A53. The input side of the convolutional layer 1A51 is connected to the output side of the activation function 1A43. The output side of the convolutional layer 1A51 is connected to the input side of the activation function 1A53.

[0026] In the example shown in Figure 2, the classification model 1A further comprises a convolutional layer 1A61, a batch normalization unit 1A62, an activation function 1A63, and a residual block 1A64. The input side of the convolutional layer 1A61 is connected to the output side of the residual block 1A54 (specifically, the output side of the second activation function 1AF of the residual block 1A54). The output side of the convolutional layer 1A61 is connected to the input side of the batch normalization unit 1A62. The output side of the batch normalization unit 1A62 is connected to the input side of the activation function 1A63. The output side of the activation function 1A63 is connected to the input side of the residual block 1A64. The residual block 1A64 is configured similarly to the residual block 1A24 shown in Figure 2(B).

[0027] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, further comprises a convolutional layer 1A61 and an activation function 1A63. The input side of the convolutional layer 1A61 is connected to the output side of the activation function 1A53. The output side of the convolutional layer 1A61 is connected to the input side of the activation function 1A63.

[0028] In the example shown in Figure 2, the classification model 1A further comprises a convolutional layer 1A71, a batch normalization unit 1A72, an activation function 1A73, a residual block 1A74, a capsule layer 1A8, and an output layer 1A9. The input side of the convolutional layer 1A71 is connected to the output side of the residual block 1A64 (more specifically, to the output side of the second activation function 1AF of the residual block 1A64). The output side of the convolutional layer 1A71 is connected to the input side of the batch normalization unit 1A72. The output side of the batch normalization unit 1A72 is connected to the input side of the activation function 1A73. The output side of the activation function 1A73 is connected to the input side of the residual block 1A74. The residual block 1A74 is configured similarly to the residual block 1A24 shown in Figure 2(B). The input side of capsule layer 1A8 is connected to the output side of residual block 1A74 (specifically, the output side of the second activation function 1AF of residual block 1A74). The output side of capsule layer 1A8 is connected to output layer 1A9.

[0029] On the other hand, in the comparative example shown in Figure 3, the capsule network 1AN, to which skip connections are not applied, further comprises a convolutional layer 1A71, an activation function 1A73, a capsule layer 1A8, and an output layer 1A9. The input side of the convolutional layer 1A71 is connected to the output side of the activation function 1A63. The output side of the convolutional layer 1A71 is connected to the input side of the activation function 1A73. The input side of capsule layer 1A8 is connected to the output side of activation function 1A73. The output side of capsule layer 1A8 is connected to output layer 1A9.

[0030] Figure 4 is a flowchart showing an example of the process performed in the diagnostic support device 1 of the first embodiment. In the example shown in Figure 4, in step S1, the learning unit 12 performs supervised learning of a classification model 1A that classifies chest X-ray images based on the presence or absence of pulmonary hypertension. The classification model 1A includes a capsule network 1AN (see Figure 3) capable of learning the positional relationships of features, and a skip connection 1AG that suppresses gradient vanishing due to the depth of the layers. In the supervised learning performed in step S1, training chest X-ray images with known presence or absence of pulmonary hypertension are used as training data. Next, in step S2, the prediction unit 11 makes a prediction using the classification model 1A, which underwent supervised learning in step S1. The prediction results from step S2 support, for example, the diagnosis of the presence or absence of pulmonary hypertension based on chest X-ray images by a specialist.

[0031] <First Example> (Background of the research) Pulmonary hypertension refers to a condition in which blood pressure in the pulmonary arteries is high, causing shortness of breath and leading to heart failure. Early detection of pulmonary hypertension is crucial because early treatment can improve the prognosis. However, unlike blood pressure in other blood vessels throughout the body, which can be easily measured with non-invasive devices such as Manshent, pulmonary artery blood pressure can only be measured through highly invasive cardiac catheterization, often leading to delayed diagnosis. Furthermore, the scarcity of doctors and nurses familiar with pulmonary hypertension is another reason for delayed diagnosis. Furthermore, in recent years, it has become known that increased pulmonary artery pressure due to lung damage caused by COVID-19 can worsen the prognosis of patients, further increasing the clinical importance of diagnosing pulmonary hypertension. Chest X-rays, commonly used in medical settings and minimizing patient burden, offer several advantages: ease of acquisition, high accessibility (imaging devices can be mounted in vehicles), and high reproducibility (the same image is obtained regardless of who takes the image). While chest X-rays are used to suspect pulmonary hypertension, the detection accuracy by experts is only around 60%, and the development of more accurate methods is desired. The inventors have been conducting research on improving the accuracy of identifying myocardial infarction and evaluating cardiac function by using artificial intelligence (AI) in medical images in the cardiovascular field. In the first embodiment, regarding pulmonary hypertension, the inventors investigated whether it is possible to accurately detect elevated pulmonary artery pressure, which can only be accurately diagnosed by cardiac catheterization, using chest X-ray images, and whether it is possible to predict future disease progression, by using AI technology, particularly deep learning, a new type of AI.

[0032] (Research results) The inventors have developed artificial intelligence that detects pulmonary hypertension from chest X-ray images, which corresponds to classification model 1A (see Figure 2) of the diagnostic support device 1 of the first embodiment.

[0033] Figure 5 is a diagram illustrating the classification model 1A (neural network) of the first embodiment. In the first embodiment, as shown in Figure 5, a chest X-ray image is input to classification model 1A (neural network). Classification model 1A outputs information as a classification result indicating whether the input chest X-ray image corresponds to the presence of pulmonary hypertension (High PAP) or the absence of pulmonary hypertension (Normal PAP).

[0034] (Data collection) In the supervised learning of classification model 1A, 900 cases of chest X-ray images and mean pulmonary artery pressure combinations (439 cases of pulmonary hypertension, 461 cases of normal) were used as training data. Mean pulmonary artery pressure is a value that can be measured by right heart catheterization and is an indicator for diagnosing pulmonary hypertension. In the first example, a mean pulmonary artery pressure of 20 mmHg or higher was defined as pulmonary hypertension, and anything below that was defined as normal.

[0035] (Data preprocessing) The extracted chest X-ray images included low-resolution images and images containing foreign objects (pacemakers, stents, etc.). Because these images could hinder the training of classification model 1A, three datasets were created. The first dataset consists of all 900 chest X-ray images. The second dataset consists of the first dataset with the low-resolution images removed. The third dataset consists of the second dataset with the images containing foreign objects removed.

[0036] (Construction and training of classification model 1A) As classification model 1A, we constructed a model that adds skip connections 1AG, which allow for deeper convolutional layers while suppressing vanishing gradients, to capsule network 1AN (see Figure 3), which is capable of learning the spatial relationships of feature quantities. To validate classification accuracy, we used 10-fold cross-validation and trained the model on each dataset. In the first embodiment, since the number of data points was small, with a maximum of 900 cases, fine tuning was performed using the weights of a model trained on 25,684 chest X-ray images (Normal: 8,525, Lung Opacity: 5,659, Not Opacity Not Normal: 11,500) published in the RSNA-Pneumonia Detection Challenge (https: / / www.kaggle.com / c / rsna-pneumonia-detection-challenge) as pre-training.

[0037] (Results / Discussion) After 10-fold cross-validation, the average AUC (Area Under the Curve) for the first, second, and third datasets was 0.67. Furthermore, in order to compare the accuracy with previous studies and those of specialists, evaluation was performed using the k=1 evaluation set that achieved the highest classification accuracy in the first dataset. Compared to the AUC of 0.71 for the evaluation set in the first example, the AUCs of conventional methods for measuring anatomical structures (https: / / pubmed.ncbi.nlm.nih.gov / 25191469 / ) and those of specialists were 0.60 and 0.63, respectively.

[0038] Figure 6 shows the evaluation set, the conventional method for measuring anatomical structures, and the ROC (Receiver Operating Characteristic) curve of the specialist in the first embodiment. In Figure 6, "Hilum" shows the ROC curve of a conventional method for measuring anatomical structures, "Expert" shows the ROC curve of a specialist, and "AI" shows the ROC curve of the evaluation set in the first embodiment.

[0039] In our research, we found that when the prediction unit 11 of the diagnostic support device 1 of the first embodiment performed a prediction using classification model 1A and cases in which pulmonary hypertension was suspected were followed up over a long period, the probability of subsequent disease deterioration was approximately twice as high compared to cases in which pulmonary hypertension was not predicted. These results suggest that using the diagnostic support device 1 of the first embodiment may allow for the determination of not only the presence of pulmonary hypertension but also its subsequent prognosis, compared to human observation. Furthermore, a problem has been that the decision-making processes of many deep learning systems to date are black boxes, making it difficult even experts to explain the reasons and rationale behind the AI's answers. To solve this problem, the inventors applied an algorithm (Grad-CAM) that visualizes the area of ​​interest, displaying the areas in the image that the AI ​​is focusing on and making decisions about, using shades of color on a plain chest X-ray image.

[0040] Figure 7 visualizes the areas of interest identified by AI (Classification Model 1A) in chest X-ray images. In detail, Figure 7(A) shows a chest X-ray image of a patient with pulmonary hypertension, and the AI ​​(Classification Model 1A) focuses on the areas of interest in the chest X-ray image classified as having pulmonary hypertension by Classification Model 1A. In the example shown in Figure 7(A), AI (Classification Model 1A) focuses on the upper right lung and the area around the heart. Figure 7(B) shows a chest X-ray image of a healthy individual without pulmonary hypertension, and indicates the region of focus of AI (Classification Model 1A) in the chest X-ray image classified as not having pulmonary hypertension by Classification Model 1A. In the example shown in Figure 7(B), AI (Classification Model 1A) focuses on the area around both lung hilum. The inventors confirmed from the results shown in Figure 7 that the areas of focus of the AI ​​and the areas of focus of the physicians coincide. Therefore, it is expected that this AI model will be used in clinical practice as a more reliable and "explainable AI."

[0041] (Significance of the research) By applying the AI ​​technology developed in this study to commonly used chest X-ray images, it is expected that pulmonary hypertension can be detected non-invasively with higher accuracy than conventional visual image diagnosis, and that prognosis can be estimated. This is a key research project for future applications of AI technology in the field of cardiovascular imaging, and its application to various diseases that cause pulmonary hypertension (such as COVID-19) is also expected.

[0042] <Second Example> The inventors constructed a classification model 1A (see Figure 2) for the presence or absence of pulmonary hypertension by applying a deep neural network to chest X-ray images, which can be examined simply and at low cost. Pulmonary hypertension is a disease that leads to a poor prognosis once it develops, but because there are effective treatments, early detection is crucial. Accurate diagnosis of this disease requires invasive catheterization, so ultrasound has traditionally been used as a screening test. However, ultrasound is costly and requires skilled technicians, making it difficult for everyone to easily undergo the examination. It has long been known that pulmonary hypertension causes abnormalities in the anatomical distance ratios of the heart and pulmonary vascular shadows in chest X-ray images. Therefore, the inventors developed a deep neural network that directly predicts pulmonary hypertension using chest X-ray images as input (without measuring anatomical quantities). The inventors have constructed a new classification model 1A (see Figure 2) based on a capsule network 1AN (see Figure 3) capable of learning the positional relationships of feature quantities, by adding a skip connection 1AG (see Figure 2) that allows for deeper convolutional layers while suppressing gradient vanishing. As a result, they were able to classify pulmonary hypertension and normal with higher accuracy than conventional methods and specialists. Using classification model 1A (see Figure 2), which utilizes chest X-ray images for simple and low-cost examination, a new screening test for pulmonary hypertension can be implemented to identify patients suspected of having pulmonary hypertension simply by inputting chest X-ray images taken at a hospital into a cloud-based system.

[0043] Figure 8 is a diagram illustrating the accuracy and AUC of the classification model 1A in the second embodiment when 10-fold cross-validation was performed. In detail, Figure 8(A) shows a comparison of the accuracy of the classification model 1A of the second embodiment (i.e., the classification model 1A shown in Figure 2, to which the capsule network 1AN and skip connection 1AG are applied) when 10-fold cross-validation was performed, and the accuracy of the classification model of the comparative example (i.e., the classification model shown in Figure 3, to which the capsule network 1AN is applied, but the skip connection 1AG is not applied) when 10-fold cross-validation was performed. Figure 8(B) shows a comparison of the AUC for classification model 1A of the second embodiment (i.e., classification model 1A shown in Figure 2, in which the capsule network 1AN and skip connection 1AG are applied) when 10-fold cross-validation was performed, and the AUC for the classification model of the comparative example (i.e., the classification model shown in Figure 3, in which the capsule network 1AN is applied but the skip connection 1AG is not applied) when 10-fold cross-validation was performed.

[0044] [Second Embodiment] A second embodiment of the diagnostic support device, diagnostic support method, and program of the present invention will be described below. The diagnostic support device 1 of the second embodiment is configured in the same way as the diagnostic support device 1 of the first embodiment described above, except for the points described later. Therefore, the diagnostic support device 1 of the second embodiment can achieve the same effects as the diagnostic support device 1 of the first embodiment described above, except for the points described later.

[0045] In the example shown in Figure 2 of the diagnostic support device 1 of the first embodiment, the classification model 1A is provided with six convolutional layers 1A21 to 1A71, six batch normalization units 1A22 to 1A72, six activation functions 1A23 to 1A73, and six residual blocks 1A24 to 1A74. In other words, in the example shown in Figure 2 of the diagnostic support device 1 of the first embodiment, the classification model 1A is provided with six sets of layers, each consisting of a convolutional layer, a batch normalization unit, an activation function, and a residual block. In the diagnostic support device 1 of the second embodiment, the number of layers in the classification model 1A, which are composed of convolutional layers, batch normalization units, activation functions, and residual blocks, is any number other than 6 (provided that it is 2 or more).

[0046] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and modifications can be made as appropriate without departing from the spirit of the present invention. The configurations described in each of the embodiments and examples above may be combined.

[0047] In addition, all or part of the diagnostic support device 1 in the above embodiment may be implemented by dedicated hardware, or it may be implemented by memory and a microprocessor. Furthermore, the diagnostic support device 1 may consist of all or part of memory and a CPU (central processing unit), and may achieve its functions by loading and executing programs into memory to realize the functions of each part of the system. Alternatively, a program to implement all or part of the functions of the diagnostic support device 1 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform the processing of each part. The term "computer system" here includes hardware such as the operating system and peripheral devices. Furthermore, if a WWW system is used, the "computer system" also includes the homepage provisioning environment (or display environment). Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. In addition, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system. [Explanation of symbols]

[0048] 1...Diagnostic support device, 11...Prediction unit, 12...Learning unit, 1A...Classification model, 1A1...Input layer, 1A21, 1A31, 1A41, 1A51, 1A61, 1A71...Convolutional layer, 1A22, 1A32, 1A42, 1A52, 1A62, 1A72...Batch normalization unit, 1A23, 1A33, 1A43, 1A53, 1A63, 1A73...Activation function, 1A24, 1A34, 1A44, 1A54, 1A64, 1A74…Residual Blocks, 1A8…Capsule Layer, 1A9…Output Layer, 1AA…First Convolutional Layer, 1AB…First Batch Normalizer, 1AC…First Activation Function, 1AD…Second Convolutional Layer, 1AE…Second Batch Normalizer, 1AF…Second Activation Function, 1AG…Skip Connection, 1AN…Capsule Network

Claims

1. A diagnostic support device that assists in the diagnosis of the presence or absence of pulmonary hypertension based on chest X-ray images, A classification model that classifies the aforementioned chest X-ray images based on the presence or absence of pulmonary hypertension, A prediction unit that performs predictions using the aforementioned classification model, The system comprises a learning unit that performs supervised learning of the classification model before the classification model is used by the prediction unit, In the supervised learning performed by the aforementioned learning unit, training chest X-ray images with or without pulmonary hypertension are used as training data. The classification model comprises a capsule network capable of learning the spatial relationships of feature quantities, and skip connections that suppress gradient vanishing due to increasing layer depth. The classification model includes multiple residual blocks that suppress gradient vanishing due to increasing layer depth, Each residual block is, The first convolutional layer and A first batch normalization unit connected to the output side of the first convolutional layer, The first activation function connected to the output side of the first batch normalization unit, A second convolutional layer connected to the output side of the first activation function, A second batch normalization unit connected to the output side of the second convolutional layer, The second activation function connected to the output side of the second batch normalization unit, The system comprises a skip connection that connects the input side of the first convolutional layer, the output side of the second batch normalization unit, and the input side of the second activation function. Diagnostic support device.

2. The classification model comprises, as the plurality of residual blocks, a first residual block, a second residual block, a third residual block, a fourth residual block, a fifth residual block, and a sixth residual block. The diagnostic support device according to claim 1.

3. The output side of the activation function is connected to the input side of each residual block. The output side of the batch normalization unit is connected to the input side of the activation function connected to the input side of each residual block. The input side of the batch normalization unit, which is connected to the input side of each residual block via an activation function, is connected to the output side of the convolutional layer. A diagnostic support device according to claim 1 or claim 2.

4. A diagnostic support method that assists in the diagnosis of the presence or absence of pulmonary hypertension based on chest X-ray images, A prediction step in which, after supervised learning has been performed, the chest X-ray image is input to a classification model that classifies the chest X-ray image according to the presence or absence of pulmonary hypertension, and the classification model outputs information indicating whether or not the person has pulmonary hypertension as a classification result, The prediction step includes a learning step in which supervised learning of the classification model is performed before the classification model is used in the prediction step. In the supervised learning performed in the aforementioned learning step, training chest X-ray images with or without pulmonary hypertension are used as training data. The classification model comprises a capsule network capable of learning the spatial relationships of feature quantities, and skip connections that suppress gradient vanishing due to increasing layer depth. The classification model includes multiple residual blocks that suppress gradient vanishing due to increasing layer depth, Each residual block is, The first convolutional layer and A first batch normalization unit connected to the output side of the first convolutional layer, The first activation function connected to the output side of the first batch normalization unit, A second convolutional layer connected to the output side of the first activation function, A second batch normalization unit connected to the output side of the second convolutional layer, The second activation function connected to the output side of the second batch normalization unit, The system comprises a skip connection that connects the input side of the first convolutional layer, the output side of the second batch normalization unit, and the input side of the second activation function. Diagnostic support methods.

5. On the computer, A prediction step involves using a classification model that classifies chest X-ray images based on the presence or absence of pulmonary hypertension, and A program for causing a learning step to perform supervised learning of the classification model before the classification model is used in the prediction step, In the supervised learning performed in the aforementioned learning step, training chest X-ray images with or without pulmonary hypertension are used as training data. The classification model comprises a capsule network capable of learning the spatial relationships of feature quantities, and skip connections that suppress gradient vanishing due to increasing layer depth. The classification model includes multiple residual blocks that suppress gradient vanishing due to increasing layer depth, Each residual block is, The first convolutional layer and A first batch normalization unit connected to the output side of the first convolutional layer, The first activation function connected to the output side of the first batch normalization unit, A second convolutional layer connected to the output side of the first activation function, A second batch normalization unit connected to the output side of the second convolutional layer, The second activation function connected to the output side of the second batch normalization unit, The system comprises a skip connection that connects the input side of the first convolutional layer, the output side of the second batch normalization unit, and the input side of the second activation function. program.

Images