Abnormal shadow extraction method, abnormal shadow extraction program, abnormal shadow extraction device, and image diagnostic support device

Abstract

This invention provides an abnormal shadow extraction method, an abnormal shadow extraction program, an abnormal shadow extraction device, and an image diagnostic support device that enable improved diagnostic accuracy even with a single image capture. [Solution] The abnormal shadow extraction method includes the steps of: acquiring an image with abnormal shadows and an image without abnormal shadows associated with the image with abnormal shadows as a training dataset; using the training dataset, using the image with abnormal shadows from the training dataset as input data and the image without abnormal shadows associated with the image with abnormal shadows as output data, and training a neural network to learn by associating the two; inputting an image to be classified into the trained neural network and outputting an image without abnormal shadows corresponding to the image to be classified; and obtaining and extracting the difference between the image to be classified and the output image without abnormal shadows.

Description

Technical Field

[0001] This invention relates to an abnormal shadow extraction method, an abnormal shadow extraction program, an abnormal shadow extraction device, and an image diagnosis support device.

Background Art

[0002] In diagnosis using medical images, in order to perform diagnosis with high accuracy, a method of taking a plurality of images by changing imaging conditions for the same part is known. As a method of obtaining one X-ray image by performing two X-ray shootings at different tube voltages, there is a technique called dual energy subtraction shooting (see, for example, Patent Document 1). In this shooting method, after performing the first X-ray shooting at a high energy first tube voltage, the second X-ray shooting is performed at a low energy second tube voltage, and the difference between these two types of X-ray shooting image data can be taken. Thereby, from the captured images, for example, an image with the bone part removed or an image with abnormal shadows extracted can be obtained, and the diagnostic accuracy can be improved.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in this shooting method, since two shootings, one with high energy X-rays and the other with low energy X-rays, are required, there is a problem that the radiation dose increases. Therefore, the main object of the present invention is to provide an abnormal shadow extraction method, an abnormal shadow extraction program, an abnormal shadow extraction device, and an image diagnosis support device that can improve diagnostic accuracy even with a single shooting.

Means for Solving the Problems

[0005] The abnormal shadow extraction method according to the present invention is A training data acquisition step in which an image with abnormal shadowing and an image without abnormal shadowing associated with the image with abnormal shadowing are obtained as a training dataset, A learning step in which the neural network is trained to learn by associating the two, using the aforementioned training dataset, with the aforementioned images with abnormal shadows from the training dataset as input data and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, The image output step involves inputting the image to be classified into the pre-trained neural network and outputting an image without abnormal shading corresponding to the image to be classified. The method is characterized by performing an extraction step of obtaining and extracting the difference between the image to be identified and the output image without abnormal shadows.

[0006] In the abnormal shadow extraction method of the present invention, it is preferable to have an abnormal shadow enhancement step in which the difference is visually enhanced and combined with the target image for discrimination or the output abnormal shadow-free image.

[0007] In the abnormal shadow extraction method of the present invention, it is preferable to use an image set as the training dataset, which consists of a pair of medical images without abnormal shadows and images with abnormal shadows obtained by combining the medical images with simulated lesion images.

[0008] The abnormal shadow extraction program according to the present invention is A training data acquisition step in which an image with abnormal shadowing and an image without abnormal shadowing associated with the image with abnormal shadowing are obtained as a training dataset, A learning step in which the neural network is trained to learn by associating the two, using the aforementioned training dataset, with the aforementioned images with abnormal shadows from the training dataset as input data and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, The image output step involves inputting the image to be classified into the pre-trained neural network and outputting an image without abnormal shading corresponding to the image to be classified. This is an abnormal shadow extraction program for executing an extraction step that obtains and extracts the difference between the image to be identified and the output image without abnormal shadows.

[0009] In the abnormal shadow extraction program of the present invention, it is preferable to have an abnormal shadow enhancement step in which the difference is visually enhanced and combined with the target image for discrimination or the output image without abnormal shadows.

[0010] In the abnormal shadow extraction program of the present invention, it is preferable to use an image set as the training dataset, which consists of a pair of medical images without abnormal shadows and images with abnormal shadows obtained by combining the medical images with simulated lesion images.

[0011] The abnormal shadow extraction device according to the present invention is A training data acquisition unit that acquires an image with abnormal shadows and an image without abnormal shadows associated with the image with abnormal shadows as a training data set, A learning unit that uses the aforementioned training dataset, takes the aforementioned images with abnormal shadows from the training dataset as input data, and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, and performs machine learning on a neural network by associating the two. When the pre-trained neural network receives an image to be classified, it outputs an image without abnormal shadows corresponding to the image to be classified. The system is characterized by comprising an extraction unit that acquires and extracts the difference between the image to be discriminated against and the output image without abnormal shadows.

[0012] In the abnormal shadow extraction device of the present invention, it is preferable to have an abnormal shadow enhancement unit that visually enhances the difference and synthesizes it with the target image for discrimination or the output image without abnormal shadows.

[0013] In the abnormal shadow extraction device of the present invention, it is preferable to use an image set as the training dataset, which consists of a pair of medical images without abnormal shadows and images with abnormal shadows obtained by combining the medical images with simulated lesion images.

[0014] The image diagnosis support device according to the present invention includes the abnormal shadow extraction device of the present invention, and is provided with a display unit, wherein the display unit is characterized in that it displays, in a comparable state, the discrimination target image and an image obtained by visually emphasizing the extracted difference and synthesizing it with the discrimination target image or the output abnormal shadow-free image.

Effect of the Invention

[0015] According to the present invention, it is possible to provide an abnormal shadow extraction method, an abnormal shadow extraction program, an abnormal shadow extraction device, and an image diagnosis support device that can improve the diagnosis accuracy even with a single imaging. As a result, an abnormal shadow can be extracted with a small amount of radiation exposure.

[0016] The above objects, other objects, features, and advantages of this invention will become more apparent from the following detailed description of examples of embodiments made with reference to the drawings.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram for explaining a machine learning step in the abnormal shadow extraction method of the present invention. [Figure 2] It is a diagram for explaining a step of outputting an abnormal shadow-free image in the abnormal shadow extraction method of the present invention. [Figure 3] It is a diagram for explaining an extraction step in the abnormal shadow extraction method of the present invention. [Figure 4] It is a flowchart of an example of a processing procedure for abnormal shadow extraction. [Figure 5] It is an example of an image pair for machine learning. [Figure 6] FIG. 6(A) is an example of a discrimination target image. FIG. 6(B) is an abnormal shadow-free image corresponding to the discrimination target image of FIG. 6(A). [Figure 7] FIG. 7 is an abnormal shadow extraction image showing the difference between FIG. 6(A) and FIG. 6(B) as an image. [Figure 8] Figure 8(A) shows the extracted abnormal shadows visually enhanced and superimposed onto the target image. Figure 8(B) shows the image superimposed with the abnormal shadows enhanced using a heatmap display. [Figure 9] This is an example of a training image pair for chest CT. [Figure 10] Figure 10(A) is the chest CT image to be identified. Figure 10(B) is the image without abnormal shadows corresponding to the image to be identified in Figure 10(A). Figure 10(C) is the image with extracted abnormal shadows, showing the difference between Figure 10(A) and Figure 10(B). Figure 10(D) is the image to be identified, with the extracted abnormal shadows visually enhanced and composited onto the image to be identified. Figure 10(E) is the image with the abnormal shadows enhanced using a heatmap display and then composited. [Figure 11] This is an example of a training image pair for head MRI. [Figure 12] Figure 10(A) is the chest CT image to be identified. Figure 10(B) is the image without abnormal shadows corresponding to the image to be identified in Figure 10(A). Figure 10(C) is the image with extracted abnormal shadows, showing the difference between Figure 10(A) and Figure 10(B). Figure 10(D) is the image to be identified, with the extracted abnormal shadows visually enhanced and composited onto the image to be identified. Figure 10(E) is the image with the abnormal shadows enhanced using a heatmap display and then composited. [Figure 13] This block diagram shows an example of the configuration of a medical support system to which the present invention is applied. [Modes for carrying out the invention]

[0018] The present invention will be explained with examples. However, the present invention is not limited to the following examples. Note that the drawings referenced below include some that have been created for illustrative purposes from the standpoint of protecting personal information. In addition, the ratios of dimensions of objects depicted in the drawings may differ from the ratios of dimensions of actual objects, and the ratios of dimensions of objects may also differ between drawings.

[0019] The method for extracting abnormal shadows in the present invention comprises a training data acquisition step of acquiring a training dataset, a training step of machine learning a neural network, an image output step of inputting an image to be classified and outputting an image without abnormal shadows corresponding to the image to be classified, and an extraction step of acquiring and extracting the difference between the image to be classified and the output image without abnormal shadows.

[0020] The aforementioned training dataset consists of images with abnormal shading and images without abnormal shading that are associated with the images with abnormal shading.

[0021] In machine learning, it is crucial to train the model on a variety of patterns. For example, in chest radiographs, the resulting image pattern may differ depending on the subject's body type (thin or obese), gender, and sexual orientation. Obese individuals and women may show breast shadows. Furthermore, the image differs depending on whether or not the patient has a history of surgery for breast cancer or other conditions. To prevent the model from misidentifying these shadows or surgical history-derived features as abnormal shadows, it is desirable to train the model on images without abnormal shadows, and also to train it on images with simulated lesions added to these images as images with abnormal shadows. Additionally, since bones are relatively clearly visible in younger individuals but may not be clearly visible in older individuals, it is important to collect chest radiographs from different age groups. While it is generally difficult to collect many images with abnormal shadows from younger individuals, even in populations with few such images, creating images with simulated lesions by adding them to images without abnormal shadows can yield a large amount of data on images with abnormal shadows. The aforementioned training dataset may be prepared by collecting real-world images, but images without abnormal shadows can be collected relatively easily through regular health checkups, etc., and by generating images with lesions (abnormal shadows) using the method described above, a sufficient number of image sets (training dataset) with and without abnormal shadows can be prepared for machine learning.

[0022] In the aforementioned learning step, the training dataset is used, with the abnormally shaded images from the training dataset as input data and the abnormally unshaded images associated with the abnormally shaded images as output data, and the neural network is trained to learn by associating the two. Specifically, the AI ​​is trained using autoencoder technology or the like so that it can create the original abnormally unshaded image from the created (or collected) abnormally shaded images.

[0023] An autoencoder is a type of neural network (a mathematical model that mimics the structure of the human brain), consisting of an input layer, an output layer, and at least one hidden layer, and is used for data generation, anomaly detection, noise reduction, etc. In training, first, many pairs of images with and without anomaly shading are prepared. In each pair, there is no difference in the image except for the anomaly shading. Then, as shown in Figure 1, by training the autoencoder with the image with anomaly shading as input to each pair and the image without anomaly shading as the correct answer, a model for removing anomaly shading can be realized. In an autoencoder, the encoder extracts feature data from the input image, and the decoder receives the extracted feature data as input and reconstructs it into the original image format before encoding. Since the autoencoder can be trained to discover and remove hidden patterns of anomaly shading in the data, it can be suitably used for anomaly shading removal in this invention.

[0024] In the image output step, the image to be classified is input to the trained neural network, and an image without abnormal shadows corresponding to the image to be classified is output. Figure 2 is a diagram illustrating the step of outputting an image without abnormal shadows. By using the model, which has been trained for all pairs, when an unknown image with abnormal shadows is input, an image without abnormal shadows is generated and output. For example, when there is a shadow in the area where the lung and bone overlap, it was difficult to determine from a single image whether the shadow was in the lung soft tissue or the bone tissue. Therefore, conventionally, in order to improve the diagnostic accuracy of X-ray images, a method has been used in which two images are taken by irradiating with X-rays of different energies, such as high-energy X-rays and low-energy X-rays, the difference between the two images is obtained, and abnormal shadows are extracted. In the present invention, by using an AI trained to generate an image without abnormal shadows from an image with abnormal shadows, an image without abnormal shadows that can be used to take the difference corresponding to the image to be classified is obtained. Since two images that capture the difference can be obtained with just one scan, accurate diagnosis can be achieved while suppressing the radiation exposure of the patient.

[0025] In the extraction step, the difference between the image to be identified and the output image without abnormal shadows is obtained and extracted. As shown in Figure 3, by obtaining and extracting the difference between the image to be identified and the output image without abnormal shadows, the abnormal shadows can be made to stand out (come to light). In Figure 3, for clarity, the area with abnormal shadows is enclosed in a white rectangle.

[0026] Figure 4 shows a flowchart of an example of the processing procedure for abnormal shadow extraction. This example includes the step of creating images with abnormal shadows to obtain a training dataset. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, depending on the embodiment, steps in the following processing procedure can be omitted, replaced, and added as appropriate.

[0027] In step S1, a large number of medical images are prepared. In step S3, simulated lesions are added to the medical images prepared in step S1 to create images with abnormal shadows. An example of the prepared medical images is shown in Figure 5(A), and an example of the image with abnormal shadows obtained in step S3 is shown in Figure 5(B).

[0028] In step S5, the AI ​​is trained using autoencoder technology so that it can create the original image without abnormal shadows from the image with abnormal shadows created in step S3 (see Figure 1).

[0029] In step S7, the image to be classified is passed to the AI ​​that was trained in step S5. Figure 6(A) shows an example of an image to be classified, which has an abnormal shadow that is not a simulated lesion. In the figure, the area circled with a white line is the actual abnormal shadow.

[0030] In step S9, an image without abnormal shadows corresponding to the image to be identified is output. Figure 6(B) is the image without abnormal shadows output corresponding to the image to be identified in Figure 6(A). As shown in Figure 6(B), the shadows that were present in the area circled with a white line in Figure 6(A) are gone in the output.

[0031] In step S11, the difference between the image to be identified used in step S7 and the image without abnormal shadows output in step S9 is obtained and extracted. Figure 7 is an image with extracted abnormal shadows, showing the difference as an image. In this figure, the area circled with a white line is the abnormal shadow extracted as the difference.

[0032] The image without abnormal shadows output in this invention can be used for image diagnostic support. By displaying the output image without abnormal shadows alongside the original image with abnormal shadows (the image to be identified), it becomes easier to recognize the presence of abnormal shadows in the image to be identified, and this can be effectively used during image interpretation and when explaining to patients. By comparing the image without abnormal shadows side by side, the anatomical location can also be easily grasped.

[0033] Furthermore, by creating a composite image of the extracted abnormal shadows visually enhanced and combined with the target image or the output image without abnormal shadows, and displaying it alongside the original image with abnormal shadows, the location of the abnormal shadows can be made clearer. Figure 8(A) shows the extracted abnormal shadows visually enhanced and combined with the target image. When visually enhancing, it is preferable to display the strength of the difference using color coding, such as in a heat map display, to make it easier to see. Figure 8(B) is an image created by combining abnormal shadows with a heat map display that highlights the strength of the difference using color coding (in the patent drawings, the color image is displayed in grayscale). When viewing images, for example, by displaying the target image on the left and the image with the abnormal shadows visually enhanced on the right, image diagnosis can be supported. In this case, by confirming the location in the image on the right and observing the location indicated in the image on the left in detail, it is possible to diagnose abnormal areas without overlooking them during image interpretation.

[0034] Furthermore, the type of medical image is not limited as long as it can be used to support diagnosis. The method of the present invention can be applied to any medical image, such as radiographic images (X-rays), CT images, MRI images, and ultrasound images. The body part to be identified may be any part such as the chest, abdomen, or head, a specific organ, tissue, or a portion of the body, or the whole body.

[0035] Figure 9 shows an example of a training image pair for chest CT. Figure 9(A) is an image without abnormal shadows before the addition of a simulated lesion, and Figure 9(B) is an image after the addition of a simulated lesion. In Figure 9(B), the area circled with a white line is the simulated lesion. Many such image pairs are prepared to train the neural network. Then, an image with an actual abnormal shadow (the image to be discriminated) is input to the trained neural network. Here, the chest CT image to be discriminated shown in Figure 10(A) is input, and the output result is the image with the abnormal shadow removed shown in Figure 10(B) (the image without abnormal shadows corresponding to the image to be discriminated in Figure 10(A)). In Figure 10(A), the area circled with a white line is the actual abnormal shadow. Figure 10(C) is the image obtained by taking the difference between Figure 10(A) and Figure 10(B) and extracting only the abnormal shadow portion. In Figure 10(C), the area circled with a white line is the extracted abnormal shadow portion. Figure 10(D) is an image with abnormal shading emphasized. Figure 10(E) is a grayscale display of the image with the abnormal shading emphasized using a heatmap display, where the strength of the difference is color-coded.

[0036] Figure 11 shows an example of a training image pair for a head MRI. Figure 11(A) is an image without abnormal shadows before the addition of a simulated lesion, and Figure 11(B) is an image after the addition of a simulated lesion. In Figure 11(B), the area circled with a white line is the simulated lesion. Many such image pairs are prepared to train the neural network. Then, an image containing an actual abnormal shadow (the image to be discriminated) is input to the trained neural network. Here, the head MRI image to be discriminated shown in Figure 12(A) is input, and the output result is the image with the abnormal shadow removed shown in Figure 12(B). In Figure 12(A), the area circled with a white line is the actual abnormal shadow. Figure 12(C) is the image obtained by taking the difference between Figure 12(A) and Figure 12(B) and extracting only the abnormal shadow area. In Figure 12(C), the area circled with a white line is the extracted abnormal shadow area. Figure 12(D) is an image with the abnormal shadow emphasized. Figure 12(E) shows an image in grayscale with abnormal shadows highlighted using a heatmap display, where the strength of the difference is color-coded and emphasized.

[0037] Referring to Figure 13, a medical diagnostic support system that uses training images obtained according to the present invention consists, for example, of an image diagnostic support device 10, modality 20, PACS 30, and machine learning system (analysis device) 40 connected to each other via an in-hospital LAN 50.

[0038] In the image diagnostic support device 10, the bus BS1 is connected to a communication interface 12cm, a CPU 12pr, a keyboard / mouse 12km, a DRAM 12mm, an HDD 12hd, a main monitor (first monitor) 12m1, and a sub-monitor (second monitor) 12m2.

[0039] The sub-monitor 12m2 is attached to the main monitor 12m1. The radiologist (diagnosticator) faces the main monitor 12m1 and the sub-monitor 12m2 and performs image diagnosis by operating the keyboard / mouse 12km.

[0040] The machine learning system 40 has a classifier trained on training images (a training dataset). The machine learning system 40 may include, for example, an NIH classification specialized machine learning system or a lesion location specialized machine learning system. Here, the NIH classification specialized machine learning system is a machine learning system that has been trained using deep learning on chest X-ray image data with findings, which is openly available from the National Institutes of Health in the United States, sorted into 15 categories.

[0041] The aforementioned training dataset consists of images with abnormal shadows and images without abnormal shadows associated with the images with abnormal shadows, and is used in an image diagnostic support device having a learning unit, an image output unit, an extraction unit, and a display unit, for example, in the following learning scenarios.

[0042] The learning unit uses a training dataset consisting of images with abnormal shading and images without abnormal shading associated with the images with abnormal shading to perform machine learning on a neural network. The images with abnormal shading from the training dataset are used as input data, and the images without abnormal shading associated with the images with abnormal shading are used as output data, associating the two. A sufficient number of training datasets are prepared for machine learning.

[0043] When the image output unit receives an image to be classified as input to the trained neural network, it outputs an image without abnormal shading corresponding to the image to be classified.

[0044] The extraction unit obtains and extracts the difference between the image to be identified and the output image without abnormal shadows.

[0045] The display unit displays the image to be identified, the output image without abnormal shadows, and an image obtained by visually enhancing the difference and combining it with the image to be identified or the output image without abnormal shadows, on a display (main monitor (first monitor) 12m1 or sub-monitor (second monitor) 12m2) in a comparable manner.

[0046] According to the present invention, it is possible to provide an abnormal shadow extraction method, an abnormal shadow extraction program, an abnormal shadow extraction device, and an image diagnostic support device that can improve diagnostic accuracy even with a single image, without having to take multiple images of the same location as in the conventional method. In examinations using radiation, the amount of radiation exposure to the patient can be suppressed, and in other examinations, the burden on the patient can be reduced. Furthermore, during diagnosis, abnormal shadows can be highlighted and the image indicating their location can be displayed in a way that allows comparison with the image to be identified, which is expected to prevent oversights. This will enable even inexperienced medical professionals to make diagnoses equivalent to those of experienced medical professionals. It can also be used in image diagnostic support in combination with AI or other technologies that have clinical judgment support functions. These also contribute to reducing the burden on medical professionals. [Explanation of Symbols]

[0047] 10 ...Image diagnostic support device 12pr…CPU 12km ... Keyboard / Mouse 12hd …HDD 12m1 ... Main monitor 12m2… Sub-monitor 40…Machine learning systems

Claims

1. A training data acquisition step in which an image with abnormal shadowing and an image without abnormal shadowing associated with the image with abnormal shadowing are obtained as a training dataset, A learning step in which the neural network is trained to learn by associating the two, using the aforementioned training dataset, with the aforementioned images with abnormal shadows from the training dataset as input data and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, The image output step involves inputting the image to be classified into the pre-trained neural network and outputting an image without abnormal shading corresponding to the image to be classified. An abnormal shadow extraction method characterized by performing an extraction step of obtaining and extracting the difference between the image to be identified and the output image without abnormal shadows.

2. The abnormal shadow extraction method according to claim 1, further comprising an abnormal shadow enhancement step of visually enhancing the difference and synthesizing it with the target image for discrimination or the output abnormal shadow-free image.

3. The abnormal shadow extraction method according to claim 1, wherein the training dataset uses an image set consisting of a pair of a medical image without abnormal shadows and an image with abnormal shadows obtained by combining the medical image with a simulated lesion image.

4. A training data acquisition step in which an image with abnormal shadowing and an image without abnormal shadowing associated with the image with abnormal shadowing are obtained as a training dataset, A learning step in which the neural network is trained to learn by associating the two, using the aforementioned training dataset, with the aforementioned images with abnormal shadows from the training dataset as input data and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, The image output step involves inputting the image to be classified into the pre-trained neural network and outputting an image without abnormal shading corresponding to the image to be classified. An abnormal shadow extraction program for executing an extraction step of obtaining and extracting the difference between the image to be identified and the output image without abnormal shadows.

5. The abnormal shadow extraction program according to claim 4, further comprising an abnormal shadow enhancement step of visually enhancing the difference and synthesizing it with the target image for discrimination or the output abnormal shadow-free image.

6. The abnormal shadow extraction program according to claim 4, wherein the training dataset uses an image set consisting of a pair of a medical image without abnormal shadows and an image with abnormal shadows obtained by combining the medical image with a simulated lesion image.

7. A training data acquisition unit that acquires an image with abnormal shadows and an image without abnormal shadows associated with the image with abnormal shadows as a training data set, A learning unit that uses the aforementioned training dataset, takes the aforementioned images with abnormal shadows from the training dataset as input data, and the aforementioned images without abnormal shadows associated with the aforementioned images with abnormal shadows as output data, and performs machine learning on a neural network by associating the two. When the pre-trained neural network receives an image to be classified, it outputs an image without abnormal shadows corresponding to the image to be classified. An abnormal shadow extraction device comprising: an extraction unit that acquires and extracts the difference between the image to be determined and the output image without abnormal shadows.

8. The abnormal shadow extraction device according to claim 7, further comprising an abnormal shadow enhancement unit that visually enhances the difference and synthesizes it with the target image for discrimination or the output abnormal shadow-free image.

9. The abnormal shadow extraction device according to claim 7, wherein the training dataset uses an image set consisting of a pair of a medical image without abnormal shadows and an image with abnormal shadows obtained by combining the medical image with a simulated lesion image.

10. An image diagnostic support device including an abnormal shadow extraction device according to any one of claims 7 to 9, Equipped with a display unit, The image diagnostic support device is characterized in that the display unit displays the image to be discriminated against, and an image obtained by visually enhancing the extracted difference and combining it with the image to be discriminated against or the output image without abnormal shadows, in a manner that allows for comparison.

Images