Image generation method for deep learning training, image generation program for deep learning training, and image generation apparatus for deep learning training

Abstract

This invention provides a method for generating images for deep learning training, a program for generating images for deep learning training, and a device for generating images for deep learning training, which enable the creation of large datasets by easily preparing a sufficient number of training images for training a discriminator. [Solution] The deep learning training image generation method is a method for generating images for deep learning training in which a discriminator learns to identify lesion regions in a target image when a target image is input, and the deep learning training image generation device performs the steps of: acquiring a medical image including a human body and a simulated lesion image representing a simulated lesion; and generating a deep learning training image corresponding to the target image by combining the simulated lesion image with the medical image.

Description

Technical Field

[0004] , , , , , ,

[0003] , , , ,

[0001] The present invention relates to an image generation method for deep learning training, an image generation program for deep learning training, and an image generation apparatus for deep learning training.

Background Art

[0002] In recent years, the development of technologies for assisting in the diagnosis of medical images using artificial intelligence has been progressing. For example, a trained first classification model generated by unsupervised learning using a plurality of learning medical images of normal cases, and a trained second classification model generated by supervised learning using a plurality of learning data sets including normal cases and abnormal cases are used to determine whether the body part of the examinee shown in the target medical image is normal or not. A diagnostic support device has been proposed (see, for example, Patent Document 1). Here, learning medical images of normal cases can be collected relatively inexpensively and easily through regular health checkups and the like, whereas it is difficult and costly to collect a learning data set including learning medical images of abnormal (non-normal) cases.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] AI (Artificial Intelligence) is typically used for segmenting medical images by photographing training datasets that have been segmented by physicians to identify lesions. However, since medical images such as radiographs that actually contain lesions are rare, it is difficult to easily obtain large datasets. Therefore, the present invention aims to provide a deep learning training image generation method, a deep learning training image generation program, and a deep learning training image generation device that enable the creation of a large dataset by easily preparing a sufficient number of training images for training a discriminator to identify lesion regions in input medical images using machine learning. [Means for solving the problem]

[0005] The image generation method for deep learning training according to the present invention is A deep learning training image generation method for training a discriminator that identifies lesion areas in a target image when a target image is input, The deep learning training image generation device acquires medical images including the human body and simulated lesion images representing simulated lesions. The method is characterized by performing the step of generating a deep learning training image corresponding to the target image by synthesizing the simulated lesion image with the medical image.

[0006] In the deep learning training image generation method of the present invention, the simulated lesion image is a mask image containing the signal values ​​of the simulated lesion, and the synthesis of the simulated lesion image to the medical image is preferably performed by adding the signal values ​​of the mask image that generated the simulated lesion to the signal values ​​of the medical image.

[0007] In the deep learning training image generation method of the present invention, it is preferable that the simulated lesion image has had its signal value distribution adjusted using a Gaussian filter.

[0008] In the deep learning training image generation method of the present invention, it is preferable that the method includes a step of segmenting the region in the medical image including the human body where the lesion region exists, and that the simulated lesion image is synthesized in the segmented region.

[0009] The deep learning training image generation program according to the present invention is a program for generating deep learning training images for training a discriminator that identifies lesion regions in a target image when a target image is input, The deep learning training image generation device acquires medical images including the human body and simulated lesion images representing simulated lesions. This is a deep learning training image generation program for performing the steps of generating a deep learning training image corresponding to the target image by synthesizing the simulated lesion image with the medical image.

[0010] In the deep learning training image generation program of the present invention, it is preferable that the simulated lesion image is a mask image containing the signal values ​​of the simulated lesion, and that the synthesis of the simulated lesion image to the medical image is performed by adding the signal values ​​of the mask image that generated the simulated lesion to the signal values ​​of the medical image.

[0011] In the deep learning training image generation program of the present invention, it is preferable that the simulated lesion image has had its signal value distribution adjusted using a Gaussian filter.

[0012] In the deep learning training image generation program of the present invention, it is preferable that the program includes a step of segmenting the region in the medical image including the human body where the lesion region exists, and that the simulated lesion image is synthesized in the segmented region.

[0013] The deep learning training image generation device according to the present invention is a device that generates deep learning training images for training a discriminator that identifies lesion regions in a target image when a target image is input, An image acquisition unit that acquires medical images including the human body, and simulated lesion images representing simulated lesions, The system is characterized by comprising: an image generation unit that generates a deep learning training image corresponding to the target image by synthesizing the simulated lesion image with the medical image.

[0014] In the deep learning training image generation device of the present invention, the simulated lesion image is a mask image containing the signal values ​​of the simulated lesion, and the synthesis of the simulated lesion image to the medical image is preferably performed by adding the signal values ​​of the mask image that generated the simulated lesion to the signal values ​​of the medical image.

[0015] In the deep learning training image generation device of the present invention, it is preferable that the simulated lesion image has had its signal value distribution adjusted using a Gaussian filter.

[0016] In the deep learning training image generation device of the present invention, it is preferable that the medical image including the human body is segmented in the region where the lesion region exists, and that the simulated lesion image is synthesized in the segmented region. [Effects of the Invention]

[0017] According to the present invention, it is possible to provide a deep learning training image generation method, a deep learning training image generation program, and a deep learning training image generation apparatus that enable the creation of a large dataset by easily preparing a sufficient number of training images for training a discriminator. As a result, it is possible to construct a discriminator with high detection accuracy for lesions.

[0018] The above-mentioned objectives, other objectives, features, and advantages of this invention will become even clearer from the following detailed description of examples of embodiments with reference to the drawings. [Brief explanation of the drawing]

[0019] [Figure 1] This is an example of a segmented image used to determine the location for adding simulated lesions. [Figure 2]Examples of images before and after the addition of simulated lesion images. The left figure is an image before the addition of a simulated lesion, and the right figure is an image after the addition of a simulated lesion. [Figure 3] A diagram for explaining the adjustment of the signal value distribution of simulated lesions. [Figure 4] An example of a mask image. [Figure 5] Examples of images obtained by adding different simulated lesions to the same medical image (original image). [Figure 6] An example of the flow of generating images for deep learning training. [Figure 7] Figure 7(A) is an image obtained by segmenting the lung field and adding a simulated lesion in a chest CT. Figure 7(B) is an image obtained by segmenting the whole brain and adding a simulated lesion in a head MRI. [Figure 8] A block diagram showing an example of the configuration of a medical support system to which the present invention is applied.

Embodiments for Carrying Out the Invention

[0020] The present invention will be described with examples. However, the present invention is not limited and restricted to the following examples. The drawings referred to below include those created simulatedly from the viewpoint of personal information protection. Also, the ratios of the dimensions of the objects drawn in the drawings may be different from the ratios of the dimensions of the actual objects, and the object dimension ratios may also be different between the drawings.

[0021] The image generation device for deep learning training in the present invention includes an image acquisition unit and an image generation unit. The image acquisition unit acquires a medical image (original image) including an arbitrary subject in order to generate an image for deep learning training (also referred to as a teacher data image). Also, the image acquisition unit acquires a simulated lesion image representing a simulated lesion in order to generate an image for deep learning training.

[0022] A simulated lesion image is, for example, an image representing a lesion created using computer graphics. The image generation unit generates deep learning training images for training a classifier that identifies lesion areas in a target image when a target image to be classified is input, by combining the simulated lesion image with the medical image (original image). For the purpose of training the classifier, which will be described later, the image generation unit generates multiple deep learning training images by combining the acquired medical image (original image) with simulated lesion images that have been randomly generated with varying positions, orientations, major and minor axes in the case of ellipses, and sizes. As a result, a composite deep learning training image is generated that appears as if the medical image (original image) was taken when a lesion was present.

[0023] This section explains a specific example of generating images for deep learning training to identify pulmonary nodules, using chest radiographs as an example. First, the lung field is segmented to determine where to place simulated lesions. This is because pulmonary nodules develop within the lung field. Lung field segmentation can be performed manually or automatically using AI. Figure 1 shows an example of a segmented image. The area enclosed by the white line is the segmented lung field.

[0024] Next, a simulated lesion image is synthesized into the segmented lung field. As shown in the figure, the shape and size of the lung field vary considerably, so in the deep learning training image generation method of the present invention, it is preferable to segment the medical image to include a step of segmenting the region where the lesion area exists (target organ, etc.), and to synthesize the simulated lesion image into the segmented region. The simulated lesion image is preferably a mask image containing the signal value of the simulated lesion, and the synthesis of the simulated lesion image into the medical image is preferably performed by adding the signal value of the mask image that generated the simulated lesion to the signal value of the medical image. In the portion that overlaps with the simulated lesion portion of the original image, when adding the signal value, it is preferable to blend the overlapping rather than filling it in, for example, by setting the transparency of the original medical image to 50% and the simulated lesion image to 50%.

[0025] In radiographic images, bones and other organs besides the target organ are also visible. In chest radiographs, the spine, clavicle, bronchi, heart, and blood vessels are visible, so if a lesion exists in an area overlapping with these, it may be difficult to see in the image. However, lesions can occur in these hard-to-see areas, so it is important to create deep learning training images that can be identified. Medical images contain a lot of tissue information in their signal distribution (signal value), and directly generating images by filling in simulated lesions within the image leads to a loss of tissue information. Therefore, in the overlapping areas, by reducing the transparency of both the original image and the mask image and blending them, the resulting image will still show (not disappear) bronchi, blood vessels, etc., and since it is an addition of signal values, the signal distribution of the original image can also be reflected, and no information is lost. In addition, in the images obtained in this way, the simulated lesion may be difficult for the human eye to see depending on its location, but a computer (AI) can distinguish even slight differences in signal values ​​and detect it. In signal value addition, variations in the intensity of the lesion in the resulting image can be obtained by changing the transparency ratio of both images.

[0026] Figure 2 shows examples of images before and after the addition of simulated lesion images. In the figure, the left image is the image before the addition of simulated lesions, and the right image is the image after the addition of simulated lesions. In the right image, the area circled with a white line is the area where the simulated lesion was added. Training image data is obtained by pairing the image before and after the addition of simulated lesions. Medical images of normal cases can be collected relatively inexpensively and easily through regular health checkups, etc., but images showing lesions are not readily available. However, using this method, if you have 1000 chest X-rays, you can easily obtain 1000 images × 100 sets of training data by overlaying 100 different simulated lesions onto a single chest X-ray.

[0027] For example, pulmonary nodules are generally found in radiographic images as elliptical shapes, and their direction (the major axis of the ellipse), size, density in the image, and location within the lung field occur randomly. Furthermore, actual lesions are often thick and non-homogeneous, so when they appear in medical images, their boundaries are often unclear and the image is blurred. Therefore, in order to make the simulated lesion similar to the image of an actual lesion, it is preferable to generate the major axis, minor axis, and direction randomly within the lung field, and to reproduce various patterns by adjusting the signal value distribution, adjusting the density, and blurring the edges. Figure 3 is an explanatory diagram of the adjustment of the signal value distribution of a simulated lesion. Figure 3(A) shows the location, size, and shape of the simulated lesion to be formed in the mask image. In this figure, the area to be the simulated lesion is shown as a white, homogeneous ellipse. Various patterns are reproduced using a weighted averaging filter such as a Gaussian filter to approximate the image of an actual lesion in this homogeneous elliptical area. A Gaussian filter is a filter used to smooth images. It approximates the weight distribution to a Gaussian distribution (normal distribution) based on the distance from the pixel of interest, and is effective for natural smoothing. Figure 3(B) schematically shows the blurring process using a Gaussian filter. In this way, the signal value distribution can be adjusted as shown in Figure 3(C). It is preferable to have not just one simulated lesion, but multiple variations. This is because in chest medical images, there are cases where there are 30 lesions in a single image. When generating simulated lesions, it is best to create a mask image that is anatomically consistent based on information such as the shape of the lesion in the actual lesion image, the signal value distribution, and where it is most frequently distributed within the lung field.

[0028] In deep learning training, it is crucial to train the model on a variety of patterns. In chest radiographs, the pattern of the image obtained may differ depending on the subject's body shape (thin or obese), and whether they are male or female. In obese individuals or women, breast shadows may be visible. Furthermore, the image obtained differs depending on whether or not the patient has a history of surgery for breast cancer, etc. To prevent the model from misidentifying these shadows or features derived from surgical history as lesions, it is desirable to train the model on images without lesions, and also to train it on images with simulated lesions added to these images as images with lesions. In addition, since bones are relatively clearly visible in younger individuals but may not be clearly visible in older individuals, it is important to collect chest radiographs from different age groups. Moreover, as a general trend, it is difficult to collect many images with lesions from younger individuals. However, even in populations with attributes where there are few images with lesions, according to this invention, it is possible to obtain a large amount of data on images with lesions by using simulated lesion images.

[0029] In generating mask images containing signal values ​​for simulated lesions, random numbers should be generated and the images should be generated randomly. The target organ and the size and shape of the lesions should be varied within the pattern. Because different diseases tend to occur in specific locations, mask images should be created using parameters appropriate to the target area (organ) and the examination device. Mask images can be obtained by automatically creating simulated lesions with randomized size, position (within the segmented lung field shown in Figure 1), angle, transparency, and number. An example of a mask image is shown in Figure 4. The mask image on the left has two simulated lesions slightly below the center of the left region and one near the edge slightly below the center of the right region. These three simulated lesions include those that are relatively clear white, those that are blurry and small, and those with different major and minor axes of the ellipse. The mask image on the right has three simulated lesions in the central part slightly below the center of the right region. These three simulated lesions also differ in size, transparency, angle, and major / minor axis of the ellipse. A single mask image can be used, or multiple images can be layered together. If multiple simulated lesions are to be added, a single mask image with multiple simulated lesions can be used, or multiple mask images can be layered to add the desired number and location of simulated lesions. It is also possible to layer another mask image on top of the blended result. Furthermore, if pulmonary nodules are located on the front and back sides of the lung, the oval shapes of the simulated lesions can be overlapped to correspond to this. When creating the simulated lesions, the location and number should be set according to the lungs in the case of a chest X-ray. In this way, a large number of deep learning training images with various variations in lesion location, shape, and density can be obtained.

[0030] Figure 5 shows an example of images in which different simulated lesions have been added to the same medical image (original image). In Figure 5, the areas circled with white lines are the areas to which simulated lesions have been added. These images were created by adding simulated lesions to the same medical image (original image) using different mask images. By overlaying mask images with randomly created simulated lesion areas onto the same medical image, it is possible to obtain a large number of images with different lesion locations and numbers.

[0031] Figure 6 shows a flowchart of an example of the processing procedure for generating images for deep learning training. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, depending on the embodiment, steps in the following processing procedure can be omitted, replaced, or added as appropriate.

[0032] In step S1, signal value data for the major axis, minor axis, and direction of a simulated lesion are randomly generated. Here, "random generation" does not mean completely random; rather, an AI that has learned the typical appearance and shape of lesions, such as the approximate size range for a lung nodule, may randomly create a model within the possible shapes for that disease. In step S3, a mask image is created that includes the signal values ​​of the simulated lesion with the major axis, minor axis, and direction created in step S1. In step S5, a Gaussian filter or the like is used to adjust the signal value data so that it has a signal value distribution similar to that of an actual lesion, and the adjusted mask image is obtained. In step S7, the obtained adjusted mask image is added to the original image (image before the simulated lesion is added) to obtain the image after the simulated lesion is added. The mask image has lesions randomly placed within the region identified as the lung field, and when adding them, it is preferable to adjust the transparency at a specific ratio before adding them. In step S9, the two images obtained in step S7, the image after the simulated lesion is added and the image before the simulated lesion is added, are obtained as a training dataset for deep learning.

[0033] In this invention, an AI that has learned the typical appearance and shape of a lesion can randomly create a lesion model from among the possible shapes of that disease, and then overlay it with images without abnormalities to create training data.

[0034] In image interpretation, lesions in common areas tend to be less likely to be overlooked. On the other hand, lesions in irregular locations are prone to being overlooked. However, by using a classifier trained with deep learning, it is expected that lesions in areas that humans tend to miss will not be overlooked. However, if training is done using only real data, there is a lot of data for common areas but little data for irregular areas, which may result in insufficient training. Therefore, with this invention, by generating data for irregular areas as well and training the AI ​​with this data, it is possible to construct an unbiased AI.

[0035] For training the diagnostic support device, the acquired medical images (original images) and the generated deep learning training images can be used as image sets of images with and without lesions to train the classifier. Images without lesions can be collected relatively easily through regular health checkups, and by generating images with lesions using the method described above, a sufficient number of image sets of images with and without lesions can be prepared for deep learning.

[0036] In supervised learning for deep learning training, one factor that affects the accuracy of lesion region discrimination is the number of training datasets used. That is, the more training datasets used in supervised learning, and the more diverse the training data, the higher the discrimination accuracy can be expected to be. According to the present invention, it is possible to create a large number of datasets within one's own facility.

[0037] Furthermore, the type of medical image is not limited as long as it can be used to support diagnosis. The method of the present invention can be applied to any medical image, such as radiographic images (X-rays), CT images, MRI images, and ultrasound images. The body part to be identified may be any part such as the chest, abdomen, or head, a specific organ, tissue, or a portion of the body, or the whole body.

[0038] Figure 7(A) is a chest CT image in which the lung field has been segmented and simulated lesions have been added. Figure 7(B) is a head MRI image in which the entire brain has been segmented and simulated lesions have been added. The appearance of the simulated lesions (location, size, number, etc.) differs depending on the examination device (modality) and the area being examined. For example, if there are patterns such as lesions occurring more frequently in the periphery of an organ, the settings when creating the mask image should be changed.

[0039] Here, "no lesion (normal)" may mean, for example, that a person or machine diagnoses that there is no lesion area, or that there is a high probability that there is no lesion area, and may include the case where a lesion area actually exists. On the other hand, "lesion present (abnormal)" may mean, for example, that a person or machine diagnoses that there is a lesion area, or that there is a high probability that there is a lesion area, and may include the case where a lesion area actually does not exist.

[0040] Referring to Figure 8, a medical diagnostic support system that uses training images obtained according to the present invention consists, for example, of an image diagnostic support device 10, modality 20, PACS 30, and machine learning system (analysis device) 40 connected to each other via an in-hospital LAN 50.

[0041] In the image diagnostic support device 10, the bus BS1 is connected to a communication interface 12cm, a CPU 12pr, a keyboard / mouse 12km, a DRAM 12mm, an HDD 12hd, a main monitor (first monitor) 12m1, and a sub-monitor (second monitor) 12m2.

[0042] The sub-monitor 12m2 is attached to the main monitor 12m1. The radiologist (diagnosticator) faces the main monitor 12m1 and the sub-monitor 12m2 and performs image diagnosis by operating the keyboard / mouse 12km.

[0043] The machine learning system 40 has a classifier trained on training images obtained by the present invention. The machine learning system 40 may include, for example, an NIH classification specialized machine learning system or a lesion location specialized machine learning system. Here, the NIH classification specialized machine learning system is a machine learning system that has been trained using deep learning on chest X-ray image data with findings, which is openly available from the National Institutes of Health in the United States, sorted into 15 categories.

[0044] The deep learning training images obtained by the present invention can be used in an image diagnostic support device having a learning unit, a detection unit, and a display unit, for example, in the following learning scenarios.

[0045] The learning unit trains a classifier to identify lesion regions in input medical images using training data consisting of training images, ground truth data in which lesion regions have been identified in the training images, and medical images that do not contain lesion regions. A sufficient number of training data are prepared for machine learning.

[0046] A machine learning model can be used as the discriminator. An example of a machine learning model is a neural network model. The discriminator is trained to output the probability that each pixel in the training data image is a lesion region when it is input as training data image.

[0047] The detection unit is equipped with a pre-trained classifier. When a target medical image (the image to be classified) is input to the detection unit, the detection unit detects the lesion region by having the classifier extract the lesion region contained in the target medical image.

[0048] The display unit highlights the lesion area detected by the detection unit from the medical image to be detected and displays the medical image on a display (main monitor (first monitor) 12m1 or sub-monitor (second monitor) 12m2).

[0049] According to the present invention, it is possible to provide a deep learning training image generation method and a deep learning training image generation apparatus that enable the easy preparation of a sufficient number of training images for training a discriminator and the creation of a large dataset. As a result, it is possible to construct a discriminator with high accuracy in detecting lesions, and the burden on medical professionals can be reduced.

[0050] In the above explanation, chest radiographs were mainly used as examples of medical images, but the present invention is not limited to these and can be applied to various types of medical images. [Explanation of symbols]

[0051] 10 ...Image diagnostic support device 12pr…CPU 12km ... Keyboard / Mouse 12hd …HDD 12m1 ... Main monitor 12m2… Sub-monitor 40…Machine learning systems

Claims

1. A deep learning training image generation method for training a discriminator that identifies lesion areas in a target image when a target image is input, The deep learning training image generation device acquires medical images including the human body and simulated lesion images representing simulated lesions. The steps include generating a deep learning training image corresponding to the target image by combining the simulated lesion image with the medical image, A method for generating images for deep learning training, characterized by performing the following:

2. The aforementioned simulated lesion image is a mask image containing the signal values ​​of the simulated lesion. The method for generating images for deep learning training according to claim 1, wherein the synthesis of the simulated lesion image onto the medical image is performed by adding the signal value of the mask image from which the simulated lesion was generated to the signal value of the medical image.

3. The deep learning training image generation method according to claim 1, wherein the simulated lesion image is obtained by adjusting the signal value distribution using a Gaussian filter.

4. The medical image including the human body includes a step of segmenting the region in which the lesion region exists. The image generation method for deep learning training according to claim 1, wherein the simulated lesion image is synthesized onto the segmented region.

5. A program for generating deep learning training images for training a classifier that identifies lesion areas in a target image when a target image is input, The deep learning training image generation device acquires medical images including the human body and simulated lesion images representing simulated lesions. A deep learning training image generation program for performing the steps of generating a deep learning training image corresponding to the target image by synthesizing the simulated lesion image with the medical image.

6. The aforementioned simulated lesion image is a mask image containing the signal values ​​of the simulated lesion. The deep learning training image generation program according to claim 5, wherein the synthesis of the simulated lesion image onto the medical image is performed by adding the signal value of the mask image that generated the simulated lesion to the signal value of the medical image.

7. The deep learning training image generation program according to claim 5, wherein the simulated lesion image has had its signal value distribution adjusted using a Gaussian filter.

8. The medical image including the human body includes a step of segmenting the region in which the lesion region exists. The deep learning training image generation program according to claim 5, wherein the simulated lesion image is synthesized onto the segmented region.

9. A device that generates deep learning training images for training a discriminator that identifies lesion areas in a target image when a target image is input, An image acquisition unit that acquires medical images including the human body, and simulated lesion images representing simulated lesions, A deep learning training image generation device comprising: an image generation unit that generates a deep learning training image corresponding to the target image by synthesizing the simulated lesion image with the medical image.

10. The aforementioned simulated lesion image is a mask image containing the signal values ​​of the simulated lesion. The deep learning training image generation apparatus according to claim 9, wherein the synthesis of the simulated lesion image onto the medical image is performed by adding the signal value of the mask image from which the simulated lesion was generated to the signal value of the medical image.

11. The deep learning training image generation apparatus according to claim 9, wherein the simulated lesion image has had its signal value distribution adjusted using a Gaussian filter.

12. The deep learning training image generation apparatus according to claim 9, wherein the medical image including the human body is segmented in the region where the lesion region exists, and the simulated lesion image is synthesized in the segmented region.

Images