Program for indicating hunner lesion, learned model, and method for generating same

Abstract

The present disclosure relates to a method for generating a learning model, a learned model, a program, and a controller of a bladder endoscope in which the program or the model is recorded. The method including: acquiring, as teaching data, endoscope image data on a Hunner lesion in a bladder; and generating the learning model by using the teaching data such that a bladder endoscope image serves as an input and the position indication of a Hunner lesion in the bladder endoscope image serves as an output. The program causing a computer to perform acquiring endoscope image data on a Hunner lesion in a bladder, inputting a target bladder endoscope image to a learning model in which a bladder endoscope image serves as an input and position-indication data on a Hunner lesion in an endoscope image serves as an output, and outputting the position indication of the Hunner lesion.

Description

Program for identifying Hanna lesions, trained model and method for generating same

[0001] The present invention relates to a program useful for identifying Hannah lesions, and more particularly to a program / trained model for identifying Hannah lesions in a subject, useful in the field of bladder abnormalities, particularly interstitial cystitis, and a method for generating the same.

[0002] Interstitial cystitis (IC) is a chronic disease characterized by symptoms such as frequent urination, urgency, and bladder pain and discomfort when the bladder is full. In severe cases, patients may urinate as many as 60 times per day, significantly impacting daily life to the point of difficulty. It is more common in women than men, with over 1 million of the approximately 1.3 million IC patients in the United States being women. However, despite numerous epidemiological studies, the cause remains unknown. Furthermore, the definition, diagnostic criteria, and even terminology of "IC" vary by country and region. Therefore, the term "IC" in this application encompasses the concepts of bladder pain syndrome and overactive bladder pain syndrome.

[0003] The criteria for interstitial cystitis routinely used in the United States include the Interstitial Cystitis Data Base (ICDB)18 criteria, which do not require cystoscopic findings. The National Institute of Diabets, Digestives, and Kidney Diseases (NIDDK) criteria, which are often cited, are more stringent because they require cystoscopic findings and are used for strict case selection in research. Reports suggest that fewer than half of patients diagnosed with interstitial cystitis according to the ICDB criteria meet the NIDDK criteria.

[0004] Another characteristic feature of interstitial cystitis is that it can be broadly divided into Hanna type, which has Hanna lesions, and non-Hanna type, which does not. Hanna lesions are characteristic reddened mucosa lacking normal capillary structures. Pathologically, the epithelium is often exfoliated (eroded), and the submucosal tissue exhibits proliferation of new blood vessels and clusters of inflammatory cells. Hanna type has clear abnormal findings both endoscopically and pathologically, and is a characteristic reddened mucosa lacking normal capillary structures. As mentioned above, because international standards have not been established, Hanna lesions are sometimes referred to as Hanna ulcers or simply ulcers in some regions.

[0005] Because the Hannah type is symptomatically more severe, earlier and more accurate diagnosis and treatment are required. However, as mentioned above, there are regions where the presence or absence of Hannah lesions is not considered a prerequisite for diagnosing interstitial cystitis. Furthermore, because global standard definitions and diagnostic criteria have not yet been established, there are very few physicians who have accurate knowledge of and can diagnose interstitial cystitis and Hannah lesions. Therefore, despite the existence of U.S. Patent No. 8,080,185, disclosed by Tomohiro Ueda, one of the inventors of the present invention, potential patients are still being overlooked, resulting in problems such as inadequate diagnosis and inadequate treatment, including misdiagnosis.

[0006] One factor that complicates the above-mentioned problem is that in some regions, cystoscopic findings are not necessarily required for the diagnosis of interstitial cystitis, and therefore cystoscopic examinations are often not performed. In such cases, accurate identification of Hannah lesions is difficult, and potential patients may be overlooked, as mentioned above. Even when cystoscopic examinations are performed, they are often overlooked due to a lack of physician knowledge. Furthermore, cystoscopic examinations are mostly performed on patients who have a bladder abnormality. As a result, there are very few opportunities to observe normal bladders (bladders in healthy individuals without disease) and very few cystoscopic images of normal bladders. Not only that, some physicians lack the knowledge to distinguish between normal bladders and abnormal bladders without Hannah lesions. These circumstances create a vicious cycle that makes it even more difficult to establish global standard definitions and diagnostic criteria.

[0007] However, if a versatile technique were provided that could accurately and quickly identify a patient's Hannah lesion, regardless of the physician's knowledge or level of expertise, and if a technique were provided that could properly recognize a normal bladder, significant progress would be made in the accurate diagnosis of interstitial cystitis, relief for patients with interstitial cystitis, and the formation of an international consensus.

[0008] In order to solve the above problems, the present invention comprises a learning model generation method that acquires endoscopic image data of Hannah lesions in the bladder as training data, inputs a cystoscope image using the training data, and generates a learning model that outputs the location of Hannah lesions in the cystoscope image.

[0009] The present invention also comprises a program that causes a computer to execute a process of acquiring endoscopic image data of a Hannah lesion in the bladder, inputting a target cystoscope image into a learning model that inputs cystoscope images and outputs location indication data for Hannah lesions in the endoscopic images, and outputting the location indication data for Hannah lesions.

[0010] The present invention also provides a trained model using cystoscope images acquired using a cystoscope system, comprising an input layer to which the cystoscope images are input, an output layer that outputs data indicating the location of Hannah lesions in the endoscopic images, and an intermediate layer in which parameters are trained using training data that inputs endoscopic image data of Hannah lesions in the bladder and outputs data indicating the location of Hannah lesions in the bladder images, and the trained model causes a computer to function by inputting a target cystoscope image to the input layer, performing calculations in the intermediate layer, and outputting data indicating the location of Hannah lesions in the image.

[0011] Furthermore, it is preferable that the above-mentioned learning model generation method, program, and learned model include as training data at least one of endoscopic image data of air present in the bladder, endoscopic image data of a normal bladder, and endoscopic image data of a bladder that does not contain Hannah lesions but is not a normal bladder.

[0012] It is also preferable that the image data include both narrow-band light observation images and white-light observation images.

[0013] It is also preferable that the device is capable of determining whether the bladder is a normal bladder and outputting the result, and that the program or trained model is installed in the control device of the cystoscope.

[0014] The program and trained model of the present invention can be implemented by employing a known configuration, such as a control device or server equipped with a CPU, GPU, ROM, RAM, a communication interface, etc. It can also be configured as a cloud-based system. Furthermore, the present invention preferably includes a means for indicating and displaying the estimated Hannah lesion location on a display or other display means as a visually recognizable means, such as a frame or coloring. Furthermore, the present invention may be used as a system independent of an endoscope, or may be installed in a control device of a cystoscope system and used in real time simultaneously with intravesical observation.

[0015] In addition, in the present invention, a deep learning model is used, typically deep learning using a neural network, preferably a convolutional neural network. When an image is input, the convolutional neural network functions as an estimation means for estimating the location of a Hanna lesion. However, the present invention is not limited to the above as long as the effects of the present invention can be achieved.

[0016] Deep learning models have a large number of internal parameters. These internal parameters are adjusted to obtain output results for input data that are as close as possible to the training data. This adjustment process is generally called "learning." In order to generate a high-performance model, the model structure and training method as well as the quantity and quality of the training data (the set of input data and training data) used for training are important.

[0017] In the present invention, endoscopic images of Hannah lesions in the bladder are used as training data. A model capable of identifying Hannah lesions is generated by identifying and training these Hannah lesions. Research has revealed that air (bubbles) present in the bladder may be recognized as Hannah lesions, and that Hannah lesions may be identified in normal bladders (presumably due to the influence of wrinkle shadows and elevation differences in the normal bladder). Therefore, to avoid identifying these elements as Hannah lesions, air (bubbles) are treated as air (bubbles). Furthermore, because the epidermal condition of a normal bladder differs from the bladder condition of an interstitial cystitis patient, regardless of whether or not a Hannah lesion is present, the model is trained not only on images of normal bladders but also on images of bladders that do not contain Hannah lesions but are not normal.

[0018] In this invention, by training a normal bladder, it is effective in avoiding false positive judgments for normal bladders. Furthermore, by training images of bladders that do not have Hannah lesions but are not normal, it is effective in avoiding false positive judgments for abnormal bladders that do not have Hannah lesions. Furthermore, if a model is generated that outputs a normal bladder, it becomes possible to realize a configuration that judges and indicates an abnormal bladder that is neither a normal bladder nor a bladder with Hannah lesions.

[0019] Furthermore, to obtain a highly accurate algorithm, it is necessary to train the system on numerous morphologies of Hannah lesions. However, the global database of cystoscopic images of interstitial cystitis patients is extremely limited. However, the inventor, Tomohiro Ueda, has accumulated the largest number of cystoscopic images of interstitial cystitis patients in the world (as well as images of normal bladders and images of non-normal bladders without Hannah lesions). This allows for a wide variety of Hannah lesion images to be trained, enabling a sufficient number of images of sufficient quality to create a model. Furthermore, the present invention trains the system on images obtained using both narrowband imaging (NBI) and white light imaging (WLI), resulting in a model that can be used with both types of images. Narrowband imaging uses narrowband light of two wavelengths: blue (wavelengths of 390-445 nm) and green (wavelengths of 530-550 nm), enhancing the contrast of microvascular images before output. Blue light indicates the presence or absence of neovascularization on the mucosal surface, while green light indicates the presence or absence of blood vessels deep within the mucosa. White light observation is performed using illumination light synthesized from the three primary colors of blue, green, and red emitted from the tip of the endoscope. While narrowband light observation makes it easier for doctors to visualize lesions in actual diagnoses, it is expected that white light observation is still used more frequently worldwide, including in developing countries. While white light observation makes diagnosis difficult because both the lesion and the background appear red, the present invention is applicable to white light observation images, making it highly versatile and useful for both identifying patients with interstitial cystitis and building an international consensus.

[0020] Examples of WLI and NBI images of normal, bubbles (present), and Hanna lesions. Examples of IoU scores. Conceptual diagram of TP, FP, and FN. Examples of NBI image prediction results from the detection model (Cascade R-CNN). Examples of NBI image prediction results from the detection model and segmentation model. Examples of WLI image prediction results from the detection model and segmentation model. Examples of detection results for NBI normal bladder images from the detection model and segmentation model.

[0021] Dataset Based on a video taken with a cystoscope, annotation of this video was performed using the following procedure: 1. Candidate images were extracted every 10 frames from all frames, and candidate images of Hanna lesions were selected. 2. Annotation of the selected images was added. 3. Annotation of similar images in the frames before and after the image to which annotation was added was added.

[0022] The data with the correct answer information was divided into NBI and WLI, and a database of still images was created. The extracted images were then trimmed to remove the black areas outside the endoscopic image. This process resulted in an image size of approximately 1000 x 900 pixels. Annotation of the Hanna lesion location was performed using software called Label me.

[0023] Number of videos

[0024] Still image database configuration *Normal images include images of normal bladders and images of bladders without Hannah lesions but not normal bladders, but do not include images with Hannah lesions. Bubbles include both normal bladders and those with Hannah lesions.

[0025] Examples of WLI and NBI images of normal, bubbles (present), and Hanna lesions are shown in Figure 1. The bubble image in Figure 1 is an example of an image without Hanna lesions but not a normal bladder. Furthermore, all of the original images are color images, and the difference in visibility between WLI and NBI can be seen, demonstrating the advantages of NBI.

[0026] Models Experiments were conducted using a detection model and a segmentation model. The detection model is a model that estimates a rectangular area that contains the Hanna lesion area, and outputs the position, size, and lesion confidence of the rectangular area. The segmentation model is a model that outputs the lesion confidence for each pixel, and estimates the Hanna lesion area including its shape.

[0027] For the experiments, we used the following models, which have shown high performance on general image datasets (COCO, CITYSCAPES): Detection model: Cascade R-CNN Segmentation model: Cascade Mask R-CNN Segmentation model: OCNet We trained the above three models with NBI and WLI data, respectively, to create a total of six models.

[0028] Experimental Settings and Results In the experiment, the dataset was randomly divided into 85% training data and 15% test data five times, and training and evaluation were performed five times. The data was divided on a case-by-case basis. Tables 3 and 4 show the number of images and cases in the NBI and WLI datasets, respectively.

[0029] Number of images and cases in the NBI dataset

[0030] Number of images and cases in the WLI dataset

[0031] Three models (Cascade R-CNN, Cascade Mask R-CNN, and OCNET) were trained for each image type and each split dataset, and model performance was evaluated using sensitivity and positive predictive value (PPV) per Hanna lesion area. Sensitivity and positive predictive value (PPV) are defined as follows: Sensitivity = #TP / (#TP + #FN) PPV = #TP / (#TP + #FP) where #TP is the number of true positives, #FN is the number of false negatives, and #FP is the number of false positives.

[0032] Next, we will explain how to determine true positives (TP), false negatives (FN), and false positives (FP) for each Hanna lesion area. First, the degree of overlap between the predicted area and the correct answer area is calculated using IoU (Intersection over Union). IoU is the ratio of the number of overlapping pixels between the predicted area and the correct answer area to the number of pixels in the union of the predicted area and the correct answer area. If the two areas do not overlap at all, the score is 0, and if they completely match, the score is 1. Figure 2 shows an example of an IoU score for a rectangular area. The dotted line indicates the correct answer area, and the solid line indicates the predicted area.

[0033] In this embodiment, TP, FN, and FP are defined as follows: TP ≡ Correct answer region where IoU with all prediction regions overlapping by one pixel or more exceeds 0.3 FN ≡ Correct answer region where IoU with all prediction regions overlapping by one pixel or more is 0.3 or less FP ≡ Prediction region where IoU with the correct answer region is 0.1 or less

[0034] The detection model, Cascade R-CNN, evaluates rectangular regions as described above, while the segmentation models, Cascade Mask R-CNN and OCNET, also consider the region shape for evaluation. The model consists of a ground truth region and a predicted region. As shown in Figure 3, the ground truth region (left) at the top and the predicted region have little overlap, so they are classified as FN and FP. The ground truth region (center) at the bottom has sufficient overlap with multiple predicted regions, so they are classified as TP.

[0035] A conceptual diagram of the above TP, FP, and FN is shown in FIG.

[0036] Tables 5 to 8 show the evaluation results for each model. Figures 4 to 7 show examples of NBI image prediction results for each model. Figure 6 shows examples of detection results for NBI normal bladder images using the detection model and segmentation model. Note that the original images in all of Figures 4 to 7 are color photographs. Table 5: Performance of the detection model (Cascade R-CNN) evaluated with NBI images Table 6: Performance of the segmentation model evaluated with NBI images Table 7: Performance of the detection model (Cascade R-CNN) evaluated with WLI images Table 8: Performance of the segmentation model evaluated with WLI images

[0037] Performance of the detection model (Cascade R-CNN) evaluated on NBI images

[0038] Performance of segmentation models evaluated on NBI images *Evaluated on all predicted pixels where the model responded (pixels with a confidence level other than 0%)

[0039] Performance of the detection model (Cascade R-CNN) evaluated on WLI images

[0040] Performance of the segmentation model evaluated on WLI images *Evaluated on all predicted pixels where the model responded (pixels with a confidence level other than 0%)

[0041] Figure 4 shows an example of the NBI image prediction results of the detection model (Cascade R-CNN). All predicted regions with a confidence level higher than 0% are displayed. The dotted line (green in the original image) indicates the correct region, and the solid line (red in the original image) indicates the predicted region, demonstrating sufficient accuracy in identifying Hunnner lesions.

[0042] Figure 5 shows examples of NBI image prediction results from the detection model and segmentation model. In addition to the region designation, rectangular output is displayed along with the confidence level. The top row, from left to right, indicates a Hunners lesion with a confidence level of 0.78 (left), 0.9 (center), and 0.97 (right). The bottom row, from left to right, indicates a Hunners lesion with a confidence level of 0.33 and a bubble with a confidence level of 0.79 (left), a bubble with a confidence level of 0.97 (center), a Hunners lesion with a confidence level of 0.98, and a bubble with a confidence level of 0.4. In this example, bubbles are output and displayed as bubbles, successfully distinguishing them from Hunners lesions. Note that the correct region is indicated in green in the original image.

[0043] Figure 6 shows an example of the WLI image prediction results of the detection model and segmentation model. In addition to the region designation, the rectangular output is displayed along with the confidence level. From left to right, the Hunners lesions were identified with confidence levels of 0.96 and 0.99 (left), 0.97 (center), and 0.997 (right). Therefore, the Hunners lesions were identified successfully. Note that the correct regions are shown in green in the original image.

[0044] Examples of detection results for NBI normal bladder images using the detection model and segmentation model are shown in Figure 7. In these normal bladders, no Hunner lesions were detected, and false positives were avoided even for bladders with different image appearances.

[0045] As described above, we have created a trained model applicable to both narrowband and white light observation images. Furthermore, it is now possible to identify bubbles. This invention provides physicians with the opportunity to accurately and quickly identify Hannah lesions in both narrowband and white light observation images, regardless of their level of knowledge or expertise. Furthermore, even when a normal bladder exhibits reddish phases due to height differences, shadows, etc., false positives are avoided. This contributes to the accurate diagnosis of interstitial cystitis, relief for interstitial cystitis patients, and the formation of an international consensus.

Claims

1. A learning model generation method comprising: acquiring endoscopic image data of Hannah lesions in the bladder as training data; and using the training data to generate a learning model that inputs a bladder endoscopy image and outputs the location of Hannah lesions in the bladder endoscopy image.

2. The learning model generating method according to claim 1, further comprising endoscopic image data of air present in the bladder as training data.

3. A learning model generating method according to claim 1 or 2, further comprising endoscopic image data of a normal bladder and endoscopic image data of a bladder that is not normal but does not contain a Hanna lesion as training data.

4. A learning model generating method according to any one of claims 1 to 3, wherein the endoscopic image data in the training data includes both narrowband light observation images and white light observation images.

5. The learning model generating method according to claim 4, further comprising, as an output, a determination as to whether the bladder in the input cystoscope is a normal bladder.

6. A program that causes a computer to execute a process of acquiring endoscopic image data of Hannah lesions in the bladder, inputting a target bladder endoscopic image into a learning model that inputs a bladder endoscopic image and outputs location data of Hannah lesions in the endoscopic image, and outputting the location of Hannah lesions.

7. The program according to claim 6, wherein the endoscopic image data in the training data includes both narrowband light observation images and white light observation images.

8. The program according to claim 6 or 7, wherein the learning model further includes, as training data, endoscopic image data of a normal bladder in which no Hannah lesions are present, and endoscopic image data of a non-normal bladder in which no Hannah lesions are present.

9. The program according to any one of claims 6 to 8, further comprising endoscopic image data of air present in the bladder as training data.

10. The program according to claim 7 or 9, further comprising a process for determining whether or not the bladder is normal and outputting the result.

11. A trained model using bladder endoscopy images acquired using a bladder endoscopy system, comprising: an input layer to which the bladder endoscopy images are input; an output layer which outputs data indicating the location of Hanna lesions in the endoscopic images; and an intermediate layer in which parameters are trained using teacher data which inputs endoscopic image data of Hanna lesions in the bladder and outputs data indicating the location of Hanna lesions in the bladder images, said trained model for causing a computer to function by inputting a target bladder endoscopy image to the input layer, performing calculations in the intermediate layer, and outputting data indicating the location of Hanna lesions in the image.

12. The trained model of claim 11, further comprising endoscopic image data of air present in the bladder as training data.

13. The trained model according to claim 11 or 12, further comprising, as training data, endoscopic image data of a normal bladder and endoscopic image data of a bladder that is not free of Hanna lesions but is not a normal bladder.

14. A trained model according to any one of claims 11 to 13, wherein the endoscopic image data in the training data includes both narrowband light observation images and white light observation images.

15. The trained model of claim 13 or 14, further comprising as an output whether the bladder is normal or abnormal.

16. A control device for a cystoscope, having the program according to any one of claims 6 to 10 recorded thereon.

17. A control device for a bladder endoscope, on which a trained model according to any one of claims 11 to 15 is recorded.