A program for identifying Hannah lesions

Abstract

To provide a versatile technique in accurately and quickly understanding Hunner lesion of a patient without relying on a doctor's knowledge and experience level, and to provide a technique which is obviously advanced in an accurate diagnosis of interstitial cystitis, a remedy for a patient suffering from the interstitial cystitis, and an international agreement.SOLUTION: A program causes a computer to execute a process of outputting a presence / absence of Hunner lesion or if the patient has Hunner-type interstitial cystitis or not, by inputting a target bladder endoscope image in a learning model which acquires an endoscope image data of Hunner lesion in a bladder as training data, inputs bladder endoscope image, and outputs presence / absence of Hunner lesion in the endoscope image.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to a program useful for identifying Hannah lesions, and more particularly to a program / trained model for identifying Hannah lesions in a subject, useful in the field of bladder abnormalities, particularly interstitial cystitis, and a method for generating the same. [Background technology]

[0002] Interstitial cystitis (IC) is a chronic disease characterized by symptoms such as frequent urination, urgency, and bladder pain and discomfort when the bladder is full. In severe cases, patients may urinate as many as 60 times a day, significantly impacting daily life to the point of difficulty. It is more common in women than men, with over 1 million of the approximately 1.3 million IC patients in the United States being women. However, despite numerous epidemiological studies, the cause remains unknown. Furthermore, the definition, diagnostic criteria, and even terminology of "IC" vary by country and region. Therefore, the term "IC" in this application encompasses the concepts of bladder pain syndrome and overactive bladder pain syndrome.

[0003] The criteria for interstitial cystitis routinely used in the United States are those of the Interstitial Cystitis Data Base (ICDB), a case series. 18 These criteria do not require cystoscopic findings. The National Institute of Diabets, Digestives, and Kidney Diseases (NIDDK) criteria, often cited, are more stringent because they require cystoscopic findings and are used for strict case selection in research. Reports suggest that fewer than half of patients diagnosed with interstitial cystitis according to the ICDB criteria meet the NIDDK criteria.

[0004] Another characteristic feature of interstitial cystitis is that it can be broadly divided into Hanna type, which has Hanna lesions, and non-Hanna type, which does not. Hanna lesions are characteristic reddened mucosa that lacks a normal capillary structure. Pathologically, the epithelium is often peeled off (eroded), and the submucosal tissue exhibits proliferation of new blood vessels and clusters of inflammatory cells. Hanna type has clear abnormal findings both endoscopically and pathologically, and is a characteristic reddened mucosa that lacks a normal capillary structure. As mentioned above, because no international standards have been established, Hanna lesions are sometimes referred to as Hanna ulcers or simply ulcers in some regions.

[0005] Because the Hannah type is symptomatically more severe, earlier and more accurate diagnosis and treatment are required. However, as mentioned above, there are regions where the presence or absence of Hannah lesions is not considered a prerequisite for diagnosing interstitial cystitis. Furthermore, because global standard definitions and diagnostic criteria have not yet been established, there are very few physicians who have accurate knowledge of and can diagnose interstitial cystitis and Hannah lesions. Therefore, despite the existence of U.S. Patent No. 8,080,185, disclosed by Tomohiro Ueda, one of the inventors, regarding the diagnosis of interstitial cystitis, potential patients are still being overlooked, resulting in problems such as inadequate diagnosis and treatment, including misdiagnosis. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0006] One factor that complicates the above-mentioned problem is that in some regions, cystoscopic findings are not necessarily required for the diagnosis of interstitial cystitis, and therefore cystoscopic examinations are often not performed. In such cases, accurate identification of Hannah lesions becomes difficult, and potential patients may be overlooked, as mentioned above. Even when cystoscopic examinations are performed, they are often overlooked due to a lack of physician knowledge. Furthermore, cystoscopic examinations are almost always performed on patients who have a bladder abnormality. As a result, there are very few opportunities to observe normal bladders (bladders in healthy individuals without disease), and there are very few cystoscopic images of normal bladders. Not only that, but some physicians lack the knowledge to distinguish between normal bladders and abnormal bladders that do not contain Hannah lesions. These circumstances create a vicious cycle that makes it even more difficult to establish global standard definitions and diagnostic criteria.

[0007] However, if a versatile technique were provided that could accurately and quickly identify a patient's Hannah lesion, regardless of the physician's knowledge or level of expertise, and if a technique were provided that could properly recognize a normal bladder, significant progress would be made in the accurate diagnosis of interstitial cystitis, relief for patients with interstitial cystitis, and the formation of an international consensus. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention comprises a learning model generation method that acquires endoscopic image data of Hannah lesions in the bladder as training data, inputs a cystoscope image using the training data, and generates a learning model that outputs the location of Hannah lesions in the cystoscope image.

[0009] The present invention also comprises a program that causes a computer to execute a process of acquiring endoscopic image data of a Hannah lesion in the bladder, inputting a target cystoscope image into a learning model that inputs cystoscope images and outputs location indication data for Hannah lesions in the endoscopic images, and outputting the location indication data for Hannah lesions.

[0010] The present invention also provides a trained model using cystoscope images acquired using a cystoscope system, comprising an input layer to which the cystoscope images are input, an output layer that outputs data indicating the location of Hannah lesions in the endoscopic images, and an intermediate layer in which parameters are trained using training data that inputs endoscopic image data of Hannah lesions in the bladder and outputs data indicating the location of Hannah lesions in the bladder images, and the trained model causes a computer to function by inputting a target cystoscope image to the input layer, performing calculations in the intermediate layer, and outputting data indicating the location of Hannah lesions in the image.

[0011] Furthermore, it is preferable that the above-mentioned learning model generation method, program, and learned model include as training data at least one of endoscopic image data of air present in the bladder, endoscopic image data of a normal bladder, and endoscopic image data of a bladder that does not contain Hannah lesions but is not a normal bladder.

[0012] It is also preferable that the image data include both narrow-band light observation images and white-light observation images.

[0013] It is also preferable that the device is capable of determining whether the bladder is a normal bladder and outputting the result, and that the program or trained model is installed in the control device of the cystoscope.

[0014] The program and trained model of the present invention can be implemented by employing a known configuration, such as a control device or server equipped with a CPU, GPU, ROM, RAM, a communication interface, etc. It can also be configured as a cloud-based system. Furthermore, the present invention preferably includes a means for indicating and displaying the estimated Hannah lesion location as a visually recognizable means, such as a frame or coloring, on a display or other display means. Furthermore, the present invention may be used as a system independent of an endoscope, or may be installed in a control device of a cystoscope system and used in real time simultaneously with intravesical observation.

[0015] In addition, in the present invention, a deep learning model is used, typically deep learning using a neural network, preferably a convolutional neural network. When an image is input, the convolutional neural network functions as an estimation means for estimating the location of a Hanna lesion. However, the present invention is not limited to the above as long as the effects of the present invention can be achieved.

[0016] Deep learning models have a large number of internal parameters. These internal parameters are adjusted to obtain output results for input data that are as close as possible to the training data. This adjustment process is generally called "learning." In order to generate a high-performance model, in addition to the model structure and training method, the quantity and quality of the training data (the set of input data and training data) used for training are also important.

[0017] In the present invention, endoscopic images of Hannah lesions in the bladder are used as training data. A model capable of identifying Hannah lesions is generated by identifying and training these Hannah lesions. Research has revealed that air (bubbles) present in the bladder may be recognized as a Hannah lesion, and that a normal bladder may be identified as a Hannah lesion (presumably due to the influence of the shadows of wrinkles and elevation differences in the normal bladder). Therefore, in order to avoid identifying these elements as Hannah lesions, air (bubbles) are treated as air (bubbles). Furthermore, because the epidermal condition of a normal bladder differs from the bladder condition of an interstitial cystitis patient, regardless of whether or not a Hannah lesion is present, the model is trained not only on images of normal bladders but also on images of bladders that do not contain Hannah lesions but are not normal.

[0018] In this invention, by training a normal bladder, it is effective in avoiding false positive judgments for normal bladders. Furthermore, by training images of bladders that do not have Hannah lesions but are not normal, it is effective in avoiding false positive judgments for abnormal bladders that do not have Hannah lesions. Furthermore, if a model is generated that outputs a normal bladder, it becomes possible to realize a configuration that judges and indicates an abnormal bladder that is neither a normal bladder nor a bladder with Hannah lesions.

[0019] Furthermore, to obtain a highly accurate algorithm, it is necessary to train the system on numerous Hannah lesion patterns. However, the global database of cystoscopic images of interstitial cystitis patients is extremely limited. However, the inventor, Tomohiro Ueda, has accumulated the largest number of cystoscopic images of interstitial cystitis patients in the world (as well as images of normal bladders and images of non-normal bladders without Hannah lesions). This allows for a wide variety of Hannah lesion images to be trained, enabling a sufficient number of images of sufficient quality to create a model. Furthermore, the present invention trains the system on images obtained using both narrowband imaging (NBI) and white light imaging (WLI), resulting in a model that can be used with both types of images. Narrowband imaging uses narrowband light with two wavelengths: blue (wavelengths of 390-445 nm) and green (wavelengths of 530-550 nm), which enhances the contrast of microvascular images before output. Blue light indicates the presence or absence of neovascularization on the mucosal surface, while green light indicates the presence or absence of blood vessels deep within the mucosa. White light imaging is performed using illumination light from the tip of the endoscope, which is a composite of the three primary colors: blue, green, and red. In actual diagnoses by physicians, narrowband imaging makes lesions easier to visualize, but it is expected that white light imaging is still more widely used worldwide, including in developing countries. One factor that makes diagnosis difficult with white light imaging is that both the lesion and the background appear red. However, since the present invention can be applied to white light imaging, it is highly versatile and useful for both identifying patients with interstitial cystitis and building an international consensus. DETAILED DESCRIPTION OF THE INVENTION

[0020] Dataset Based on the video taken with a cystoscope, the correct information was added (annotated) to the video using the following procedure. 1. Candidate images are extracted every 10 frames from all frames, and Hanna lesion candidate images are selected. 2. Add correct information to the selected image 3. Add correct information to similar images in the frames before and after the image to which correct information has been added.

[0021] The data with correct answer information was divided into NBI and WLI, and a database of still images was created. The extracted images were then cropped to remove the black areas outside the endoscopic image. This process resulted in an image size of approximately 1000 x 900 pixels. Annotation of the Hanna lesion location was performed using software called Label me.

[0022] Number of videos [Table 1]

[0023] Still image database configuration [Table 2] *Normal images include images of normal bladders and images of bladder without Hannah lesions but not normal bladders, but do not include images with Hannah lesions. The bubbles include both normal bladders and those with Hannah lesions.

[0024] Image example JPEG0007752348000003.jpg80112*The image of the bubbles above is an example of a bladder that does not have Hanna's disease but is not a normal bladder. *The original images are all color images.

[0025] Model Experiments were conducted using a detection model and a segmentation model. The detection model estimates a rectangular region encompassing the Hanna lesion region, outputting the position, size, and lesion confidence of the rectangular region. The segmentation model outputs a lesion confidence for each pixel, and estimates the Hanna lesion region, including its shape.

[0026] For the experiments, we used the following models, which have shown high performance on general image datasets (COCO, CITYSCAPES). Detection model: Cascade R-CNN Segmentation model: Cascade Mask R-CNN Segmentation model: OCNet The above three models were trained using the NBI and WLI data, respectively, to create a total of six models.

[0027] Experimental setup and results In the experiment, the dataset was randomly divided five times into 85% training data and 15% test data, and training and evaluation were performed five times. The data was divided on a case-by-case basis. Tables 3 and 4 show the number of images and cases in the NBI and WLI datasets, respectively.

[0028] Number of images and cases in the NBI dataset [Table 3]

[0029] Number of images and cases in the WLI dataset [Table 4]

[0030] Three models (Cascade R-CNN, Cascade Mask R-CNN, and OCNET) were trained for each image type and each split dataset, and the model performance was evaluated in terms of sensitivity and positive predictive value per Hanna lesion area. Sensitivity and positive predictive value (PPV) were calculated as the number of true positive areas (#TP), number of false negative areas (#FN), and number of false positive areas (#FP). Sensitivity = #TP / (#TP + #FN) PPV = #TP / (#TP+#FP) is.

[0031] Next, we will explain how to determine true positives (TP), false negatives (FN), and false positives (FP) for each Hanna lesion area. First, the degree of overlap between the predicted area and the correct area is calculated using IoU (Intersection over Union). IoU is the ratio of the number of overlapping pixels between the predicted area and the correct area to the number of pixels in the union of the predicted area and the correct area; if the two areas do not overlap at all, it is 0, and if they match perfectly, it is 1. Below is an example of an IoU score for a rectangular area.

[0032] Example of IoU score (light lines indicate correct regions, dark lines indicate predicted regions) JPEG0007752348000006.jpg37105

[0033] In this example, TP, FN, and FP are defined as follows: A conceptual diagram is shown in FIG. TP ≡ Correct region with IoU of more than 0.3 with all predicted regions overlapping by 1 pixel or more FN ≡ Correct region with IoU of 0.3 or less with all predicted regions that overlap with 1 pixel or more FP ≡ Predicted region with IoU of 0.1 or less with the correct region

[0034] Note that the detection model, Cascade R-CNN, was evaluated using rectangular regions as described above, while the segmentation models, Cascade Mask R-CNN and OCNET, were evaluated taking region shape into consideration as shown below. The correct region is shown in light color, and the predicted region in dark color. The correct region and predicted region at the top have little overlap, so are FN and FP, respectively. The correct region at the bottom has sufficient overlap with multiple predicted regions, so is TP.

[0035] Conceptual diagram of TP, FP, and FN JPEG0007752348000007.jpg6375

[0036] Tables 5 to 8 show the evaluation results of each model. Tables 9 to 11 show examples of NBI image prediction results for each model. Table 11 shows examples of detection results for NBI normal bladder images in the detection model and segmentation model. Table 5: Performance of the detection model (Cascade R-CNN) evaluated on NBI images. Table 6: Performance of segmentation models evaluated on NBI images Table 7: Performance of the detection model (Cascade R-CNN) evaluated on WLI images Table 8: Performance of segmentation models evaluated on WLI images

[0037] Performance of the detection model (Cascade R-CNN) evaluated using NBI images [Table 5]

[0038] Performance of segmentation models evaluated with NBI images [Table 6] *Evaluated on all predicted pixels where the model responded (pixels with a confidence level other than 0%)

[0039] Performance of the detection model (Cascade R-CNN) evaluated on WLI images [Table 7]

[0040] Performance of segmentation models evaluated on WLI images [Table 8] *Evaluated on all predicted pixels where the model responded (pixels with confidence levels other than 0%)

[0041] Example of NBI image prediction results for the detection model (Cascade R-CNN) [Table 9] *All prediction regions with non-0% confidence are shown. The light lines (green in the original image) are the correct regions, and the dark lines (red in the original image) are the predicted regions.

[0042] Example of NBI image prediction results for detection model and segmentation model [Table 10] *The light lines (green in the original image) are the correct area, and the dark lines (red in the original image) are the predicted area. The rectangular output is displayed with the confidence level. In this example, bubbles are output and displayed as bubbles.

[0043] Example of WLI image prediction results for detection and segmentation models [Table 11] *The filled area (yellow in the original image) is the correct area, and the white line (red in the original image) is the predicted area. The rectangular output is displayed with the confidence level.

[0044] Example of detection results for NBI normal bladder images using the detection model and segmentation model [Table 12]

[0045] As described above, we have created a trained model applicable to both narrowband and white light observation images. Furthermore, it is now possible to identify bubbles. This invention provides an opportunity for physicians, regardless of their level of knowledge or expertise, to accurately and quickly identify Hannah lesions in both narrowband and white light observation images. Furthermore, even when a normal bladder exhibits reddish phases due to height differences, shadows, etc., false positives are avoided. This contributes to the accurate diagnosis of interstitial cystitis, relief for interstitial cystitis patients, and the formation of an international consensus.

Claims

1. Endoscopic image data of Hannah's lesions in the bladder was acquired as training data. A program that causes a computer to execute a process in which a target cystoscope image is input into a learning model that inputs a cystoscope image and outputs the presence or absence of Hannah lesions in the endoscopic image, and outputs the presence or absence of Hannah lesions or whether it is Hannah type interstitial cystitis.

2. 2. The program according to claim 1, wherein the endoscopic image data of the Hannah lesion in the bladder as training data includes both narrow-band light observation images and white-light observation images.

3. The learning model is based on endoscopic image data of a normal bladder without Hannah's lesions, Endoscopic image data of a non-normal bladder but without Hannah's lesions, The program according to claim 1 , further comprising the following as training data:

4. The program according to claim 1 or 3, further comprising endoscopic image data of air present in the bladder as training data.

5. The program according to claim 3 , further comprising a process for determining whether or not the bladder is normal and outputting the result.

6. A control device for a cystoscope, in which the program according to claim 1 or 2 is recorded.

7. A trained model using cystoscopy images acquired using a cystoscopy system, an input layer to which cystoscopy images are input; an output layer that outputs the presence or absence of Hannah lesions in the endoscopic image; Enter the endoscopic image data of the Hanna lesion in the bladder. an intermediate layer in which parameters are trained using training data that outputs the presence or absence of Hannah lesions in bladder images; A cystoscopic image of a target is input to the input layer, and the image is calculated in the intermediate layer, and the presence or absence of Hanna lesions or Hanna type interstitial cystitis is output. A trained model that makes a computer work.

8. The trained model of claim 7 further includes endoscopic image data of air present in the bladder as training data.

9. Endoscopic image data of a normal bladder; The trained model according to claim 7, further comprising endoscopic image data that does not contain Hannah lesions but is not a normal bladder as training data.

10. The trained model according to claim 7 , wherein the endoscopic image data in the training data includes both narrow-band light observation images and white-light observation images.

11. The trained model of claim 9 further comprising as an output whether the bladder is normal or abnormal.

12. A control device for a cystoscope, in which the trained model according to any one of claims 7 to 11 is recorded.

Images