Program and system related to bladder endoscopic image

Abstract

The present invention is composed of a program for causing a computer to function as a means for detecting new blood vessels in a bladder endoscopic image, a means for calculating a ratio or an amount of the detected new blood vessels in the image, and a means for outputting the ratio or the amount. The present invention is also composed of a device or a system, the device comprising a memory that stores the program and a processor that executes the program, wherein, by executing the program, the processor functions as a means for detecting new blood vessels of the bladder surface layer in the bladder endoscopic image, a means for calculating the ratio or the amount of the detected new blood vessels in the bladder endoscopic image, and a means for outputting the ratio or the amount. The present invention enables objectification, visualization, quantification and / or standardization of the bladder condition of a patient, regardless of whether or not the patient has been diagnosed as having a disease.

Description

Cystoscopy image related program and system

[0001] The present invention relates to a program or system relating to cystoscopy images, and more particularly to a program and system relating to cystoscopy images that are useful in determining abnormal bladder conditions.

[0002] Among bladder diseases, interstitial cystitis is a chronic disease characterized by symptoms such as frequent urination, urgency, and bladder pain and discomfort when the bladder is full. Although there have been many epidemiological survey reports, the cause has not been identified, and the disease is designated as an intractable disease in Japan. Furthermore, the definition, diagnostic criteria, and even terminology of "interstitial cystitis" vary by country and region. Therefore, the term "interstitial cystitis" in this application also encompasses the concepts of bladder pain syndrome and overactive bladder pain syndrome.

[0003] The criteria for interstitial cystitis routinely used in the United States are those of the Interstitial Cystitis Data Base (ICDB), a case series. 18 These criteria do not require cystoscopic findings. The National Institute of Diabets, Digestives, and Kidney Diseases (NIDDK) criteria, often cited, are more stringent because they require cystoscopic findings and are used for strict case selection in research. Reports suggest that fewer than half of patients diagnosed with interstitial cystitis according to the ICDB criteria meet the NIDDK criteria.

[0004] Interstitial cystitis can be broadly divided into Hanna type, which is characterized by Hanna lesions, and non-Hanna type, which is characterized by non-Hanna lesions. Hanna lesions are characterized by a distinctive reddened mucosa lacking a normal capillary structure. Pathologically, the epithelium is often eroded, and the submucosal tissue exhibits proliferation of new blood vessels and clusters of inflammatory cells. Hanna type lesions exhibit clear abnormalities both endoscopically and pathologically, and are characterized by a distinctive reddened mucosa lacking a normal capillary structure. As mentioned above, due to the lack of established international standards, Hanna lesions are sometimes referred to as Hanna ulcers or simply ulcers in some regions. In response to this situation, the present inventor disclosed the invention of Japanese Patent No. 7124041, which presented a technology useful for identifying Hanna lesions.

[0005] However, even if Hannah lesions disappear through treatment or other lower urinary tract disorders are cured, this does not mean that all is resolved for the patient. Interstitial cystitis and other lower urinary tract disorders (including bladder cancer and other bladder-related diseases) do not suddenly return to normal. Even if a patient is not diagnosed as a disease, abnormal states exist, causing frequent urination and pain. However, there has been little research into such conditions in the bladder. The present inventors have focused on bladder abnormalities that are neither normal nor diagnosed as a disease. These include patients in the process of developing Hannah lesions even without Hannah lesions, patients whose Hannah lesions have disappeared but are in the process of recurrence, patients whose bladder conditions are worsening, and patients in the process of recovery or normalization. These processes include conditions not necessarily diagnosed as interstitial cystitis or other specific diseases. For patients who suffer from frequent urination, pain during urination, or discomfort, regardless of whether they have been diagnosed with a specific disease, information on whether their symptoms are worsening or improving is extremely useful. At the same time, such objective information will greatly advance the development of new drugs and the measurement of treatment and drug effectiveness.

[0006] Recently, advances in special light and image processing have enabled early detection of minute lesions and the observation of subtle mucosal thickening and deep blood vessels. Examples of special light imaging methods in practical use include narrowband imaging (NBI), autofluorescent imaging (AFI), and infrared imaging. NBI uses filtered endoscopic illumination light with wavelengths of 400–430 nm, preferably 410–420 nm, and particularly preferably 415 nm, and light with wavelengths of 520–560 nm, preferably 530–550 nm, and particularly preferably 540 nm. Both wavelengths are highly absorbed by hemoglobin, highlighting capillaries in black. 415 nm light is used to observe the superficial mucosal layer, while 540 nm light is used to observe deeper layers. NBI enhances the contrast with surrounding tissue, making the protrusions clear, making it possible to detect very small tumors that would be easily overlooked with a regular endoscope. The tumor boundaries are also much clearer than with other methods, making it a more advantageous method than white light observation.

[0007] Specifically, intravesical examination using NBI can observe the angiogenesis in the mucosal surface, which is characteristic of interstitial cystitis, using light with a wavelength of 415 nm. Meanwhile, bladder cancer can be diagnosed by observing angiogenesis present in relatively deep parts of the mucosa using light with a wavelength of 540 nm. Observing the mucosal layer where angiogenesis exists using light of each wavelength makes it possible to clearly distinguish and diagnose cancer and interstitial cystitis (as disclosed in U.S. Patent No. 8,080,185, an invention by the present inventor).

[0008] However, unlike the stomach or lungs, it is still rare to perform cystoscopic examination of the bladder when cancer is not suspected and no disease has developed, and bladder findings must be judged based on the ability and experience of each individual physician. Therefore, there are few physicians who can accurately grasp the condition of the bladder, and there are no reports that objectively visualize, quantify, and / or standardize the patient's bladder condition.

[0009] Therefore, the present invention aims to provide a technology that can objectively visualize, quantify, and / or standardize a patient's bladder condition, regardless of whether or not the patient has been diagnosed with a disease.

[0010] The inventors have come to realize that the state of neovascularization in a patient's bladder is an important factor in determining the recovery / worsening process of lower urinary tract disorders, particularly interstitial cystitis, and that it can be evaluated in stages. In other words, symptoms worsen as bladder neovascularization, particularly in the bladder surface, increases, and tend to improve as neovascularization decreases. If neovascularization in the bladder surface is concentrated and worsening progresses, the likelihood of developing / presenting as Hanna lesions increases. Therefore, visualization, quantification, and / or standardization of bladder status would not only aid physicians in diagnosis and assessment, but also lead to progress in terms of patient relief, global standardization of assessment, and the development of new therapeutic agents.

[0011] Therefore, the present invention comprises a program for causing a computer to function as a means for detecting neovascularization in cystoendoscopic images, a means for calculating the ratio or amount of detected neovascularization in the images, and a means for outputting said ratio or amount. The present invention also comprises an apparatus including a memory for storing the program and a processor for executing the program, wherein, by executing the program, the processor functions as a means for detecting neovascularization on the bladder surface in cystoendoscopic images, a means for calculating the ratio or amount of detected neovascularization in the cystoendoscopic images, and a means for outputting said ratio or amount. Preferably, the apparatus further comprises a step of comparing the ratio with a plurality of predetermined thresholds and a step of outputting the threshold interval between which the ratio lies.

[0012] The present invention also provides a system including a memory for storing a program, a processor for executing the program, and a display device, wherein the processor execution step includes steps of acquiring a cystoscopic image, detecting neovascularization of the bladder, calculating the ratio in the image of the detected neovascularization, comparing the ratio with a plurality of predetermined thresholds, outputting the threshold interval between which the ratio lies, and displaying the output on the display device.

[0013] It is also preferable that the neovascularization is neovascularization on the surface of the bladder, and that the cystoscopy image is an image acquired by endoscopic narrow band imaging (NBI).

[0014] That is, the present invention makes it possible to visualize, quantify, and / or normalize bladder conditions by detecting the proportion or amount of neovascularization in cystoscopic images, which are difficult to accurately grasp, compare, standardize, and normalize from a global perspective or from the perspective of individual physicians. A high proportion or amount of neovascularization in an image (or a tendency for it to increase over time) can indicate a high possibility of interstitial cystitis or other diseases, or a tendency for the bladder condition to worsen. On the other hand, a low proportion of neovascularization in an image (or a tendency for it to decrease over time) can indicate a low possibility of interstitial cystitis or other diseases, or a tendency for the bladder condition to improve. Regarding interstitial cystitis, neovascularization in the superficial layer of the bladder is an indicator of bladder abnormalities. The present invention can also be applied to assessing abnormal conditions (including bladder cancer) by detecting the proportion or amount of neovascularization in the deep layer of the bladder. The present invention can also be understood as a method for diagnosing / assisting bladder conditions, or as a method for producing a program or system.

[0015] Since the present invention measures the ratio or amount in a given image, a narrow endoscopic image provides a localized image, whereas a broader image provides a global image, allowing for real-time confirmation of the desired area. Furthermore, from the perspective of standardization and regulation, by defining a specific intrabladder location and area to be imaged, it becomes possible to assess bladder status using a common intrabladder image area. While merely a reference example, for example, "the area (apex) where air bubbles are observed behind the left and right ureteral orifices (posterior trigone) after injecting 100 ml (~150 ml) of saline." Furthermore, by observing a specific intrabladder location and area over time in the same patient, recovery / deterioration trends can be more clearly understood. The "ratio or amount" may be anything that allows for objective comparison, most typically the "ratio of neovascularization in cystoscopic images." Furthermore, although the present invention refers to "ratio in a cystoscope image," the entire image may be the subject of the comparison as long as it allows for comparison, or if the cystoscope image contains a background outside the bladder, the image may be processed or converted to an image containing only the inside of the bladder by trimming or other means to remove that background, and then the image may be used to calculate the ratio. The "amount" may also be calculated and quantified based on the ratio, and provided as a comparable value in a specified area.

[0016] The present invention facilitates comprehensive assessment while visually confirming various intrabladder area conditions. In this regard, the output is not limited to a simple numerical display of ratios or amounts. It is preferable to provide multiple thresholds and display each level using visual elements, such as colors or patterns, similar to the precipitation forecast displayed in weather information. Such a display makes it easier for physicians and patients to visually recognize and understand the condition. Furthermore, since NBI can distinguish between neovascularization in the deep layer of the bladder and neovascularization in the superficial layer, the condition can also be grasped based on the ratio of neovascularization in the superficial layer to neovascularization in the deep layer. Since neovascularization in the deep layer of the bladder can also include abnormal conditions, a configuration that detects neovascularization in the deep layer of the bladder is also conceivable.

[0017] Furthermore, in the present invention, the acquired image is not limited to NBI, as long as neovascularization at least on the surface of the bladder can be detected. For example, even a white light observation image can be adopted, since it is possible to acquire an image including at least neovascularization on the surface, it is possible to see clear differences from a normal bladder, and if neovascularization is clearly visible, it is assumed that it is also present on the surface. While the ideal from the perspective of interstitial cystitis is complete detection of neovascularization only on the surface of the bladder, one advantageous feature of the present invention is that it provides an objectively comparable guideline for neovascularization, enabling grading.

[0018] Regarding thresholds, it is preferable to use a database to set multiple thresholds, referencing which ranges correspond to the thresholds, and displaying visual elements such as colors or patterns corresponding to the ranges in the image. For example, normal values ​​(0-5%) can be displayed as white, generally good values ​​(5-10%) as blue, nearly good but abnormal values ​​(10-15%) as yellow, abnormal values ​​(15-25%) as orange, and high suspicion of illness (25% or higher) as red. Since the present invention is sufficient as long as the information can be visually grasped, the ranges and colors are merely examples, and the number and settings of thresholds and visual elements are not limited. Furthermore, various forms of output results are possible, such as displaying them in the image or outputting them separately from the image in a diagnosis report or the like, and are not particularly limited.

[0019] The present invention can be implemented using known hardware, software, and other devices, such as input terminals, processors (various arithmetic / processing devices such as CPUs and GPUs), various recording devices (various memories such as ROMs and RAMs), databases and / or programs recorded in various recording devices, display devices, mobile terminals, servers, etc. It can also be implemented as a standalone system or on a cloud platform. Furthermore, the endoscopic images in the present invention include not only still images but also moving images, and can be realized as a system mounted on, for example, a cystoscope control device to enable real-time display.

[0020] Various known techniques can be employed for detecting blood vessels from endoscopic images, and are not particularly limited. For example, relatively simple detection methods using the HSV color space or RGB values ​​are possible, as are image processing techniques using edge detection. Specifically, the edges of blood vessels in an image are detected using a Canny edge detector or a Sobel filter. For hue separation, since the color characteristics of blood vessels in NBI can clearly indicate the difference between the superficial and deep layers, methods that identify blood vessel regions using hue, texture analysis based on the continuity of vascular muscle excluding blue, or a combination of these are possible. Note that with NBI, neovascularization in the superficial layer of the bladder is primarily characterized by high brown and red intensities, while neovascularization in the deep layers is characterized by high green and blue intensities.

[0021] Another typical example is the adoption of machine learning technology. For example, a convolutional neural network (CNN) is used to train the CNN to learn about neovascularization areas on the bladder surface using endoscopic image data, and then the CNN is automatically detected. NBI is suitable for generating a learning model because the characteristics of blood vessel color and muscle continuity can clearly show the difference between the superficial and deep layers. The learning model is typically a trained model using cystoscope images acquired using a cystoscope system, and includes an input layer to which the cystoscope image is input, an output layer that outputs neovascularization in the surface layer and / or deep layer of the bladder in the endoscopic image, and an intermediate layer in which parameters are trained using training data that inputs the cystoscope image data and outputs neovascularization in the surface layer and / or deep layer of the bladder in the cystoscope image, and can be configured as a trained model for causing a computer to function by inputting a target cystoscope image to the input layer, performing calculations in the intermediate layer, outputting neovascularization in the surface layer and / or deep layer of the bladder in the cystoscope image, and calculating and outputting the ratio of the output neovascularization in the cystoscope image. Note that the model for detecting blood vessels and the means for calculating and outputting the ratio of the output neovascularization in the cystoscope image may be configured separately, or may be configured as incorporated within the model.

[0022] In addition, white light observation images cannot distinguish between superficial and deep layers, but neovascularization can be detected. Also, a feature detection method using computer vision technology such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) can be used to detect blood vessels. The present invention is not limited to any particular detection method as long as neovascularization can be detected.

[0023] 1A and 1B are diagrams showing cystoscopy images illustrating the flow of a program according to the present invention, 1A and 1B are diagrams showing cystoscopy images as an example of an embodiment of the present invention, and 1B are diagrams showing another cystoscopy image as an example of an embodiment of the present invention.

[0024] FIG. 1 shows the program flow of the present invention. First, an input cystoscope image is acquired (S1), and neovascularization in this image is detected (S2). Depending on the implementation, detection may be limited to the superficial layer of the bladder, or may include detection of the deep layer of the bladder. The ratio of detected neovascularization in the image is then calculated (S3). The obtained ratio is compared with a number of predetermined thresholds by referencing a database (S4). Therefore, the database records a number of predetermined thresholds and the display content between the predetermined thresholds. The obtained ratio and the threshold value between which the ratio falls are then output in the image (S5).

[0025] Figure 2 is an example of a cystoscopic image taken by NBI (the original is in color; the same applies to the other images). While the image in question does not contain any Hannah lesions, it is reddish overall and has many superficial neovascularizations, which is not normal and raises concerns that if it worsens, Hannah lesions may develop or some other symptoms may occur.

[0026] The following method was used for the detection process for FIG.

[0027] 1. Image loading and preprocessing The image was acquired and converted into HSV (Hue, Saturation, Value) color space. HSV color space is suitable for color detection, making it easier to distinguish colors than RGB.

[0028] 2. Definition of color ranges To detect the green, red / brown, and white areas separately, the range of each color was defined by HSV values. Green: Hue: 35-85, Saturation: 50-200, Value: 50-200 Red / brown: Red (Range 1): Hue: 0-10, Saturation: 10-255, Value: 10-255 Red (Range 2): Hue: 160-180, Saturation: 10-255, Value: 10-255 Brown: Hue: 10-20, Saturation: 100-255, Value: 20-200 White: Hue: 0-180, Saturation: 0-40, Value: 180-255

[0029] 3. Mask Generation: Based on the defined color ranges, we generated a mask for each color. We used the cv2.inRange function to find pixels that match each color range and create a binary mask.

[0030] 4. Exclusion of overlapping areasTo avoid overlapping masks, we set color priorities and excluded overlapping areas.(1) The red and brown masks were given first priority, and these areas were excluded from the green mask.(2) The green mask was given second priority, and the green, red, and brown areas were excluded from the white mask.

[0031] 5. Calculating the percentage of each color: The number of pixels in each mask was calculated, and the percentage was calculated based on the total number of pixels in the image. The number of green, red / brown, and white pixels was counted, and divided by the total number of pixels in the image to calculate the percentage. The percentages were then adjusted so that the total added up to 100%.

[0032] As a result, the obtained ratio is displayed in the image as Ratio 2 (14%), as shown in Figure 3. Furthermore, a database is used to determine whether the ratio falls within any of multiple threshold ranges, and visual element 3 is displayed in the image using a color or pattern corresponding to that range. For example, thresholds may be white for normal or near-normal values ​​(0-5%), blue for low-grade abnormality (5-10%), yellow for moderate abnormality (10-15%), orange for high-grade abnormality (15-25%), and red for a condition requiring high treatment (25% or higher). Note that these ranges and visual elements are merely examples, and the number of thresholds and visual elements are not limited. Color variations may also be displayed using gradational transitions.

[0033] Figure 4 shows image 1' showing the bladder condition. Ratio 2 (1%) and visual element 3 indicate at a glance that there is less neovascularization on the bladder surface compared to Figure 3, indicating that the bladder is in good condition. If the same patient's condition transitions from Figure 4 to Figure 3, it indicates a worsening of the condition, even if it is not necessarily diagnosed as disease, while the reverse process indicates an improvement. Therefore, it serves as a guideline for physicians and patients to understand the condition. Furthermore, it facilitates understanding and judgment of the condition, even when objective judgment and comparison are difficult due to the individual physician's abilities. Furthermore, since it also facilitates understanding of the progression of the condition, it is useful for new drug development, efficacy assessment, research, etc., such as therapeutic and preventive drugs. Furthermore, given the current situation in which many physicians worldwide have no experience using cystoscopy, the number of physicians capable of diagnosing interstitial cystitis is low, and even fewer physicians have visual experience of the almost complete absence of neovascularization on the bladder surface in normal bladders, the present invention provides a clear and concise guideline, and is particularly useful in areas where bladder-related medical care is underdeveloped and for less experienced physicians.

Claims

1. A program for causing a computer to function as a means for detecting neovascularization in cystoscopy images, a means for calculating the ratio or amount of detected neovascularization in the image, and a means for outputting said ratio or amount.

2. The program according to claim 1, wherein the new blood vessels are new blood vessels on the surface of the bladder.

3. The program according to claim 2, wherein the cystoscopy image is an image acquired by endoscopic narrow band imaging (NBI).

4. The program of claim 1, wherein the output includes visual means other than numbers.

5. A program according to any one of claims 1 to 4, further comprising: means for comparing the ratio with a plurality of predetermined threshold values; and means for outputting the ratio between the threshold values.

6. A cystoscope device equipped with the program according to claim 1.

7. A cystoscope device having the program according to claim 5 installed therein.

8. An apparatus comprising a memory for storing a program and a processor for executing the program, wherein, by executing the program, the processor functions as: a means for detecting neovascularization on the surface of the bladder in a cystoscopy image; a means for calculating the ratio or amount of detected neovascularization in the cystoscopy image; and a means for outputting the ratio or amount.

9. The apparatus of claim 8, further comprising: means for comparing said ratio with a plurality of predetermined thresholds; and means for outputting between which thresholds said ratio lies.

10. The apparatus according to claim 8, wherein the cystoscopy image is an image acquired by endoscopic narrow band imaging.

11. The device of any one of claims 8 to 10, wherein the output includes output by visual means other than numbers.

12. A system including a memory for storing a program, a processor for executing the program, and a display device, wherein the steps executed by the processor include: acquiring a cystoscopy image; detecting neovascularization of the bladder in the endoscopic image; calculating the ratio of the detected neovascularization in the cystoscopy image; comparing the ratio with a plurality of predetermined thresholds; outputting the thresholds between which the ratio falls; and displaying the output on the display device.

13. The system of claim 12, wherein the memory or processor is installed in a cystoscopy device.

14. The system of claim 13, wherein the output includes output by visual means other than numbers.

15. The system according to any one of claims 12 to 14, wherein the new blood vessels are new blood vessels on the surface of the bladder.

Images