Program and system relating to bladder endoscope image

Abstract

To provide a program, a device, and a system for objectively visualizing, quantifying, and / or standardizing the bladder condition of a patient, regardless of whether the patient is diagnosed with a disease or not.SOLUTION: In the system, the program causes the computer to function as means for detecting a new blood vessel in a bladder image, means for calculating a ratio or an amount of the detected new blood vessel in the image, and means for outputting the ratio or the amount. In addition, a device including a memory that stores a program and a processor that executes the program functions as means for detecting a new blood vessel of a bladder surface layer in a bladder image, means for calculating a ratio or an amount of the detected new blood vessel in the bladder image, and means for outputting the ratio or the amount by executing the program.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program or system relating to cystoscopy images, and more particularly to a program and system relating to cystoscopy images that are useful in determining abnormal bladder conditions. [Background technology]

[0002] Among bladder diseases, interstitial cystitis is a chronic disease characterized by symptoms such as frequent urination, urgency, and bladder pain and discomfort when the bladder is full. Although there have been many epidemiological survey reports, the cause has not been identified, and the disease is designated as an intractable disease in Japan. Furthermore, the definition, diagnostic criteria, and even terminology of "interstitial cystitis" vary by country and region. Therefore, the term "interstitial cystitis" in this application also encompasses the concepts of bladder pain syndrome and overactive bladder pain syndrome.

[0003] The criteria for interstitial cystitis routinely used in the United States are those of the Interstitial Cystitis Data Base (ICDB), a case series. 18 These criteria do not require cystoscopic findings. The National Institute of Diabets, Digestives, and Kidney Diseases (NIDDK) criteria, often cited, are more stringent because they require cystoscopic findings and are used for strict case selection in research. Reports suggest that fewer than half of patients diagnosed with interstitial cystitis according to the ICDB criteria meet the NIDDK criteria.

[0004] Interstitial cystitis can be broadly divided into Hanna type, which is characterized by Hanna lesions, and non-Hanna type, which is characterized by non-Hanna lesions. Hanna lesions are characterized by characteristic reddened mucosa lacking normal capillary structures. Pathologically, the epithelium is often exfoliated (eroded), and the submucosal tissue exhibits proliferation of new blood vessels and clusters of inflammatory cells. Hanna type exhibits clear abnormal findings both endoscopically and pathologically, and is characterized by characteristic reddened mucosa lacking normal capillary structures. As mentioned above, due to the lack of established international standards, Hanna lesions are sometimes referred to as Hanna ulcers or simply ulcers in some regions. In response to this situation, the present inventor disclosed the invention of Japanese Patent No. 7124041, which presented a technology useful for identifying Hanna lesions. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0005] However, even if Hannah lesions disappear through treatment or other lower urinary tract disorders are cured, this does not mean that all is resolved for the patient. Interstitial cystitis and other lower urinary tract disorders (including bladder cancer and other bladder-related diseases) do not suddenly return to normal. Even if a patient is not diagnosed as a disease, abnormal states exist, causing frequent urination and pain. However, there has been little research into such conditions in the bladder. The present inventors have focused on bladder abnormalities that are neither normal nor diagnosed as a disease. These include patients in the process of developing Hannah lesions even without Hannah lesions, patients whose Hannah lesions have disappeared but are in the process of recurrence, patients whose bladder conditions are worsening, and patients in the process of recovery or normalization. These processes include conditions not necessarily diagnosed as interstitial cystitis or other specific diseases. For patients suffering from frequent urination, pain during urination, or discomfort, regardless of whether they have been diagnosed with a specific disease, information on whether their symptoms are worsening or improving is extremely useful. At the same time, such objective information will greatly advance the development of new drugs and the measurement of treatment and drug effectiveness.

[0006] Recently, advances in special light and image processing have enabled early detection of minute lesions and the observation of subtle mucosal thickening and deep blood vessels. Examples of special light observation methods in practical use include narrowband imaging (NBI), autofluorescent imaging (AFI), and infrared imaging. NBI uses filtered light from the endoscope, extracted from the illumination light of the endoscope, with wavelengths of 400–430 nm, preferably 410–420 nm, and particularly preferably 415 nm, and light of wavelengths of 520–560 nm, preferably 530–550 nm, and particularly preferably 540 nm. Both wavelengths are highly absorbed by hemoglobin, highlighting capillaries in black. 415 nm light is used to observe the superficial layer of the mucosa, while 540 nm light is used to observe slightly deeper layers. NBI enhances the contrast with surrounding tissue, making the protrusions clear, making it possible to detect very small tumors that would be easily overlooked with a regular endoscope. The tumor boundaries are also much clearer than with other methods, making it a more advantageous method than white light observation.

[0007] Specifically, in intravesical examinations using NBI, the angiogenesis in the mucosal surface, characteristic of interstitial cystitis, can be observed using light with a wavelength of 415 nm. Meanwhile, bladder cancer can be diagnosed by observing angiogenesis present in relatively deep parts of the mucosa using light with a wavelength of 540 nm. Observing the mucosal layer where angiogenesis exists with light of each wavelength makes it possible to clearly distinguish and diagnose cancer and interstitial cystitis (as disclosed in U.S. Pat. No. 8,080,185, an invention by the present inventor).

[0008] However, unlike the stomach or lungs, it is still rare to perform cystoscopic examination of the bladder when cancer is not suspected and no disease has developed, and bladder findings must be judged based on the ability and experience of each individual physician. Therefore, there are few physicians who can accurately grasp the condition of the bladder, and there are no reports that objectively visualize, quantify, and / or standardize the patient's bladder condition.

[0009] Therefore, the present invention aims to provide a technology that can objectify, visualize, quantify, and / or standardize a patient's bladder condition, regardless of whether or not the patient has been diagnosed with a disease. [Means for solving the problem]

[0010] The inventors have come to realize that the state of neovascularization in a patient's bladder is an important factor in determining the recovery / worsening process of lower urinary tract disorders, particularly interstitial cystitis, and that it can be evaluated in stages. In other words, symptoms worsen as neovascularization of the bladder, particularly in the bladder surface, increases, and tend to improve as neovascularization decreases. If neovascularization in the bladder surface is concentrated and worsening progresses, the likelihood of developing / presenting as a Hanna lesion increases. Therefore, visualization, quantification, and / or standardization of bladder status would not only aid physicians in diagnosis and judgment, but also lead to progress in terms of patient relief, global standardization of judgment, and the development of new therapeutic agents.

[0011] Therefore, the present invention comprises a program for causing a computer to function as a means for detecting neovascularization in a bladder image, a means for calculating the ratio or amount of detected neovascularization in the image, and a means for outputting said ratio or amount. The present invention also comprises an apparatus including a memory for storing the program and a processor for executing the program, wherein, by executing the program, the processor functions as a means for detecting neovascularization on the bladder surface in a bladder image, a means for calculating the ratio or amount of detected neovascularization in the bladder image, and a means for outputting said ratio or amount. Preferably, the apparatus further comprises a step of comparing the ratio with a plurality of predetermined thresholds and a step of outputting the threshold interval between which the ratio lies.

[0012] The present invention also comprises a system including a memory for storing a program, a processor for executing the program, and a display device, wherein the processor execution step includes steps of acquiring an endoscopic bladder image, detecting neovascularization of the bladder, calculating the ratio in the image of the detected neovascularization, comparing the ratio with a plurality of predetermined thresholds, outputting the threshold interval between which the ratio falls, and displaying the output on the display device.

[0013] It is preferable that the neovascularization is neovascularization on the surface layer of the bladder, and that the bladder image is an image acquired by endoscopic narrow band imaging (NBI).

[0014] That is, the present invention makes it possible to visualize, quantify, and / or normalize bladder conditions by detecting the proportion or amount of neovascularization in cystoscopic images, which are difficult to accurately grasp, compare, standardize, and normalize from a global perspective or from the perspective of individual physicians. If the proportion or amount of neovascularization in the images is high (or if it tends to increase over time), it can be said that there is a high possibility of interstitial cystitis or other diseases, or that the bladder condition is tending to deteriorate. On the other hand, if the amount of neovascularization in the images is low (or if it tends to decrease over time), it can be said that there is a low possibility of interstitial cystitis or other diseases, or that the bladder condition is tending to improve. Regarding interstitial cystitis, neovascularization in the superficial layer of the bladder is an indicator of bladder abnormalities. The present invention can also be applied to assessing abnormal conditions (including bladder cancer) by detecting the proportion or amount of neovascularization in the deep layer of the bladder. The present invention can also be understood as a method for diagnosing / assisting bladder conditions, or as a method for producing a program or system.

[0015] Since the present invention measures the ratio or amount in a given image, if the endoscopic image shows a small area, it will be localized, and if it shows a large area, it will be globalized, allowing the status of the desired area to be confirmed in real time. Furthermore, from the perspective of standardization and regulation, by specifying the specific location and area within the bladder to be imaged, it becomes possible to determine the bladder condition using a common intrabladder image area. While merely a reference example, for example, "the area (apex) where air bubbles are observed behind the left and right ureteral orifices (posterior trigone) after injecting 100 ml (~150 ml) of saline." Furthermore, by observing the specific location and area within the bladder over time in the same patient, the recovery / deterioration trend can be more clearly understood. The "ratio or amount" may be anything that allows for objective comparison, and the most typical example is "the ratio of neovascularization in bladder images." Furthermore, although the present invention refers to "ratio in bladder image," the entire image may be the subject as long as "comparison" is possible, or if the bladder image contains a background outside the bladder, the image may be processed or converted to an image containing only the inside of the bladder by trimming or other means to remove the background, and then the ratio may be calculated. The "amount" may be calculated and quantified based on the ratio, and provided as a comparable value in a specified area.

[0016] The present invention facilitates comprehensive assessment while visually confirming various intrabladder area conditions. In this regard, the output is not limited to a simple numerical display of ratios or amounts. It is preferable to provide multiple thresholds and display each level using visual elements, such as colors or patterns, similar to the precipitation forecast displayed in weather information. Such a display makes it easier for physicians and patients to visually recognize and understand the condition. Furthermore, since NBI can distinguish between neovascularization in the deep layer of the bladder and neovascularization in the superficial layer, the condition can also be grasped based on the ratio of neovascularization in the superficial layer to neovascularization in the deep layer. Since neovascularization in the deep layer of the bladder can also include abnormal conditions, a configuration that detects neovascularization in the deep layer of the bladder is also conceivable.

[0017] Furthermore, in the present invention, the acquired image is not limited to NBI, as long as neovascularization at least on the surface of the bladder can be detected. For example, even a white light observation image can be adopted, since it is possible to acquire an image including at least neovascularization on the surface, it is possible to see clear differences from a normal bladder, and if neovascularization is clearly visible, it is assumed that it is also present on the surface. While the ideal from the perspective of interstitial cystitis is complete detection of neovascularization only on the surface of the bladder, one advantageous feature of the present invention is that it provides an objectively comparable guideline for neovascularization, enabling grading.

[0018] Regarding thresholds, it is preferable to use a database to set multiple thresholds, referencing which ranges correspond to the thresholds, and displaying visual elements such as colors or patterns corresponding to the ranges in the image. For example, normal values ​​(0-5%) can be displayed as white, generally good values ​​(5-10%) as blue, nearly good but abnormal values ​​(10-15%) as yellow, abnormal values ​​(15-25%) as orange, and high suspicion of illness (25% or higher) as red. Since the present invention is sufficient as long as information can be visually grasped, the ranges and colors are merely examples, and the number and settings of thresholds and visual elements are not limited. Furthermore, various forms of output results are possible, such as displaying them in the image or outputting them separately from the image as a judgment report, and are not particularly limited.

[0019] The present invention can be implemented using known hardware, software, and other devices, such as input terminals, processors (various arithmetic / processing devices such as CPUs and GPUs), various recording devices (various memories such as ROMs and RAMs), databases and / or programs recorded in various recording devices, display devices, mobile terminals, servers, etc. The present invention can also be implemented as a standalone system or on the cloud. Furthermore, the endoscopic images in the present invention include not only still images but also moving images, and can also be implemented as a system mounted on, for example, a cystoscope control device to enable real-time display.

[0020] Various well-known techniques can be employed to detect blood vessels from endoscopic images, and are not particularly limited. For example, relatively simple detection methods using the HSV color space or RGB values ​​are possible, as are image processing techniques using edge detection. Specifically, the edges of blood vessels in an image are detected using a Canny edge detector or a Sobel filter. Regarding hue separation, since the color characteristics of blood vessels in NBI can clearly indicate the difference between the superficial and deep layers, methods that identify blood vessel regions using hue, texture analysis based on the continuity of vascular muscle excluding blue, or a combination of these are possible. Note that with NBI, neovascularization in the superficial layer of the bladder is primarily characterized by high intensities of brown and red, while neovascularization in the deep layers is characterized by high intensities of green and blue.

[0021] Another typical example is the adoption of machine learning technology. For example, a convolutional neural network (CNN) is used to train the CNN to learn about neovascularization areas on the bladder surface using endoscopic image data, and then the CNN is automatically detected. NBI is suitable for generating a learning model because the characteristics of blood vessel color and vein continuity can clearly show the difference between the superficial and deep layers. The learning model is typically a trained model using cystoscope images acquired using a cystoscope system, and includes an input layer to which the cystoscope image is input, an output layer that outputs neovascularization in the surface layer and / or deep layer of the bladder in the endoscopic image, and an intermediate layer in which parameters are trained using training data that inputs endoscopic image data of the bladder and outputs neovascularization in the surface layer and / or deep layer of the bladder in the bladder image, and can be configured as a trained model for causing a computer to function by inputting a target cystoscope image to the input layer, performing calculations in the intermediate layer, outputting neovascularization in the surface layer and / or deep layer of the bladder in the bladder image, and calculating and outputting the ratio of the output neovascularization in the bladder image. Note that the model for detecting blood vessels and the means for calculating and outputting the ratio of the output neovascularization in the bladder image may be configured separately, or may be configured as incorporated within the model.

[0022] In addition, white light observation images cannot distinguish between superficial and deep layers, but neovascularization can be detected. Also, a feature detection method using computer vision technology such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) can be used to detect blood vessels. The present invention is not limited to any particular detection method as long as neovascularization can be detected. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a diagram showing the flow of a program according to the present invention; [Figure 2] FIG. 10 is a diagram showing a cystoscopy image. [Figure 3] FIG. 1 shows a bladder image as an example of an implementation of the present invention. [Figure 4] FIG. 10 shows another bladder image as an example of an implementation of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] FIG. 1 shows the flow of the program of the present invention. First, an input endoscopic bladder image is acquired (S1), and neovascularization in this image is detected (S2). Depending on the embodiment, detection may be limited to the superficial layer of the bladder, or may include detection of the deep layer of the bladder. Then, the ratio of the detected neovascularization in the image is calculated (S3). The obtained ratio is compared with a plurality of predetermined thresholds by referring to a database (S4). Therefore, the database records a plurality of predetermined thresholds and the display content between the predetermined thresholds. The obtained ratio and the threshold value between which the ratio falls are then output in the image (S5).

[0025] Figure 2 is an example of a cystoscopy image taken with NBI (the original is in color; the same applies to the other images). Image 1 does not contain any Hannah lesions, but the area is reddish overall and has many superficial neovascularizations, which is not normal and raises concerns that if it worsens, Hannah lesions may develop or some other symptoms may occur.

[0026] The following method was used for the detection process for Figure 2.

[0027] 1. Loading and Preprocessing Images The image was captured and converted to the HSV (Hue, Saturation, Value) color space, which is more suitable for color detection and therefore easier to distinguish colors than RGB.

[0028] 2. Defining the color range To detect the green, red / brown, and white areas separately, the range of each color was defined by the HSV value. Green: Hue: 35-85, Saturation: 50-200, Value: 50-200 Red / Brown: Red (Range 1): Hue: 0-10, Saturation: 10-255, Value: 10-255 Red (Range 2): Hue: 160-180, Saturation: 10-255, Value: 10-255 Brown: Hue: 10-20, Saturation: 100-255, Value: 20-200 White: Hue: 0-180, Saturation: 0-40, Value: 180-255

[0029] 3. Mask Generation I generated a mask for each color based on the color range I defined. I used the cv2.inRange function to find pixels that match each color range and create a binary mask.

[0030] 4. Eliminate duplicates To avoid overlapping masks, color priorities were set and overlapping areas were removed. (1) The red and brown masks were given top priority, and these areas were excluded from the green mask. (2) The green mask was given second priority, and the green, red, and brown parts were removed from the white mask.

[0031] 5. Calculate the percentage of each color The number of pixels in each color mask was calculated and their percentage was calculated relative to the total number of pixels in the image. The number of green, red / brown, and white pixels was counted separately and divided by the total number of pixels in the image to calculate the percentage. The percentages were then adjusted so that the total added up to 100%.

[0032] As a result, the obtained ratio is displayed in the image as Ratio 2 (14%), as shown in Figure 3. Furthermore, a database is used to determine whether the ratio falls within any of multiple threshold ranges, and visual element 3 is displayed in the image using a color or pattern corresponding to that range. For example, thresholds may be white for normal or near-normal values ​​(0-5%), blue for low-grade abnormality (5-10%), yellow for moderate abnormality (10-15%), orange for high-grade abnormality (15-25%), and red for a high need for treatment (25% or higher). Note that these ranges and visual elements are merely examples, and the number of thresholds and visual elements are not limited. Color variations may also be displayed using gradational transitions.

[0033] Figure 4 shows image 1' showing the bladder condition. Ratio 2 (1%) and visual element 3 clearly indicate that there is less neovascularization on the bladder surface compared to Figure 3, indicating a good bladder condition. If the same patient's condition changes from Figure 4 to Figure 3, the condition worsens, even if it is not necessarily diagnosed as disease. If the reverse process occurs, the condition improves. Therefore, it serves as a guideline for physicians and patients to understand the condition. Furthermore, even when objective judgment or comparison is difficult due to the individual physician's abilities, understanding the condition and making a judgment becomes easier. Furthermore, because it facilitates understanding the progression of the condition, it is useful for new drug development, efficacy assessment, research, etc., such as therapeutic and preventive drugs. Furthermore, given the current situation in which many physicians worldwide have no experience using cystoscopy, the number of physicians capable of diagnosing interstitial cystitis is low, and even fewer physicians have visual experience of the almost complete absence of neovascularization on the bladder surface in a normal bladder, the present invention provides a clear and concise guideline, making it particularly useful in areas where bladder-related medical care is underdeveloped and for less experienced physicians.

Claims

1. Computer means for detecting neovascularization in a bladder image; a means for calculating the ratio or amount of detected neovascularization in the image; A program for causing the program to function as a means for outputting the ratio or amount.

2. The program according to claim 1 , wherein the new blood vessels are new blood vessels on the surface of the bladder.

3. 3. The program according to claim 2, wherein the bladder image is an image acquired by endoscopic narrow band imaging (NBI).

4. 2. The program of claim 1, wherein the output includes a visual means other than numbers.

5. means for comparing said ratio with a plurality of predetermined thresholds; 5. The program according to claim 1, further comprising means for outputting between which thresholds the ratio lies.

6. A cystoscope device having the program according to claim 1 installed therein.

7. A cystoendoscopic device having the program according to claim 5 installed therein.

8. A memory for storing programs; a processor that executes the program, By executing the program, the processor a means for detecting neovascularization on the surface of the bladder in a bladder image; a means for calculating the ratio or amount of detected neovascularization in the bladder image; A device that serves as a means for outputting said ratio or amount.

9. means for comparing said ratio with a plurality of predetermined thresholds; The apparatus of claim 8 further comprising means for outputting between which thresholds the ratio lies.

10. The apparatus according to claim 8, wherein the bladder image is an image acquired by endoscopic narrow-band imaging.

11. 11. The device of claim 8, wherein the output includes output by visual means other than numbers.

12. A memory for storing programs; A system including a processor that executes the program and a display device, The steps performed by the processor include: acquiring endoscopic bladder images; detecting neovascularization of the bladder; calculating the ratio of detected neovascularization in the image; comparing said ratio to a plurality of predetermined thresholds; outputting the ratio between the thresholds; and generating a display including the output on the display device.

13. The system of claim 12, wherein the memory or processor is installed in a cystoscopy device.

14. The system of claim 13 , wherein the output includes a visual output other than a number.

15. The system according to any one of claims 12 to 14, wherein the new blood vessels are new blood vessels on the surface of the bladder.

Images