Image analysis device and image analysis method

Abstract

The present invention aims to automatically distinguish between super-magnified images and non-magnified images in a computer-aided diagnosis system that uses image analysis to analyze the state of epithelium. The image analysis device (10) of the present invention is an image analysis device (10) connected to an endoscope (20), comprising: a target image determination unit (111) that acquires an image from the endoscope (20) and determines that the image is a target image using a halo area contained in the image; and an image analysis unit (112) that, when the image is a target image, uses the target image to analyze the state of epithelium captured by the endoscope (20).

Description

Technical Field

[0001] The present disclosure relates to an image analysis device and an image analysis method. Background Art

[0002] In recent years, endoscopes with super-magnification functions with a microscope-level magnification of 380 times or more have been developed, and endocytoscopy has been developed, which can magnify and observe the epithelium of living cavities at the cellular level such as cell nuclei, blood vessels, and glandular cavities. Endocytoscopy is a contact endoscope that brings the lens surface into contact with the epithelium as the target and focuses using a zoom mechanism mounted on the endoscope to obtain super-magnified images. It is reported that super-magnified images can be used to predict pathological tissue diagnosis of organs such as the esophagus (for example, see non-patent document 1), stomach (for example, see non-patent document 2), duodenum (for example, see non-patent document 3), and large intestine (for example, see non-patent document 4).

[0003] However, even when using endocytoscopy to capture super-magnified images, a certain level of proficiency in image analysis of super-magnified images is required to predict pathological tissue diagnoses (see, for example, Non-Patent Document 4). Consequently, computer-assisted diagnosis systems have been developed to enable prediction of pathological tissue diagnoses even without this level of proficiency. This has been found to be useful for predicting pathological tissue diagnoses (see, for example, Non-Patent Documents 5 and 6).

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Y. Kumagai, K. Monma, K. Kawada, "Magnifying chromoendoscopy of the esophagus: in-vivo pat hological diagnosis using an endocytoscopy system", Endoscopy 2004; 36: 590-4.

[0007] Paper 2:H.Sato,H.lnoue,B.Hayee,et al.,“In vivo histopathologyusing endocytoscopy for non-neoplastic changes in the gastric mucosa:a prospective pilot study(with video)”,Gastrointest Endosc2015:81:875-81.

[0008] Paper 3:S.Miyamoto,T.Kudo,S.Abiko,et al.,“Endocytoscopy ofSuperficial Nonampullary Duodenal Epithelial Tumor:Two Cases of TubularAdenoc arcinoma and Adenoma,”Am J Gastroe nterol2017:112:1

[0009] Material 4:SE Kudo,K.Wakamura,N.Ikehara,et al.,"Diagnosis ofcolorectal Iesions with a novel endocytoscopic classification-a pilot study",E ndoscopy2011:43:869-75.

[0010] Paper 5:Y.Mori,S.Kudo,K.Wakamura,et al.,“Novel computer-aiddiagnostic system for colorectal lesions by using endocytoscopy(withvideos)”,Gastrointestinal Endoscopy 2015:81:621-6

[0011] Non-patent document 6: M. Misawa, S. Kudo, Y. Mori, et al., "Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-bandimaging endocytosis copy", Gast roenterolo gy2016; 150:1531-1532. Summary of the Invention

[0012] Problems to be solved by the invention

[0013] Endocytoscape can also capture non-magnified images with a lower magnification than super-magnified images. Therefore, in order to adapt computer-aided diagnosis systems to Endocytoscape, it is necessary to distinguish between super-magnified and non-magnified images within images captured by the endoscope. However, no technology exists to automatically distinguish between super-magnified and non-magnified images. Therefore, the system operator must identify the super-magnified image from the images captured by Endocytoscape as the target for epithelial status image analysis and input it into the system.

[0014] While it is conceivable to include a dedicated switch or button in the system input, adding such a switch or button is not preferred. On the other hand, if the system operator could automatically determine the presence of a super-magnified image, system operation would be simpler and would also reduce the burden on the patient. Therefore, the present disclosure aims to automatically distinguish super-magnified images from non-magnified images in a computer-aided diagnosis system that uses image analysis to analyze epithelial conditions.

[0015] Means of solving the problem

[0016] Because super-magnified images are images from a contact endoscope, they do not produce haloing from the light source. The present disclosure focuses on haloing from the light source and determines that an image is super-magnified if no halo is detected. This allows for automatic discrimination between super-magnified and non-magnified images, and enables automated selection of target images for image analysis in computer-assisted diagnostics.

[0017] The image analysis device of the present disclosure,

[0018] It is an image analysis device connected to an endoscope, including:

[0019] a target image determination unit that acquires an image from the endoscope and determines that the image is a target image using a halo area included in the image; and

[0020] The image analyzing unit analyzes the state of the epithelium imaged by the endoscope using the target image when the image is a target image.

[0021] The image analysis method disclosed herein,

[0022] The invention is an image analysis method performed by an image analysis device connected to an endoscope, wherein the image analysis device performs:

[0023] a target image determination step of acquiring an image from the endoscope and determining the image as a target image using a halo area contained in the image; and

[0024] The image analyzing step is to analyze the state of the epithelium imaged by the endoscope using the target image when the image is a target image.

[0025] The image analysis program disclosed herein is a program for implementing the various functional units of the image analysis device disclosed herein on a computer, and is a program for causing a computer to execute the various steps of the image analysis method disclosed herein, and can be stored in a computer-readable storage medium.

[0026] Beneficial effects

[0027] According to the present disclosure, a computer-assisted diagnosis system for analyzing epithelial conditions using super-magnified images can automatically distinguish between super-magnified and non-magnified images, thereby automatically selecting the target image for image analysis. In other words, the region of interest (ROI), which must be set as the analysis target in the computer-assisted diagnosis system, can be automatically selected. This simplifies system operation and reduces the burden on patients. Furthermore, since the target image for computer-assisted diagnosis is automatically selected, the time required to output predicted results for pathological tissue diagnosis can be shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An example of a computer diagnosis support system according to an embodiment is shown.

[0029] Figure 2 An example of the structure of the distal end portion of an endoscope is shown.

[0030] Figure 3 A first example of an image captured by the capturing device is shown.

[0031] Figure 4 A second example of an image captured by the capturing device is shown.

[0032] Figure 5 A third example of an image captured by the capturing device is shown.

[0033] Figure 6 A schematic diagram of a cell nucleus is shown. DETAILED DESCRIPTION

[0034] The following describes embodiments of the present disclosure in detail with reference to the accompanying drawings. The present disclosure is not limited to the embodiments described below. These embodiments are merely examples, and the present disclosure can be implemented in various modified and improved forms based on the knowledge of those skilled in the art. Furthermore, technical features with the same reference numerals in this specification and the accompanying drawings represent the same technical features.

[0035] Figure 1 An example of a computer-assisted diagnosis system according to an embodiment is shown. The computer-assisted diagnosis system according to this embodiment includes an image analysis device 10, a camera 24, and a display device 30. The image analysis device 10 includes a CPU (Central Processing Unit) 11 and a memory 12. The CPU 11 functions as a target image determination unit 111 and an image analysis unit 112. The display device 30 may also be included in the image analysis device 10.

[0036] Image analysis device 10 can be implemented by executing a computer program stored in memory 12. The computer program is a program for causing a computer to execute the various steps of the image analysis method disclosed herein. In the image analysis method disclosed herein, the target image determination step and the image analysis step are executed by image analysis device 10.

[0037] In the target image determination step, target image determination unit 111 acquires an image from the endoscope and determines it as a target image using the halo region contained in the image. If the image is a target image, image analysis unit 112 executes the image analysis step. In the image analysis step, image analysis unit 112 uses the target image to analyze the state of the epithelium imaged by the endoscope.

[0038] The imaging device 24 is an arbitrary imaging element mounted on the endoscope, and an example thereof is a CCD (Charge Coupled Device). The imaging device 24 has the function of capturing both moving and still images. Therefore, images captured by the imaging device 24 include not only moving images but also still images. When the CPU 11 acquires images captured by the imaging device 24, it displays them on the display device 30.

[0039] Figure 2 An example of the structure of the distal end of the endoscope is shown in FIG. A light guide lens 22 and an objective lens 23 are provided at the distal end of the endoscope 20. The objective lens 23 is provided at a convex portion of the distal end of the endoscope 20, and the light guide lens 22 is provided at a position lower than the objective lens 23.

[0040] Illumination light output from a light source device (not shown) is emitted from a light guide lens 22 via a light guide 21. An image of the epithelium of the lumen illuminated by the irradiation light passes through an objective lens 23 and is guided to an imaging device 24. Thus, an image of the epithelium of the lumen is captured by the imaging device 24.

[0041] The image captured by the camera 24 is transmitted to the image analyzing device 10 via the signal line 25. This transmission can also be sent to the image analyzing device 10 using a wireless communication function unit (not shown) installed in the camera 24. In addition, although more than one lens can be provided between the objective lens 23 and the camera 24, Figure 2 In addition, although Figure 2 2 shows an example in which the distal end portion of the endoscope 20 has a convex portion, but the present disclosure is not limited thereto. For example, the distal end portion of the endoscope 20 may be flat, and the objective lens 23 and the light guide lens 22 may be provided on the flat surface.

[0042] Figure 3 、 Figure 4 as well as Figure 5 , an example of an image captured by the imaging device 24 is shown. Figure 4 The image shown shows Figure 3 An image after a portion of an image is magnified and focused. Figure 5 The image shown shows Figure 4 A super-magnified image is obtained by further magnifying and focusing a part of the image. In order to apply the computer diagnosis-assisted system to the prediction of pathological tissue diagnosis, it is essential to observe the pathological tissue super-magnified to the cellular level. To this end, it is necessary to distinguish between super-magnified images and non-magnified images in the images taken by the endoscope 20; however, the endoscope 20 is generally capable of taking not only super-magnified images but also images such as Figure 3 as well as Figure 4 Non-magnified images at typical magnifications shown.

[0043] When the operator of the computer diagnosis assistance system finds a suspected lesion in the image displayed by the display device 30, Figure 3 、 4 As shown in FIG5 and FIG6 , still images are taken while zooming in sequentially. Figure 3 and Figure 4 The image shown includes the ROI and the parts outside it, so the ROI needs to be set before image analysis can be performed. Figure 5 The super-magnified image shown does not include the area outside the ROI, but is an image of the ROI itself.

[0044] The system operator needs to set the ROI for the non-magnified image, but since the super-magnified image is an image of the ROI itself, there is no need to set the ROI for the super-magnified image. Therefore, by automatically determining the super-magnified image, the image of the ROI for image analysis can be automatically selected.

[0045] exist Figure 2 When the imaging device 24 is shooting in the state where the objective lens 23 is not in contact with the epithelium, the image of the light guide lens 22 is reflected on the surface of the epithelium and is reflected into the imaging device 24. Therefore, in the image where the objective lens 23 is not in contact with the mucosal epithelium, as shown in FIG. Figure 3 and Figure 4 The area surrounded by the dot-dashed line in FIG. 1 shows an area where halo occurs.

[0046] On the other hand, when taking super-magnified images, Figure 2 The objective lens 23 shown is in contact with the epithelium, so the image of the light guide lens 22 reflected on the epithelial surface will not be reflected into the imaging device 24. In addition, the light incident on the imaging device 24 is all light that has passed through the cells of the epithelium. Figure 5 In the super-magnified image shown, no Figure 3 as well as Figure 4 In the halo generating area shown, the number of pixels in the halo area is less than a certain percentage. Here, the certain percentage is, for example, less than 0.0000077%.

[0047] Therefore, the target image determination unit 111 obtains an image from the imaging device 24 and uses the halo area included in the image to determine whether it is a super-magnified image obtained by capturing the transmitted light passing through the epithelial cells. Figure 3 as well as Figure 4 There is a halo area in the image shown, so the target image determination unit 111 determines that the image is not the target image. Figure 5 There is no halo region in the image shown, so the target image determination unit 111 determines that the image is the target image. Therefore, the present disclosure can automatically perform prediction of pathological tissue diagnosis of ROI by selecting a super-magnified image and performing image analysis on the image.

[0048] Here, video and still images are input from the endoscope 20 to the image analysis device 10. In the present disclosure, the image to be analyzed is a super-magnified image. Therefore, it is preferable that the target image determination unit 111 determines whether the image acquired from the endoscope 20 is a still image, and if so, determines whether it is the target image.

[0049] When the image acquired from the endoscope 20 is a super-magnified image, it is an image capturing the ROI. Therefore, if the image is a target image, the image analysis unit 112 stores the image determined to be the target image as the captured image of the ROI in the memory 12. Thus, the system of the present disclosure can efficiently collect ROI information.

[0050] The image analysis unit 112, when the image is a target image, uses the target image to perform image analysis to analyze the state of the epithelium photographed by the photographing device 24. The image analysis unit 112 uses the analysis results of the epithelial state to predict the pathological tissue diagnosis. The prediction of the pathological tissue diagnosis is, for example, the identification of non-tumors, adenomas, and cancers. The prediction of the pathological tissue diagnosis may also include sessile serrated adenoma / polyps (SSA / P) that may become tumors. The CPU 11 outputs the analysis results of the image analysis unit 112 to the display device 30, and the display device 30 displays the predicted results of the pathological tissue diagnosis. The CPU 11 further stores the analysis results of the image analysis unit 112 in the memory 12.

[0051] Preferably, machine learning is used for the prediction of pathological tissue diagnoses. This enables the prediction of pathological tissue diagnoses using a computer-aided diagnosis system without requiring specialized training. In this case, for the prediction of pathological tissue diagnoses, data serving as learning samples for each of non-tumor, adenoma, cancer, and SSA / P is provided to the image analysis device 10.

[0052] As machine learning, for example, SVM (Support Vector Machine), neural network, naive Bayes classifier, decision tree, cluster analysis, linear regression analysis, logistic regression analysis, and random forest can be used. The neural network can be deep learning using a multilayer neural network.

[0053] The image analysis unit 112 may also use a non-magnified image when analyzing an image. Figure 5 When using the super-magnified image shown Figure 3 as well as Figure 4 The non-magnified image also includes an area outside the ROI. Therefore, the image analysis unit 112 obtains the ROI area setting in the non-magnified image input to the image analysis device 10, and uses the image of the area defined by the area setting for image analysis.

[0054] Hereinafter, a specific example of determining whether a halo area exists in the target image determination unit 111 will be described.

[0055] The presence of halo regions is determined by extracting an image captured by the camera 24 and counting the number of pixels producing haloes within the extracted pixels. If the number of pixels producing haloes within the extracted pixels is less than a predetermined fixed ratio, the image is determined to be a super-magnified image, i.e., the image of the target for analysis. If the number of pixels producing haloes within the extracted pixels exceeds the predetermined fixed ratio, the image is determined to be a non-magnified image.

[0056] Here, extracting the image captured by the imaging device 24 means, for example, extracting Figures 3 to 5 The area surrounded by the dotted line is shown in . In addition, although the fixed ratio is arbitrary, for example, the above-mentioned 0.0000077% or less may be used.

[0057] Alternatively, for example, whether a pixel is a halo region can be determined based on whether the brightness exceeds a predetermined value. For example, if the individual color information (R value, G value, B value) of a pixel has 255 grayscale values, then a halo region is determined when each color value exceeds 240. This determination is not limited to this, as long as a white region can be extracted. For example, the brightness of white light synthesized from the individual color information (R value, G value, B value) can be used, or a color space represented by hue, saturation, and brightness can be used.

[0058] During endoscopic epithelial observation, the wavelength of the light emitted from the light guide lens 22 and the wavelength of the light captured by the imaging device 24 can differ. For example, epithelial observation can be performed using either white light or narrowband light (NBI: Narrow Band Imaging; BLI: Blue Laser Imaging). The light source for the light emitted from the light guide lens 22 can be a variety of light sources, including xenon, laser, halogen, and LED (Light Emitting Side). Therefore, it is preferable to set a threshold for determining halo regions based on the wavelength of the irradiation light emitted from the light guide lens 22 and the wavelength captured by the imaging device 24.

[0059] For example, when the irradiation light emitted from the light guide lens 22 is white light, the target image determination unit 111 determines that a halo region exists when each color information value (R value, G value, B value) is greater than or equal to 240. For example, when observing using narrow-band light (NBI: Narrow Band Imaging; BLI: Blue Laser Imaging), the target image determination unit 111 determines that a halo region exists when each color information value (R value, G value, B value) is greater than or equal to 200, greater than or equal to 240, or greater than or equal to 180.

[0060] Details of image analysis performed using the target image in the image analysis unit 112 will be described below.

[0061] Using image analysis of the target image, for example, texture analysis can be exemplified. In texture analysis, the image is extracted as Figure 5 The epithelial image shown by the dashed line is extracted and analyzed. While any texture analysis method is available, it is preferred to analyze local image features that can be used to identify objects or faces. Examples of such analysis methods include SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), and Haar-Like features.

[0062] The image analysis using the target image can be exemplified by analyzing the feature quantities acquired from the super-magnified image. The feature quantities acquired from the image are, for example, feature quantities of cell nuclei, blood vessels, and glandular lumens.

[0063] Figure 6 A schematic diagram of a cell nucleus is shown. Examples of cell nucleus characteristics include the long axis DL, the short axis DS, the circumference, the area, the roundness, and the color of the nucleus. Other cell nucleus characteristics may include eccentricity, chord-to-segment ratio, concave-convex shape, fractal dimension, line concentration, and density contrast.

[0064] When the feature amount of the cell nucleus is used, the image analysis unit 112 extracts the cell nucleus contained in the image. The method of extracting the cell nucleus is arbitrary, for example, by segmenting the cell nucleus region and removing artifacts. For the segmentation of the cell nucleus region, for example, the Otsu binarization method with the R component is used. In artifact removal, for example, continuous pixels of white pixels of the binarized image are set as one region, and the area, major axis and circularity are calculated for each region. The area is divided into a set range (for example, 30 μm 2 Up to 500 μm 2 ), with a major axis of a set value (e.g., 30 μm or less), and a roundness of a set value (e.g., 0.3 or greater) as the analysis target, while other regions are removed. The major axis and roundness are calculated, for example, using an elliptical approximation. When the number of extracted nuclei is less than a preset number (e.g., 30), they can be excluded from the feature quantity of the analysis target.

[0065] Although the characteristic value of the cell nucleus may be the characteristic value of a portion of the cell nuclei contained in the target image, it is preferred to measure the characteristic value of all the cell nuclei. Preferably, the characteristic value of the cell nucleus includes: an average value or a standard deviation calculated based on the characteristics of the cell nuclei contained in the target image.

[0066] Examples of vascular feature quantities include the maximum diameter of the largest vessel, the minimum-maximum diameter ratio of the largest vessel, and the proportion of the vascular region in the entire image. When using the vascular feature quantities, image analysis unit 112 extracts the vascular region contained in the image. The method for extracting the vascular region can be arbitrary, and for example, a linear image can be created, multiple linear images can be combined to generate a candidate vascular region image, and then non-vascular regions can be removed from the image.

[0067] The feature quantities of cell nuclei and blood vessels can be applied to image analysis targeting any organ such as the oral cavity, pharynx, mouth, esophagus, stomach, duodenum, jejunum, ileum, large intestine, trachea, bile duct, pancreatic duct, uterus, bladder, ureter, etc.

[0068] For the stomach and large intestine, glandular cavities can be observed in super-magnified images. Therefore, when predicting pathological tissue diagnoses of the stomach and large intestine, image analysis unit 112 preferably analyzes glandular lumen features. Examples of glandular lumen features include the major axis, minor axis, circumference, area, roundness, and color of the glandular lumen.

[0069] The villus structure of the duodenum, jejunum, and ileum can be observed in super-magnified images. Therefore, when predicting pathological histological diagnoses of the duodenum, jejunum, and ileum, the image analysis unit 112 preferably analyzes characteristic quantities of the villus structure. Examples of characteristic quantities of the villus structure include the long axis of the villus tip, the short axis of the villus tip, and the number of villi per field of view.

[0070] As described above, the image analysis unit 112 preferably analyzes the characteristic quantities of glandular cavities or villus structures in addition to cell nuclei and blood vessels in cylindrical epithelial regions, and analyzes the characteristic quantities of nuclei and blood vessels in multilayered squamous epithelium, trichome epithelium, etc.

[0071] Here, information regarding whether the image is focused on any of the cell nuclei, blood vessels, glandular lumens, and villus structures is not included in the image acquired by the endoscope 20. Therefore, the image analysis unit 112 preferably determines which of the cell nuclei, blood vessels, glandular lumens, and villus structures is being imaged before extracting their features. For example, the image analysis unit 112 extracts the cell nuclei, blood vessels, glandular lumens, and villus structures from the image and extracts feature quantities from the extracted objects. This reduces the computational complexity of image analysis and shortens the time required for predictive diagnosis of pathological tissues.

[0072] Here, glandular lumens are not normally observed in organs other than the stomach and large intestine. However, glandular lumens may appear in organs other than the stomach and large intestine due to tumor formation. Therefore, image analysis unit 112 preferably analyzes the characteristic quantities of glandular lumens in organs other than the stomach and large intestine.

[0073] Furthermore, normally, villus structures are not observed in organs other than the duodenum, jejunum, and ileum. However, due to tumor formation, villus structures may appear in organs other than the duodenum, jejunum, and ileum. Therefore, it is preferable that the image analysis unit 112 analyzes the characteristic amount of the villus structure in organs other than the duodenum, jejunum, and ileum.

[0074] As described above, since the present disclosure can automatically distinguish between super-magnified images and non-magnified images, it can automatically distinguish ROIs and can provide a computer diagnosis support system that automatically predicts pathological tissue diagnosis by using super-magnified images.

[0075] Explanation of symbols

[0076] 10: Image analysis device

[0077] 11: CPU

[0078] 111: Target image determination unit

[0079] 112: Image Analysis Unit

[0080] 12: Memory

[0081] 20: Endoscope

[0082] 21: Light guide

[0083] 22: Light guide lens

[0084] 23: Objective lens

[0085] 24: Shooting device

[0086] 25: Signal line

[0087] 30: Display device

Claims

1. Image analysis device, It is an image analysis device connected to an endoscope, including: a target image determination unit that acquires an image from the endoscope and determines that the image is a super-magnified image captured at a magnification of 380 times or more based on the fact that the number of pixels of a halo region included in the image is less than a certain ratio; and An image analysis unit analyzes the state of the epithelium imaged by the endoscope using the super-magnified image when the image is a super-magnified image; the super-magnified image is an image captured using transmitted light passing through cells of the epithelium, and the super-magnified image is an image captured when the objective lens of the endoscope is in contact with the epithelium. 2 . The image analysis device according to claim 1 , wherein the target image determination unit stores the image determined to be the super-magnified image in the memory as an image captured in the region of interest. 3 . The image analysis device according to claim 1 , wherein the super-magnified image is an image captured of at least one of a cell nucleus, a blood vessel, a glandular cavity, and a villus structure.

4. The image analysis device according to claim 3, wherein the image analysis unit includes a process for extracting a feature amount of at least one of a cell nucleus, a blood vessel, a glandular cavity, and a villus structure from the super-magnified image, and analyzing the state of the epithelium using the extraction result. 5 . The image analysis device according to claim 3 , wherein the image analysis unit determines whether the image captures any one of a cell nucleus, a blood vessel, a glandular cavity, and a villus structure. 6 . The image analysis device according to claim 1 , wherein the image analysis unit uses the analysis result of the epithelial state to perform prediction of pathological tissue diagnosis. 7 . The image analysis device according to claim 6 , wherein the prediction of the pathological tissue diagnosis is identification of non-tumor, adenoma, or cancer.

8. The image analysis device according to any one of claims 1 to 7, wherein the target image determination unit determines whether the image acquired from the endoscope is a still image, and if the image is a still image, determines whether it is a super-magnified image. 9 . A program for causing a computer to implement each functional unit included in the image analysis device according to claim 1 .

10. An image analysis method performed by an image analysis device, The invention is an image analysis method performed by an image analysis device connected to an endoscope, wherein the image analysis device performs: a target image determination step of acquiring an image from the endoscope and determining that the image is a super-magnified image captured at a magnification of 380 times or more based on the fact that the number of pixels in the halo region included in the image is below a certain ratio; and An image analysis step, when the image is a super-magnified image, uses the super-magnified image to analyze the state of the epithelium imaged by the endoscope; the super-magnified image is an image taken using transmitted light passing through the cells of the epithelium, and the super-magnified image is an image taken when the objective lens of the endoscope is in contact with the epithelium.

Images