Image processing method, image processing device, and image processing program

Abstract

To provide an image processing method for analyzing the choroidal vasculature.SOLUTION: An image processing method includes acquiring a fundus image, extracting a first area including a first feature from the fundus image, extracting a second area including a second feature different from the first feature from the fundus image, and generating a combined image in which the extracted first area and the extracted second area are combined.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, and an image processing program. [Background technology]

[0002] US Patent No. 10,136,812 discloses an optical coherence tomography device that selectively visualizes the choroidal vascular network. An image processing method for analyzing choroidal blood vessels is desired. Summary of the Invention

[0003] An image processing method according to a first aspect of the disclosed technology includes acquiring a fundus image, extracting a first region having a first feature from the fundus image, extracting a second region having a second feature different from the first feature from the fundus image, and generating a composite image by combining the extracted first and second regions.

[0004] An image processing device according to a second aspect of the disclosed technology includes an image acquisition unit that acquires a fundus image, a first extraction unit that extracts linear portions of blood vessels from the fundus image, a second extraction unit that extracts mass portions of blood vessels from the fundus image, and a blood vessel visualization unit that integrates the extracted images of linear portions and mass portions to generate a blood vessel image in which the blood vessels are visualized.

[0005] The image processing program of the third aspect of the technology disclosed herein causes a computer to function as an image acquisition unit that acquires a fundus image, a first extraction unit that extracts linear portions of blood vessels from the fundus image, a second extraction unit that extracts mass portions of blood vessels from the fundus image, and a blood vessel visualization unit that integrates the extracted images of linear portions and mass portions to generate a blood vessel image in which the blood vessels are visualized. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of an ophthalmology system 100. [Figure 2] 1 is a schematic diagram showing the overall configuration of an ophthalmologic apparatus 110. FIG. [Figure 3] FIG. 2 is a block diagram of the electrical configuration of the management server 140. [Figure 4] FIG. 2 is a functional block diagram of a CPU 262 of the management server 140. [Figure 5] FIG. 2 is a functional block diagram of an image processing control unit 206 of a CPU 262 of the management server 140. [Figure 6] 10 is a flowchart of an image processing program. [Figure 7] 7 is a flowchart of the choroidal vessel analysis process in step 304 of FIG. 6. [Figure 8A] 12 is a schematic diagram showing an example of extraction of linear portions 12V1, 12V2, 12V3, and 12V4 of choroidal blood vessels. FIG. [Figure 8B] 12 is a schematic diagram showing an example of extraction of ampullae 12E1, 12E2, 12E3, and 12E4 of choroidal blood vessels. [Figure 8C] This is a schematic diagram of the linear portions 12V1, 12V2, 12V3, and 12V4 of the choroidal blood vessels and the ampulla portions 12E1, 12E2, 12E3, and 12E4, respectively, being connected. [Figure 9] FIG. 5 is a schematic diagram showing a display screen 500. [Figure 10] This is a schematic diagram showing a case where each RG color fundus image of a specific area 12V3A displayed in chronological order in the follow-up observation area 570 is combined with each choroidal blood vessel extraction image of a specific area 12V3B taken on the same date. [Figure 11] This is a schematic diagram showing a case where each of the choroidal vessel extraction images of specific area 12V3B displayed in chronological order in the follow-up observation area 570 is combined with a contour-enhanced image extracted from an RG color fundus image of specific area 12V3A taken on the same date. [Figure 12] FIG. 10 is a diagram showing a choroidal blood vessel image CLA. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, the present embodiment will be described in detail with reference to the drawings.

[0008] The configuration of an ophthalmologic system 100 will be described with reference to Fig. 1. As shown in Fig. 1, the ophthalmologic system 100 includes an ophthalmologic apparatus 110, a management server apparatus (hereinafter referred to as "management server") 140, and a display device (hereinafter referred to as "viewer") 150. The ophthalmologic apparatus 110 acquires fundus images. The management server 140 stores, in association with the patient IDs, multiple fundus images and axial lengths obtained by photographing the funduses of multiple patients using the ophthalmologic apparatus 110. The viewer 150 displays the fundus images and analysis results acquired by the management server 140.

[0009] The viewer 150 includes a display 156 that displays fundus images and analysis results acquired by the management server 140, a mouse 155M, and a keyboard 155K that are used for operation.

[0010] The ophthalmic apparatus 110, the management server 140, and the viewer 150 are connected to each other via a network 130. The viewer 150 is a client in a client-server system, and multiple viewers 150 are connected via the network. Multiple management servers 140 may also be connected via the network to ensure system redundancy. Alternatively, if the ophthalmic apparatus 110 has an image processing function and the image viewing function of the viewer 150, fundus images can be acquired, processed, and viewed in a standalone state. Alternatively, if the management server 140 has the image viewing function of the viewer 150, fundus images can be acquired, processed, and viewed in a configuration of the ophthalmic apparatus 110 and the management server 140.

[0011] In addition, other ophthalmic devices (examination devices for visual field measurement, intraocular pressure measurement, etc.) and diagnostic support devices that perform image analysis using AI (Artificial Intelligence) may be connected to the ophthalmic device 110, the management server 140, and the viewer 150 via the network 130.

[0012] Next, the configuration of the ophthalmologic apparatus 110 will be described with reference to Fig. 2. For convenience of description, a scanning laser ophthalmoscope will be referred to as "SLO", and optical coherence tomography will be referred to as "OCT".

[0013] When the ophthalmic apparatus 110 is placed on a horizontal plane, the horizontal direction is defined as the "X direction," the vertical direction relative to the horizontal plane is defined as the "Y direction," and the direction connecting the center of the pupil of the anterior segment of the subject's eye 12 and the center of the eyeball is defined as the "Z direction." Therefore, the X direction, Y direction, and Z direction are perpendicular to each other.

[0014] The ophthalmologic apparatus 110 includes an imaging device 14 and a control device 16. The imaging device 14 is equipped with an SLO unit 18 and an OCT unit 20, and acquires a fundus image of the fundus of the subject's eye 12. Hereinafter, a two-dimensional fundus image acquired by the SLO unit 18 will be referred to as an SLO image. Also, a tomographic image or a front image (en-face image) of the retina created based on OCT data acquired by the OCT unit 20 will be referred to as an OCT image.

[0015] The control device 16 comprises a computer having a CPU (Central Processing Unit) 16A, a RAM (Random Access Memory) 16B, a ROM (Read-Only Memory) 16C, and an input / output (I / O) port 16D.

[0016] The control device 16 includes an input / display device 16E connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphic user interface that displays an image of the subject's eye 12 and receives various instructions from the user. An example of the graphic user interface is a touch panel display.

[0017] The control device 16 also includes an image processing device 17 connected to the I / O port 16D. The image processing device 17 generates an image of the subject's eye 12 based on data obtained by the photographing device 14. The control device 16 is connected to a network 130 via a communication interface (not shown).

[0018] 2, the control device 16 of the ophthalmic apparatus 110 includes the input / display device 16E, but the technology of the present disclosure is not limited to this. For example, the control device 16 of the ophthalmic apparatus 110 may not include the input / display device 16E, but may include a separate input / display device that is physically independent from the ophthalmic apparatus 110. In this case, the display device includes an image processing processor unit that operates under the control of the CPU 16A of the control device 16. The image processing processor unit may display an SLO image or the like based on an image signal instructed to be output by the CPU 16A.

[0019] The imaging device 14 operates under the control of the CPU 16A of the control device 16. The imaging device 14 includes an SLO unit 18, an imaging optical system 19, and an OCT unit 20. The imaging optical system 19 includes a first optical scanner 22, a second optical scanner 24, and a wide-angle optical system 30.

[0020] The first optical scanner 22 performs two-dimensional scanning in the X and Y directions with the light emitted from the SLO unit 18. The second optical scanner 24 performs two-dimensional scanning in the X and Y directions with the light emitted from the OCT unit 20. The first optical scanner 22 and the second optical scanner 24 may be optical elements that can deflect a light beam, such as a polygon mirror or a galvanometer mirror. Alternatively, a combination of these may be used.

[0021] The wide-angle optical system 30 includes an objective optical system (not shown in FIG. 2) having a common optical system 28 , and a combining unit 26 that combines the light from the SLO unit 18 and the light from the OCT unit 20 .

[0022] The objective optical system of the common optical system 28 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph the retina in the peripheral part of the fundus as well as the center of the fundus.

[0023] When a system including an elliptical mirror is used, the system using the elliptical mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may be used. The disclosures of International Publication WO2016 / 103484 and International Publication WO2016 / 103489 are each incorporated herein by reference in their entirety.

[0024] The wide-angle optical system 30 enables observation of the fundus over a wide field of view (FOV) 12A. The FOV 12A indicates the range that can be photographed by the imaging device 14. The FOV 12A can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal illumination angle and an external illumination angle. The external illumination angle is the illumination angle of the light beam irradiated from the ophthalmic device 110 to the subject's eye 12, determined with the pupil 27 as the reference. The internal illumination angle is the illumination angle of the light beam irradiated to the fundus F, determined with the eyeball center O as the reference. The external illumination angle and the internal illumination angle correspond to each other. For example, if the external illumination angle is 120 degrees, the internal illumination angle corresponds to approximately 160 degrees. In this embodiment, the internal illumination angle is 200 degrees.

[0025] Here, an SLO fundus image captured at an internal illumination angle of 160 degrees or more is referred to as a UWF-SLO fundus image. UWF stands for Ultra Wide Field.

[0026] The SLO system is realized by a control device 16, an SLO unit 18, and an imaging optical system 19 shown in Fig. 2. The SLO system includes a wide-angle optical system 30, and therefore enables fundus imaging with a wide FOV 12A.

[0027] The SLO unit 18 includes a B (blue light) light source 40, a G (green light) light source 42, an R (red light) light source 44, and an IR (infrared (e.g., near-infrared) light) light source 46, as well as optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light from the light sources 40, 42, 44, and 46 and guide them into a single optical path. The optical systems 48, 50, and 56 are mirrors, and the optical systems 52 and 54 are beam splitters. The B light is reflected by the optical system 48, passes through the optical system 50, and is reflected by the optical system 54; the G light is reflected by the optical systems 50 and 54; the R light is transmitted through the optical systems 52 and 54; and the IR light is reflected by the optical systems 52 and 56 and is each guided into a single optical path.

[0028] The SLO unit 18 is configured to be switchable between a light source that emits laser light of different wavelengths, or a combination of light sources that emit light, such as a mode that emits G light, R light, and B light, and a mode that emits infrared light. In the example shown in FIG. 2, four light sources are provided: a B light (blue light) light source 40, a G light source 42, an R light source 44, and an IR light source 46; however, the technology of the present disclosure is not limited to this. For example, the SLO unit 18 may further include a white light source and emit light in various modes, such as a mode that emits only white light.

[0029] Light incident on the photographing optical system 19 from the SLO unit 18 is scanned in the X and Y directions by the first optical scanner 22. The scanning light passes through the wide-angle optical system 30 and the pupil 27 and is irradiated onto the posterior segment (fundus) of the subject's eye 12. The light reflected by the fundus passes through the wide-angle optical system 30 and the first optical scanner 22 and is incident on the SLO unit 18.

[0030] The SLO unit 18 includes a beam splitter 64 that reflects B light and transmits all light except B light from the posterior segment (fundus) of the subject's eye 12, and a beam splitter 58 that reflects G light and transmits all light except G light from the light that has passed through the beam splitter 64. The SLO unit 18 includes a beam splitter 60 that reflects R light and transmits all light except R light from the light that has passed through the beam splitter 58. The SLO unit 18 includes a beam splitter 62 that reflects IR light from the light that has passed through the beam splitter 60. The SLO unit 18 includes a B light detecting element 70 that detects B light reflected by the beam splitter 64, a G light detecting element 72 that detects G light reflected by the beam splitter 58, an R light detecting element 74 that detects R light reflected by the beam splitter 60, and an IR light detecting element 76 that detects IR light reflected by the beam splitter 62.

[0031] Light (reflected light reflected by the fundus) incident on the SLO unit 18 via the wide-angle optical system 30 and the scanner 22 is reflected by the beam splitter 64 and received by the B light detection element 70 in the case of B light, and transmitted through the beam splitter 64, reflected by the beam splitter 58, and received by the G light detection element 72 in the case of G light. The incident light is transmitted through the beam splitters 64 and 58 in the case of R light, reflected by the beam splitter 60, and received by the R light detection element 74. The incident light is transmitted through the beam splitters 64, 58, and 60 in the case of IR light, reflected by the beam splitter 62, and received by the IR light detection element 76. The image processing device 17, which operates under the control of the CPU 16A, generates a UWF-SLO image using signals detected by the B light detection element 70, the G light detection element 72, the R light detection element 74, and the IR light detection element 76. Examples of the B photodetector element 70, the G photodetector element 72, the R photodetector element 74, and the IR photodetector element 76 include photodiodes (PDs) and avalanche photodiodes (APDs). The B photodetector element 70, the G photodetector element 72, the R photodetector element 74, and the IR photodetector element 76 correspond to the "image acquisition unit" of the technology disclosed herein. In the SLO unit 18, light reflected (scattered) by the fundus, which is the subject, reaches the photodetector element through the first optical scanner 22. Therefore, the light always returns to the same position, i.e., the position where the B photodetector element 70, the G photodetector element 72, the R photodetector element 74, and the IR photodetector element 76 are located. Therefore, the photodetector element does not need to be configured as a planar (two-dimensional) element like an area sensor; point-like (zero-dimensional) detectors such as PDs or APDs are optimal in this embodiment. However, instead of being limited to PDs or APDs, line sensors (one-dimensional) or area sensors (two-dimensional) can also be used.

[0032] UWF-SLO images include UWF-SLO images (G-color fundus images) obtained by photographing the fundus in G color, UWF-SLO images (R-color fundus images) obtained by photographing the fundus in R color, UWF-SLO images (B-color fundus images) obtained by photographing the fundus in B color, and UWF-SLO images (IR fundus images) obtained by photographing the fundus in IR color.

[0033] The control device 16 also controls the light sources 40, 42, and 44 to emit light simultaneously. By simultaneously photographing the fundus of the subject's eye 12 with B light, G light, and R light, a G-color fundus image, an R-color fundus image, and a B-color fundus image, each of which corresponds to each other, are obtained. An RGB color fundus image is obtained from the G-color fundus image, the R-color fundus image, and the B-color fundus image. The control device 16 controls the light sources 42 and 44 to emit light simultaneously, and by simultaneously photographing the fundus of the subject's eye 12 with G light and R light, a G-color fundus image and an R-color fundus image, each of which corresponds to each other, are obtained. By mixing the G-color fundus image and the R-color fundus image at a predetermined mixing ratio, an RG color fundus image is obtained.

[0034] UWF-SLO images also include UWF-SLO images (videos) taken with ICG fluorescence. When indocyanine green (ICG) is injected into a blood vessel, it reaches the fundus, first the retina, then the choroid, and passes through the choroid. UWF-SLO images (videos) are moving images from the time indocyanine green (ICG) is injected into a blood vessel and reaches the retina to after it has passed through the choroid.

[0035] The image data of the B color fundus image, G color fundus image, R color fundus image, IR fundus image, RGB color fundus image, RG color fundus image, and UWF-SLO image are sent from the ophthalmic device 110 to the management server 140 via a communication IF not shown.

[0036] The OCT system is realized by the control device 16, OCT unit 20, and imaging optical system 19 shown in Fig. 2. The OCT system includes a wide-angle optical system 30, which enables fundus imaging with a wide FOV 12A, similar to the above-mentioned SLO fundus imaging. The OCT unit 20 includes a light source 20A, a sensor (detection element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0037] Light emitted from the light source 20A is branched by the first optical coupler 20C. One of the branched beams is collimated by the collimating lens 20E as measurement light and then enters the imaging optical system 19. The measurement light is scanned in the X and Y directions by the second optical scanner 24. The scanning light passes through the wide-angle optical system 30 and the pupil 27 and is irradiated onto the fundus. The measurement light reflected by the fundus passes through the wide-angle optical system 30 and the second optical scanner 24 and enters the OCT unit 20, and then passes through the collimating lens 20E and the first optical coupler 20C and enters the second optical coupler 20F.

[0038] The other light beam emitted from the light source 20A and branched by the first optical coupler 20C is incident as reference light on the reference optical system 20D, passes through the reference optical system 20D, and then enters the second optical coupler 20F.

[0039] The light beams incident on the second optical coupler 20F, i.e., the measurement light beam reflected from the fundus and the reference light beam, interfere with each other to generate interference light. The interference light beam is received by the sensor 20B. The image processing device 17, which operates under the control of the image processing controller 206, generates OCT images such as tomographic images and en-face images based on the OCT data detected by the sensor 20B.

[0040] Here, an OCT fundus image obtained by capturing an image at an internal illumination angle of 160 degrees or more is referred to as a UWF-OCT image.

[0041] Image data of the UWF-OCT image is sent from the ophthalmologic apparatus 110 to the management server 140 via a communication IF (not shown) and stored in the storage device 254 .

[0042] In this embodiment, the light source 20A is exemplified as a wavelength-swept type SS-OCT (Swept-Source OCT), but various types of OCT systems may also be used, such as SD-OCT (Spectral-Domain OCT) and TD-OCT (Time-Domain OCT).

[0043] Next, the configuration of the electrical system of the management server 140 will be described with reference to FIG. 3. As shown in FIG. 3, the management server 140 includes a computer main unit 252. The computer main unit 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268. The input / output (I / O) port 268 is connected to a storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258. The storage device 254 is configured, for example, with a non-volatile memory. The input / output (I / O) port 268 is connected to the network 130 via the communication interface (I / F) 258. Therefore, the management server 140 can communicate with the ophthalmic device 110, the axial length measuring device 120, and the viewer 150. The storage device 254 stores an image processing program, which will be described later. The image processing program may also be stored in the ROM 264.

[0044] The management server 140 stores the data received from the ophthalmic apparatus 110 and the axial length measuring device 120 in the storage device 254 .

[0045] Next, various functions realized by the CPU 262 of the management server 140 executing the image processing program will be described with reference to Fig. 4. The image processing program has a display control function, an image processing control function, and a processing function. When the CPU 262 executes the image processing program having these functions, the CPU 262 functions as a display control unit 204, an image processing control unit 206, and a processing unit 208, as shown in Fig. 4.

[0046] The image processing control unit 206 corresponds to the "first extraction unit," "second extraction unit," "blood vessel visualization unit," and "choroidal blood vessel image generation unit" of the technology of the present disclosure.

[0047] Next, various functions of the image processing control unit 206 will be described with reference to Fig. 5. The image processing control unit 206 functions as a fundus image processing unit 2060 that performs image processing such as generating an image in which choroidal blood vessels and the like are sharpened from a fundus image, and a choroidal blood vessel analysis unit 2062 that performs image processing such as extracting each of the linear portion and the dilated portion (lump portion) of the choroid. The linear portion corresponds to the "first feature" of the technology of the present disclosure, and the dilated portion corresponds to the "second feature" of the technology of the present disclosure.

[0048] Next, the image processing by the management server 140 will be described in detail with reference to Fig. 6. The image processing (image processing method) shown in the flowchart of Fig. 6 is realized by the CPU 262 of the management server 140 executing an image processing program.

[0049] In step 300, the image processing control unit 206 acquires UWF-SLO images from the storage device 254. In step 302, the image processing control unit 206 creates a choroidal vessel image in which choroidal vessels are extracted from the acquired UWF-SLO images (the red and green fundus images). Because red light has a long wavelength, it passes through the retina and reaches the choroid. Therefore, the red fundus image contains information about blood vessels present in the retina (retinal vessels) and blood vessels present in the choroid (choroidal vessels). In contrast, green light has a shorter wavelength than red light and therefore only reaches the retina. Therefore, the green fundus image contains only information about blood vessels present in the retina (retinal vessels). Therefore, a choroidal vessel image CLA can be obtained by extracting retinal vessels from the green fundus image and removing retinal vessels from the red fundus image. The red fundus image corresponds to the "red light captured image" of the technology disclosed herein.

[0050] The specific processing executed by the image processing control unit 206 in step 302 will now be described. First, the image processing control unit 206 performs denoising processing to remove noise from each of the G-color fundus image and the R-color fundus image, using a median filter or the like. Then, the image processing control unit 206 extracts retinal blood vessels from the green fundus image by performing black hat filtering on the green fundus image after noise removal. Then, the image processing control unit 206 removes the retinal blood vessels by inpainting, which uses the position information of the retinal blood vessels extracted from the G-colored fundus image to paint the retinal blood vessel structure of the R-colored fundus image with the same value as the surrounding pixels. This process removes the retinal blood vessels from the R-colored fundus image, generating an image in which only the choroidal blood vessels are visualized. Next, the image processing control unit 206 removes low-frequency components from the red fundus image after the inpainting process. To remove the low-frequency components, well-known image processing such as frequency filtering or spatial filtering is applied.

[0051] Finally, the image processing control unit 206 performs contrast limited adaptive histograph equalization on the image data of the red fundus image from which the retinal blood vessels have been removed and the choroidal blood vessels remain, thereby enhancing the choroidal blood vessels in the red fundus image. By performing the series of processes in step 302, a choroidal blood vessel image CLA shown in FIG. 12 is created. The created choroidal blood vessel image CLA is stored in the storage device 254.

[0052] In the above example, the choroidal blood vessel image CLA is generated from a red fundus image and a green fundus image. However, the image processing control unit 206 may generate the choroidal blood vessel image CLA from a green fundus image and an IR fundus image. Alternatively, the image processing control unit 206 may generate the choroidal blood vessel image CLA from a blue fundus image and a red or IR fundus image.

[0053] Furthermore, a choroidal vascular image CLA may be generated from the UWF-SLO image (moving image) 510. As described above, the UWF-SLO image (moving image) 510 is a moving image from when indocyanine green (ICG) is injected into a blood vessel and reaches the retina to after it has passed through the choroid. The choroidal vascular image CLA may be generated from the moving image of the period from when indocyanine green (ICG) passes through the retina to when it passes through the choroid.

[0054] In step 304, a choroidal vessel analysis process is performed in which the process of extracting linear portions of choroidal vessels and the process of extracting ampulla portions are performed independently. The positions of vortex veins, which are part of choroidal vessels, are extracted from the extracted linear portions and ampulla portions. Anatomically, vortex veins are vascular regions where choroidal vessels are concentrated, and are drainage routes for blood that has flowed into the eyeball. Vortex veins are part of choroidal vessels, and there are three to seven vortex veins in the eyeball, located in the peripheral part of the fundus (near the equator of the eyeball). In the fundus image, the positions of vortex veins are recognized as a mass-like center and multiple linear portions connecting to the mass-like center. The choroidal vessel analysis in step 304 will be described in detail below.

[0055] In step 306, the analysis data obtained in the choroidal blood vessel analysis processing in step 304 is output to the storage device 254 of the management server 140. In step 308, the display control unit 2044 generates a display screen 500 (described later) that reflects patient attribute information corresponding to the patient ID (patient name, age, information on whether each fundus image is from the right eye or left eye, axial length, visual acuity, date and time of photography, etc.) along with the image from which the choroidal blood vessels have been extracted, and displays it on the display 256 of the management server 140, thereby completing the processing.

[0056] The display screen 500 is stored in the storage device 254 of the management server 140. The display screen 500 stored in the storage device 254 of the management server 140 is transmitted to the viewer 150 in response to an operation from the viewer 150, and is output to the display 156 of the viewer 150 in a viewable state.

[0057] 6 may be executed by the CPU 16A included in the control device 16 of the ophthalmic apparatus 110. When the CPU 16A of the ophthalmic apparatus 110 executes the process, the display image 500 is displayed on the display of the ophthalmic apparatus, and the display image 500 is stored in the storage device of the management server 140.

[0058] 6 may be executed by a CPU included in the viewer 150. When the CPU of the viewer 150 executes the process, the display image 500 is displayed on the display 156 of the viewer 150, and the display image 500 is stored in the storage device of the viewer 150 and the storage device of the management server 140, respectively.

[0059] FIG. 7 is a flowchart showing details of the choroidal vessel analysis process (step 304) shown in FIG. 6. In step 400, a first vessel extraction process is performed to extract linear portions from the choroidal vessel image CLA. In step 400, first, line enhancement processing is performed on the analysis image, which is the choroidal vessel image CLA shown in FIG. 12. This line enhancement processing enhances linear choroidal vessels. The line-enhanced image is then binarized. Next, image processing is performed to extract linear portions from the line-enhanced choroidal vessel image CLA.

[0060] Line enhancement processing is processing that enhances linear structures by, for example, Hessian analysis using a Hessian matrix. Hessian analysis determines whether a local structure in an image is a point, a line, or a surface by analyzing the eigenvalues ​​of a Hessian matrix whose elements are second-order partial differential coefficients calculated using a second-order differential kernel for a predetermined filter such as a Gaussian kernel.

[0061] In addition to the above-described line enhancement process, the linear portions of the choroidal vessels may be enhanced using a Gabor filter that extracts the direction of contours contained in an image, or a graph cut that separates and extracts linear portions from other portions. Edge enhancement processes such as a Laplacian filter or unsharp mask may also be used.

[0062] By the first blood vessel extraction process in step 400, linear portions of choroidal blood vessels are extracted as shown in Fig. 8A. In Fig. 8A, linear portions 12V1, 12V2, 12V3, and 12V4 are displayed clearly distinguished from other areas of the fundus.

[0063] In step 402, a second blood vessel extraction process is performed to extract dilation of choroidal vessels from the choroidal vessel image CLA. In the second blood vessel extraction process, the analysis image is first binarized. Then, a region where a predetermined number of white pixels are connected is extracted from the binarized choroidal vessel image CLA as a dilation of choroidal vessels. The predetermined number or the size of the region is a number set in advance based on the size of the vortex vein (e.g., standard data for the choroid). This extraction process may be performed by Hessian analysis using a Hessian matrix to detect irregularities in the image and extract the convex portions as dilation. Hessian analysis corresponds to the "image processing filter that extracts only lumpy portions" in the technology disclosed herein. The linear portion of the choroidal blood vessel may also be extracted along with the pixel corresponding to the ampulla. However, since the linear portion and ampulla are integrated by the data integration process described below, this does not affect the extraction of the vortex vein position.

[0064] The second blood vessel extraction process in step 402 extracts the choroidal vascular ampullae as shown in FIG. 8B. FIG. 8B shows an example in which four vortex veins are present in the eyeball (usually four to six), with ampullae 12E1, 12E2, 12E3, and 12E4 clearly distinguished from the rest of the fundus. Ampullae 12E1, 12E2, 12E3, and 12E4 can also be expressed as the points at which vortex veins present in the peripheral part of the fundus enter the sclera (the entrances to the choroid side of the vortex veins running outside the eyeball). The binarization process for the entire choroidal blood vessel image CLA in step 402 makes it possible to extract the choroidal ampullae that could not be extracted by the linear portion extraction process in the first blood vessel extraction process in step 400.

[0065] 7, the order of the first blood vessel extraction process and the second blood vessel extraction process may be reversed, with the second blood vessel extraction process being performed in step 400 and the first blood vessel extraction process being performed in step 402. This is because the first blood vessel extraction process and the second blood vessel extraction process are independent processes that are not related to each other.

[0066] In step 404, as shown in FIG. 8C , data integration is performed to integrate the linear portion resulting from the first blood vessel extraction process and the dilation portion resulting from the second blood vessel extraction process. That is, the linear portion image from which the linear portion obtained in step 400 is extracted and the dilation portion image from which the dilation portion is extracted are combined to generate a single choroidal blood vessel image. Then, the process proceeds to step 306 in FIG. 7 , where the process of analyzing the combined image begins. The choroidal blood vessel image shown in FIG. 8C corresponds to the "composite image" of the technology of the present disclosure. In Figure 8C, linear portions 12V1, 12V2, 12V3, and 12V4 and ampulla portions 12E1, 12E2, 12E3, and 12E4 are displayed in a combined state. Note that in each of Figures 8A, 8B, and 8C, the background image, which is the area where the fundus is not displayed, is drawn in white to clearly display the binary image from which the choroidal blood vessels have been extracted. However, it is also possible to draw only the outlines between the binary image and the background image in white to make the binary image clearly visible. A display screen 500, which will be described later, shows the case where the background image, which is the area where the fundus is not displayed, is drawn in white.

[0067] 9 is a schematic diagram showing a display screen 500 displayed on the display 256 of the management server 140. The display image 500 may be displayed on the display 156 of the viewer 150 or the input / display device 16E of the ophthalmologic apparatus 110, in addition to the display 256 of the management server 140.

[0068] 9, the display screen 500 has an information display area 502 and an image display area 504. The information display area 502 has a patient ID display area 512, a patient name display area 514, an age display area 516, a right eye / left eye display area 518, and an axial length display area 522.

[0069] The image display area 504 has a latest image display area 550 that displays the latest image (in FIG. 9, the fundus image taken on July 16, 2019), a previous image display area 560 that displays the previous image taken before the latest image (in FIG. 9, the fundus image taken on April 16, 2019), a follow-up observation area 570 that displays changes in the fundus over time, and a remarks field 580 that displays notes on treatment and diagnosis entered by the user. The latest image display area 550 has a photography date display area 552 at the top, and displays an RG color fundus image 554 and a choroidal vessel extraction image 556, which is a binary image in which linear portions 12V1, 12V2, 12V3, and 12V4 of the choroidal vessels and ampulla portions 12E1, 12E2, 12E3, and 12E4 are respectively combined.

[0070] The previous image display area 560 has a shooting date display area 562 at the top, and displays an RG color fundus image 564 and a choroidal blood vessel extraction image 566, which is a binary image in which linear portions 12V1, 12V2, 12V3, and 12V4 of the choroidal blood vessels and ampulla portions 12E1, 12E2, 12E3, and 12E4 are respectively combined.

[0071] The latest image display area 550 and the previous image display area 560 may each display a choroidal angiography image (ICG) or an optical coherence tomography angiography (OCTA) image instead of the RG color fundus images 554, 564 and the choroidal blood vessel extraction images 556, 566. The RG color fundus images 554, 564, ICG, and OCTA may each be displayed not only in 2D but also in 3D. The images displayed in the latest image display area 550 and the previous image display area 560 can be selected from a menu that is displayed by turning on each of display switch icons 558, 568.

[0072] The follow-up observation area 570 displays changes over time in the specific region 12V3A of the RG color fundus images 554, 564 and the specific region 12V3B of the choroidal vessel extraction images 556, 566. The follow-up observation area 570 includes a latest image display area 576 that displays the latest image of each of the specific regions 12V3A, 12V3B, a previous image display area 574 that displays the previous image that was captured before the latest image of each of the specific regions 12V3A, 12V3B, and a previous-previous image display area 572 that displays the previous image that was captured before the previous image of each of the specific regions 12V3A, 12V3B (in FIG. 9, the fundus image captured on January 16, 2019). The follow-up observation area 570 also includes a time-series blood vessel diameter display area 578 at the bottom that displays changes over time in the blood vessel diameter of the ampulla 12E3 and the peripheral portion (linear portion 12V3) at the time of capturing the latest image, the previous image, and the previous-previous image. In Figure 9, three images taken at three different times, namely the latest, the previous, and the time before that, are displayed in the follow-up observation area 570, but this is not limited to three images, and fundus images taken at four or more different dates and times may be displayed in chronological order.

[0073] 10 is a schematic diagram showing a case where each RG color fundus image of a specific area 12V3A, which is displayed in chronological order in the follow-up observation area 570, is combined with each choroidal blood vessel extraction image of a specific area 12V3B, which was taken on the same date. The follow-up observation area 570 may display a combined image such as that shown in FIG.

[0074] FIG. 11 is a schematic diagram illustrating a case in which an edge-enhanced image extracted from an RG color fundus image in the specific region 12V3A captured on the same date is superimposed on each of the choroidal vessel extraction images in the specific region 12V3B displayed in chronological order in the follow-up observation area 570. The composite image shown in FIG. 11 may be displayed in the follow-up observation area 570. The image displayed in the follow-up observation area 570 can be selected from a menu displayed by turning on the display switch icon 582. The edge-enhanced image can be generated by applying a known technique, such as a Sobel filter, to the RG color fundus image or the choroidal vessel extraction image. By superimposing the edge-enhanced image on the binary image of the choroidal vessel extraction image, the linear portion 12V3 and the ampulla 12E3 of the choroidal vessels can be more clearly identified than in a binary image. Furthermore, by using different colors for the outlines of the linear portion 12V3 and the ampulla 12E3 in the edge-enhanced image, the linear portion 12V3 and the ampulla 12E3 can be clearly distinguished from each other.

[0075] As described above, in this embodiment, linear portions of choroidal blood vessels are enhanced from the analysis image by line enhancement processing, and the image is binarized, thereby making it possible to selectively extract linear portions of choroidal blood vessels.

[0076] Furthermore, in this embodiment, the analysis image is binarized or a convex portion of the analysis image is detected using a Hessian matrix, thereby making it possible to selectively extract dilated portions of choroidal blood vessels.

[0077] Extraction of linear and bulging portions of choroidal vessels according to this embodiment allows for reliable extraction of choroidal vessels and vortex veins from fundus images. This allows for digitization of the choroidal vascular network, including vortex veins, and allows for various analyses. For example, this facilitates early detection of signs of arteriosclerosis, allowing ophthalmologists to predict the onset of diseases related to vascular disorders.

[0078] The image processing in each of the above-described embodiments is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0079] In the above-described embodiments, image processing is assumed to be performed by a software configuration using a computer, but the technology of the present disclosure is not limited to this. For example, instead of a software configuration using a computer, image processing may be performed only by a hardware configuration such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Part of the image processing may be performed by a software configuration, and the remaining part may be performed by a hardware configuration.

Claims

1. obtaining a choroidal vascular image; performing line enhancement processing and binarization processing on the choroidal vessel image to extract linear portions; performing a binarization process on the choroidal vessel image to extract a mass portion; An image processing method comprising:

2. The image processing method according to claim 1 , wherein the line enhancement process determines that a local structure in the choroidal vessel image is a line.

3. 3. The image processing method according to claim 1, wherein extracting the lumpy portion comprises extracting an area in which a predetermined number of pixels having a predetermined luminance value are connected as the lumpy portion.

4. The linear portion is a linear portion of a choroidal blood vessel, The image processing method according to claim 1 , wherein the mass is a mass of choroidal blood vessels.

5. 5. The image processing method according to claim 4, wherein the linear portion is a linear portion of a vortex vein, and the mass portion is an enlarged portion of the vortex vein.

Images