Image processing method, image processing device, and image processing program

Abstract

The image processing method includes; acquiring a fundus image; extracting a first region having a first feature from the fundus image; a second region in which a second feature different from the first feature is extracted from the fundus image; and generating a composite image in which the extracted first region and second region are synthesized.

Description

[0001] This invention is a divisional application of the invention application filed on October 17, 2019, with international application number PCT / JP2019 / 040889, national application number 201980102988.3 that entered the Chinese national phase, and entitled "Image Processing Method, Image Processing Apparatus, and Image Processing Program". Technical Field

[0002] This invention relates to image processing methods, image processing apparatus, and image processing programs. Background Technology

[0003] US Patent No. 10,136,812 discloses an optical coherence tomography (OCT) scanner apparatus for selectively visualizing the vascular network of the choroid. The pursuit is for image processing methods for resolving choroidal vessels. Summary of the Invention

[0004] The image processing method of the first aspect of the present disclosure includes acquiring a fundus image; extracting a first region having a first feature from the fundus image; extracting a second region having a second feature different from the first feature from the fundus image; and generating a composite image by combining the extracted first region and the second region.

[0005] The image processing apparatus of the second aspect of the present disclosure includes: an image acquisition unit for acquiring a fundus image; a first extraction unit for extracting linear portions of blood vessels from the fundus image; a second extraction unit for extracting block portions of blood vessels from the fundus image; and a blood vessel visualization unit that integrates the extracted images of the linear portions and the block portions to generate a blood vessel image that visualizes the blood vessels.

[0006] The image processing program of the third aspect of the present disclosure enables a computer to function as an image acquisition unit for acquiring fundus images, a first extraction unit for extracting linear portions of blood vessels from the fundus images, a second extraction unit for extracting block portions of blood vessels from the fundus images, and a blood vessel visualization unit for integrating the extracted linear portions and block portions to generate a blood vessel image that visualizes the blood vessels. Attached Figure Description

[0007] Figure 1 This is a block diagram of the ophthalmology system 100.

[0008] Figure 2 This is a schematic diagram showing the overall structure of the ophthalmic device 110.

[0009] Figure 3 This is a block diagram of the electrical configuration of the management server 140.

[0010] Figure 4This is a block diagram of the functions of managing CPU 262 on server 140.

[0011] Figure 5 This is a block diagram of the functions of the image processing control unit 206 of the CPU 262 of the management server 140.

[0012] Figure 6 It is a flowchart of an image processing procedure.

[0013] Figure 7 yes Figure 6 The flowchart for step 304, choroidal vessel analysis and processing.

[0014] Figure 8A This is a schematic diagram showing an example of extraction of the linear portions 12V1, 12V2, 12V3, and 12V4 of the choroidal vessels.

[0015] Figure 8B This is a schematic diagram showing an example of extraction of the swollen portions 12E1, 12E2, 12E3, and 12E4 of the choroidal vessels.

[0016] Figure 8C This is a schematic diagram showing the combination of the linear portions 12V1, 12V2, 12V3, 12V4 and the swollen portions 12E1, 12E2, 12E3, 12E4 of the choroidal vessels.

[0017] Figure 9 This is a schematic diagram of display screen 500.

[0018] Figure 10 This is a schematic diagram showing the synthesis of each of the RG color fundus images of a specific region 12V3A displayed in time sequence in the process observation area 570, and each of the choroidal vessel extraction images of the same specific region 12V3B photographed on the same date.

[0019] Figure 11 This is a schematic diagram showing the synthesis of contour-enhanced images extracted from RG color fundus images of the same specific region 12V3A from each of the choroidal vessel extraction images of a specific region 12V3B displayed in a time-series manner in the process observation area 570.

[0020] Figure 12 This is a diagram showing the choroidal vessels (CLA). Detailed Implementation

[0021] Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings.

[0022] Reference Figure 1 This describes the composition of the ophthalmic system 100. For example... Figure 1As shown, the ophthalmic system 100 includes an ophthalmic device 110, a management server device (hereinafter referred to as the "management server") 140, and a display device (hereinafter referred to as the "observer") 150. The ophthalmic device 110 acquires fundus images. The management server 140 stores multiple fundus images and axial lengths obtained by photographing the funduses of multiple patients using the ophthalmic device 110, corresponding to the patient's ID. The observer 150 displays the fundus images or analysis results acquired by the management server 140.

[0023] The observer 150 includes a display 156 for displaying fundus images or analysis results acquired by the management server 140, a mouse 155M for operation, and a keyboard 155K.

[0024] The ophthalmic device 110, management server 140, and observer 150 are interconnected via network 130. Observer 150 is a client in the client-server system, and multiple devices are connected via the network. Additionally, to ensure system redundancy, multiple management servers 140 can also be connected via the network. Alternatively, the ophthalmic device 110 only needs to have image processing capabilities and the observer 150 needs to have image viewing capabilities; the ophthalmic device 110 can acquire, process, and view fundus images in a standalone system state. Similarly, the management server 140 only needs to have image viewing capabilities; in this configuration of the ophthalmic device 110 and management server 140, fundus image acquisition, image processing, and image viewing can be achieved.

[0025] In addition, diagnostic support devices that perform image analysis using other ophthalmic machines (examination machines for visual field measurement, intraocular pressure measurement, etc.) or AI (Artificial Intelligence) can also be connected to ophthalmic device 110, management server 140, and observer 150 via network 130.

[0026] Next, refer to Figure 2 This section explains the composition of the ophthalmic device 110. For ease of explanation, the scanning laser ophthalmoscope will be referred to as "SLO". The optical coherence tomography scanner will be referred to as "OCT".

[0027] Furthermore, when the ophthalmic device 110 is set on a horizontal plane, the horizontal direction is defined as the "X direction," the vertical direction relative to the horizontal plane is defined as the "Y direction," and the direction connecting the center of the pupil in the anterior part of the eye being examined 12 with the center of the eyeball is defined as the "Z direction." Therefore, the X, Y, and Z directions are perpendicular to each other.

[0028] The ophthalmic device 110 includes an imaging device 14 and a control device 16. The imaging device 14 includes an SLO unit 18 and an OCT unit 20, and acquires fundus images of the fundus of the eye 12 being examined. Hereinafter, the two-dimensional fundus image acquired using the SLO unit 18 will be referred to as an SLO image. In addition, tomographic images or en-face images of the retina created based on OCT data acquired using the OCT unit 20 will be referred to as OCT images.

[0029] The control device 16 includes a computer with a CPU (Central Processing Unit) 16A, RAM (Random Access Memory) 16B, ROM (Read-Only Memory) 16C, and input / output (I / O) ports 16D.

[0030] The control device 16 includes an input / display device 16E connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphical user interface that displays an image of the eye being examined 12 or receives various instructions from the user. Examples of graphical user interfaces include touch panels and displays.

[0031] Additionally, the control device 16 has an image processing device 17 connected to the I / O port 16D. The image processing device 17 generates an image of the examined eye 12 based on data obtained from the imaging device 14. Furthermore, the control device 16 is connected to the network 130 via a communication interface not shown.

[0032] As mentioned above, in Figure 2 In the ophthalmic device 110, the control device 16 includes an input / display device 16E, but the technology disclosed herein is not limited to this. For example, the control device 16 of the ophthalmic device 110 may not include the input / display device 16E, and the ophthalmic device 110 may also have a physically independent, separate input / display device. In this case, the display device has an image processing processor unit that operates under the control of the CPU 16A of the control device 16. The image processing processor unit may also display an SLO image, etc., based on an image signal output by the CPU 16A.

[0033] The photographic device 14 operates under the control of the CPU 16A of the control device 16. The photographic device 14 includes an SLO unit 18, a photographic optical system 19, and an OCT unit 20. The photographic optical system 19 includes a first optical scanner 22, a second optical scanner 24, and a wide-angle optical system 30.

[0034] The first optical scanner 22 performs a two-dimensional scan of the light emitted from the SLO unit 18 along the X and Y directions. The second optical scanner 24 performs a two-dimensional scan of the light emitted from the OCT unit 20 along the X and Y directions. Both the first optical scanner 22 and the second optical scanner 24 can be any optical element capable of deflecting the light beam; for example, a multifaceted mirror or a current mirror can be used. Alternatively, a combination of these can also be used.

[0035] Wide-angle optical system 30 includes an object-oriented optical system with a shared optical system 28 (in Figure 2 (not shown in the figure), and a combining unit 26 that combines light from SLO unit 18 and light from OCT unit 20.

[0036] Furthermore, the object-oriented optical system of the shared optical system 28 can also be a reflective optical system using concave mirrors such as elliptical mirrors, a refractive optical system using wide-angle lenses, or a reflective-refractive optical system combining concave mirrors or lenses. By using a wide-angle optical system utilizing elliptical mirrors or wide-angle lenses, it is possible to capture not only the central part of the fundus but also the peripheral retina of the fundus.

[0037] When using a system including an elliptic mirror, the configuration of the system utilizing the elliptic mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may also be used. The disclosures of International Publication WO2016 / 103484 and International Publication WO2016 / 103489 are incorporated herein by reference in their entirety.

[0038] Using the wide-angle optical system 30, observation based on a wide field of view (FOV) 12A is achieved in the fundus. FOV 12A indicates the range that can be captured using the photographic device 14. FOV 12A can be represented as a viewing angle. In this embodiment, the viewing angle can be defined using an internal illumination angle and an external illumination angle. The external illumination angle refers to the illumination angle defined with respect to the pupil 27 for the illumination angle of the light beam illuminating the examined eye 12 from the ophthalmic device 110. The internal illumination angle refers to the illumination angle defined with respect to the center O of the eyeball for the illumination angle of the light beam illuminating the fundus F. The external illumination angle and the internal illumination angle are in a corresponding relationship. For example, when the external illumination angle is 120 degrees, the internal illumination angle is approximately equivalent to 160 degrees. In this embodiment, the internal illumination angle is set to 200 degrees.

[0039] Here, the SLO fundus image obtained by shooting with an internal illumination angle of 160 degrees or more is referred to as the UWF-SLO fundus image. In addition, UWF is short for Ultra Wide Field.

[0040] SLO system through Figure 2 This is achieved through the control device 16, the SLO unit 18, and the photographic optical system 19 shown. The SLO system has a wide-angle optical system 30, thus enabling fundus photography based on the wide FOV 12A.

[0041] SLO unit 18 includes a light source 40 for B (blue light), a light source 42 for G (green light), a light source 44 for R (red light), and a light source 46 for IR (infrared light, e.g., near-infrared light), and optical systems 48, 50, 52, 54, and 56 that reflect or transmit light from light sources 40, 42, 44, and 46 to guide a single optical path. Optical systems 48, 50, and 56 are reflectors or transmissors, and optical systems 52 and 54 are beam splitters. B light is reflected by optical system 48, transmitted through optical system 50, and reflected by optical system 54; G light is reflected by optical systems 50 and 54; R light is transmitted through optical systems 52 and 54; and IR light is reflected by optical systems 52 and 56, each guiding a single optical path.

[0042] SLO unit 18 is configured to switch between light sources emitting G, R, and B light, infrared light, and combinations of light sources emitting lasers or light sources with different wavelengths. Figure 2 In the example shown, four light sources are provided: a light source 40 for B light (blue light), a light source 42 for G light, a light source 44 for R light, and a light source 46 for IR light. However, the technology disclosed herein is not limited to these. For example, the SLO unit 18 may also include a light source for white light, emitting light in various modes such as a mode that emits only white light.

[0043] The first optical scanner 22 scans the light incident from the SLO unit 18 onto the photographic optical system 19 in the X and Y directions. The scanning light is directed to the posterior part (fundus) of the eye being examined 12 via the wide-angle optical system 30 and the pupil 27. The reflected light from the fundus is directed back onto the SLO unit 18 via the wide-angle optical system 30 and the first optical scanner 22.

[0044] The SLO unit 18 includes a beam splitter 64 that reflects B-rays from the posterior eye (fundus) of the examined eye 12 and transmits light other than B-rays, and a beam splitter 58 that reflects G-rays from the light transmitted through the beam splitter 64 and transmits light other than G-rays. The SLO unit 18 also includes a beam splitter 60 that reflects R-rays from the light transmitted through the beam splitter 58 and transmits light other than R-rays. The SLO unit 18 further includes a beam splitter 62 that reflects IR-rays from the light transmitted through the beam splitter 60. The SLO unit 18 includes a B-ray detection element 70 for detecting B-rays reflected by the beam splitter 64, a G-ray detection element 72 for detecting G-rays reflected by the beam splitter 58, an R-ray detection element 74 for detecting G-rays reflected by the beam splitter 58, and an IR-ray detection element 76 for detecting IR-rays reflected by the beam splitter 62.

[0045] When the light (reflected light from the fundus) incident on the SLO unit 18 via the wide-angle optical system 30 and scanner 22 is B-beam, it is reflected by beam splitter 64 and received by B-beam detection element 70. When it is G-beam, it is transmitted through beam splitter 64, reflected by beam splitter 58, and received by G-beam detection element 72. When the incident light is R-beam, it is transmitted through beam splitters 64 and 58, reflected by beam splitter 60, and received by R-beam detection element 74. When the incident light is IR-beam, it is transmitted through beam splitters 64, 58, and 60, reflected by beam splitter 62, and received by IR-beam detection element 76. The image processing apparatus 17, operating under the control of CPU 16A, uses the signals detected by B-beam detection element 70, G-beam detection element 72, R-beam detection element 74, and IR-beam detection element 76 to generate a UWF-SLO image. Examples of B-light detection elements 70, G-light detection elements 72, R-light detection elements 74, and IR-light detection elements 76 include PD (photodiode) and APD (avalanche photodiode). These elements correspond to the "image acquisition unit" of the present disclosure. In the SLO unit 18, light reflected (scattered) from the fundus of the object passes through the first optical scanner 22 and reaches the light detection element, thus always returning to the same position—that is, to the position where the B-light detection element 70, G-light detection element 72, R-light detection element 74, and IR-light detection element 76 are located. Therefore, it is not necessary to construct the light detection element in a planar (two-dimensional) manner like a zone sensor; a point-like (0-dimensional) detector such as a PD or APD is preferred in this embodiment. However, it is not limited to PDs or APDs; line sensors (one-dimensional) or zone sensors (two-dimensional) can also be used.

[0046] UWF-SLO images include UWF-SLO images obtained by photographing the fundus using G color (G-color fundus image) and UWF-SLO images obtained by photographing the fundus using R color (R-color fundus image). UWF-SLO images also include UWF-SLO images obtained by photographing the fundus using B color (B-color fundus image) and UWF-SLO images obtained by photographing the fundus using IR color (IR fundus image).

[0047] Furthermore, the control device 16 controls the light sources 40, 42, and 44 to emit light simultaneously. The fundus of the eye being examined 12 is simultaneously photographed using B-light, G-light, and R-light, thereby obtaining G-color fundus images, R-color fundus images, and B-color fundus images corresponding to each location. An RGB color fundus image is obtained from the G-color, R-color, and B-color fundus images. The control device 16 controls the light sources 42 and 44 to emit light simultaneously, and simultaneously photographs the fundus of the eye being examined 12 using G-light and R-light, thereby obtaining G-color and R-color fundus images corresponding to each location. An RG color fundus image is obtained by mixing the G-color and R-color fundus images at a predetermined mixing ratio.

[0048] UWF-SLO images (videos) also exist, captured using ICG fluorescence. When indocyanine green (ICG) is injected into a blood vessel, it reaches the fundus, initially reaching the retina, then the choroid, and passes through it. UWF-SLO images (videos) are dynamic images from the moment indocyanine green (ICG) is injected into the blood vessel and reaches the retina until it passes through the choroid.

[0049] Image data of B-color fundus images, G-color fundus images, R-color fundus images, IR fundus images, RGB color fundus images, RG color fundus images, and UWF-SLO images are sent from ophthalmic device 110 to management server 140 via communication IF (not shown).

[0050] OCT system through Figure 2 This is achieved using the control device 16, the OCT unit 20, and the photographic optical system 19 shown. The OCT system has a wide-angle optical system 30, thus enabling fundus photography based on a wide FOV 12A, similar to the SLO fundus image photography described above. The OCT unit 20 includes a light source 20A, a sensor (detection element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0051] The light emitted from the light source 20A is split by the first optical coupler 20C. The split light is then used as the measurement light and collimated by the collimating lens 20E before entering the photographic optical system 19. The measurement light is scanned in the X and Y directions using the second optical scanner 24. The scanning light is irradiated onto the fundus via the wide-angle optical system 30 and the pupil 27. The measurement light reflected from the fundus enters the OCT unit 20 via the wide-angle optical system 30 and the second optical scanner 24, and then enters the second optical coupler 20F via the collimating lens 20E and the first optical coupler 20C.

[0052] The light emitted from the light source 20A, which is branched off from the first optical coupler 20C, is incident on the reference optical system 20D as a reference light, and then incident on the second optical coupler 20F via the reference optical system 20D.

[0053] The light incident on the second optical coupler 20F—that is, the measurement light and reference light reflected from the fundus—is interfered with by the second optical coupler 20F to generate interference light. The interference light is received by the sensor 20B. The image processing apparatus 17, operating under the control of the image processing control unit 206, generates OCT images such as tomographic images or en-face images based on the OCT data detected by the sensor 20B.

[0054] Here, OCT fundus images obtained by shooting from an internal illumination angle of 160 degrees or more are referred to as UWF-OCT images.

[0055] Image data from the UWF-OCT image is sent from the ophthalmic device 110 to the management server 140 via a communication IF (not shown) and stored in the storage device 254.

[0056] Furthermore, in this embodiment, the light source 20A is SS-OCT (Swept-SourceOCT), which is a wavelength scanning type, but it can also be an OCT system of various types such as SD-OCT (Spectral-Domain OCT) and TD-OCT (Time-Domain OCT).

[0057] Next, refer to Figure 3 Describe the electrical configuration of the management server 140. For example... Figure 3As shown, the management server 140 has a computer body 252. The computer body 252 has a CPU 262, RAM 266, ROM 264, and input / output (I / O) ports 268. The I / O ports 268 are connected to a storage device 254, a monitor 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258. The storage device 254 is, for example, composed of non-volatile memory. The I / O ports 268 are connected to a network 130 via the communication interface (I / F) 258. Therefore, the management server 140 can communicate with the ophthalmic device 110, the axial length measuring device 120, and the observer 150. The image processing program, described later, is stored in the storage device 254. Alternatively, the image processing program can also be stored in the ROM 264.

[0058] The management server 140 stores the data received from the ophthalmic device 110 and the axial length measuring device 120 in the storage device 254.

[0059] Next, refer to Figure 4 This describes the various functions implemented by the CPU 262 of the management server 140 through the execution of an image processing program. The image processing program includes display control, image processing control, and processing functions. By executing this image processing program with these functions, the CPU 262... Figure 4 As shown, it functions as a display control unit 204, an image processing control unit 206, and a processing unit 208.

[0060] The image processing control unit 206 is equivalent to the "first extraction unit", "second extraction unit", "vascular visualization unit" and "choroidal vascular image generation unit" of the present disclosure.

[0061] Next, use Figure 5 The various functions of the image processing control unit 206 are explained. The image processing control unit 206 functions as a fundus image processing unit 2060, which performs image processing such as generating images that clarify the choroidal vessels based on fundus images, and a choroidal vessel analysis unit 2062, which performs image processing such as extracting linear portions and swollen portions (block-like portions) of the choroid. Linear portions correspond to the "first feature" of the present disclosure, and swollen portions correspond to the "second feature" of the present disclosure.

[0062] Next, use Figure 6 Detailed explanation of image processing based on management server 140. The image processing program is executed via the CPU 262 of management server 140 to achieve... Figure 6 The flowchart illustrates image processing (image processing methods).

[0063] In step 300, the image processing control unit 206 acquires the UWF-SLO image from the storage device 254. In step 302, the image processing control unit 206 creates a choroidal vessel image with extracted choroidal vessels based on the acquired UWF-SLO image (R-color fundus image and G-color fundus image). R-color light has a long wavelength and therefore travels from the retina to the choroid. Therefore, the R-color fundus image includes information on vessels present in the retina (retinal vessels) and vessels present in the choroid (choroidal vessels). In contrast, G-color light has a shorter wavelength than R-color light and therefore travels only to the retina. Therefore, the G-color fundus image only includes information on vessels present in the retina (retinal vessels). Therefore, by extracting retinal vessels from the G-color fundus image and removing retinal vessels from the R-color fundus image, a choroidal vessel image (CLA) can be obtained. The R-color fundus image corresponds to the "red light photographic image" of the present disclosure.

[0064] The following describes the specific processing performed by the image processing control unit 206 in step 302.

[0065] First, the image processing control unit 206 performs noise removal processing on each image, including the G-color fundus image and the R-color fundus image. A mid-range filter is applied to remove noise.

[0066] Then, the image processing control unit 206 extracts retinal vessels from the G-color fundus image by performing black hat filtering on the noise-removed G-color fundus image.

[0067] Then, the image processing control unit 206 removes the retinal vessels by using the location information of the retinal vessels extracted from the G-color fundus image to fill the retinal vessel structure of the R-color fundus image with the same value as the surrounding pixels through a depiction process. Using this process, an image is generated by removing the retinal vessels from the R-color fundus image, resulting in an image that only visualizes the choroidal vessels.

[0068] Next, the image processing control unit 206 removes low-frequency components from the R-color fundus image after depiction processing. To remove low-frequency components, image processing techniques such as frequency filtering or spatial filtering are applied.

[0069] Then, finally, the image processing control unit 206 appropriately performs histogram equalization processing (Contrast Limited Adaptive Histogram Equalization) on the image data of the R-color fundus image, in which retinal vessels have been removed but choroidal vessels remain, thereby enhancing the choroidal vessels in the R-color fundus image. Through a series of processing steps 302, a process is created... Figure 12The choroidal vessel image CLA is shown. The created choroidal vessel image CLA is stored in storage device 254.

[0070] Furthermore, in the above example, a choroidal vessel image (CLA) is generated from an R-color fundus image and a G-color fundus image. However, it is not limited to this; the image processing control unit 206 can also generate a choroidal vessel image (CLA) based on a G-color fundus image and an IR fundus image. Additionally, the image processing control unit 206 can also generate a choroidal vessel image (CLA) based on a B-color fundus image and an R-color fundus image or an IR fundus image.

[0071] Furthermore, a choroidal vessel image (CLA) can also be generated from the UWF-SLO image (video) 510. As described above, the UWF-SLO image (video) 510 is a dynamic image from the time indocyanine green (ICG) is injected into the blood vessel and reaches the retina until it passes through the choroid. The choroidal vessel image (CLA) can also be generated from the dynamic image during the period from when indocyanine green (ICG) passes through the retina to when it passes through the choroid.

[0072] In step 304, choroidal vessel analysis processing is performed. This process is specifically designed for choroidal vessel analysis, independently extracting the linear portions and dilated portions of the choroidal vessels. The locations of vortex veins, which are part of the choroidal vessels, are then extracted from the extracted linear and dilated portions. Anatomically, vortex veins are areas where choroidal vessels are concentrated, serving as drainage pathways for blood flowing into the eye. There are typically 3-7 vortex veins in the eye, located in the peripheral fundus (near the equator). In fundus images, the locations of vortex veins are identified as the central portion of a mass and multiple linear portions connecting to it.

[0073] The following section details the analysis of the choroidal vessels in step 304.

[0074] In step 306, the parsed data obtained in the choroidal vessel parsing process in step 304 is output to the storage device 254 of the management server 140. In step 308, the display control unit 2044 generates a display screen 500 that displays the extracted choroidal vessel image and patient attribute information corresponding to the patient ID (patient name, age, information on whether each fundus image is of the left or right eye, axial length, visual acuity, and date and time of the imaging, etc.), displays it on the display 256 of the management server 140, and ends the processing.

[0075] The display screen 500 is stored in the storage device 254 of the management server 140. The display screen 500 stored in the storage device 254 of the management server 140 is sent to the observer 150 according to the operation from the observer 150, and output to the monitor 156 of the observer 150 in a viewable state.

[0076] Figure 6 The processing shown can also be performed by the CPU 16A of the control device 16 of the ophthalmic device 110. When the processing is performed using the CPU 16A of the ophthalmic device 110, a display image 500 is displayed on the display of the ophthalmic device, and the display image 500 is stored in the storage device of the management server 140.

[0077] in addition, Figure 6 The process shown can also be executed using the CPU of the observer 150. When the process is executed using the CPU of the observer 150, the display image 500 is displayed on the monitor 156 of the observer 150, and the display image 500 is stored in various devices of the storage device of the observer 150 and the storage device of the management server 140.

[0078] Figure 7 It shows Figure 6 The flowchart shows the detailed content of the choroidal vessel analysis processing (step 304). In step 400, the first vessel extraction processing, which extracts the linear portion from the choroidal vessel image CLA, is performed. In step 400, firstly, relative to... Figure 12 The illustrated choroidal vessel image CLA is analyzed using image line enhancement processing. This line enhancement processing enhances the choroidal vessels as linear structures. Then, the line-enhanced image is binarized. Next, image processing is performed to extract the linear portions of the line-enhanced choroidal vessel image CLA.

[0079] Line enhancement processing, for example, is the process of enhancing linear structures using Hessian analysis, which employs the Hessian matrix. Hessian analysis analyzes the intrinsic values ​​of the Hessian matrix, which is calculated using the second-order partial differential coefficients of a filter with a predefined Gaussian kernel, thereby determining whether a local structure in the image is a point, line, or surface.

[0080] In addition to the line enhancement techniques described above, enhancement of the linear portions of choroidal vessels can also be achieved using Gable filters that extract the direction of the contours contained within the image, or image cropping that cuts out the linear portions from other parts. Alternatively, edge enhancement techniques such as Laplacian filters or sharpening masks can also be used.

[0081] Using the first blood vessel extraction process in step 400, such as Figure 8A The linear portion of the extracted choroidal vessels is described. Figure 8A In this process, the linear portions 12V1, 12V2, 12V3, and 12V4 are clearly distinguished from other areas of the fundus for visualization.

[0082] In step 402, a second vessel extraction process is performed to extract the swollen portion of the choroidal vessels from the choroidal vessel image CLA. The second vessel extraction process first binarizes the analytical image. Then, a region connected to a predetermined number of white pixels is extracted from the binarized choroidal vessel image CLA as the swollen portion of the choroidal vessels. This predetermined number or the size of the region is predetermined based on the size of the vortex veins (standard data for the choroid, etc.). This extraction process can also utilize Hessian analysis using a Hessian matrix to detect the concavity and convexity of the image and extract the convex portion as the swollen portion. Hessian analysis is equivalent to the "image processing filter that only extracts blocky portions" of the present disclosure.

[0083] There are also cases where pixels corresponding to the bulge and linear portions of the choroidal vessels are extracted together. However, by combining the linear portions and the bulge using the data integration processing described later, the extraction of the vortex vein location is not affected.

[0084] like Figure 8B As shown, the swollen portion of the choroidal vessels is extracted using the second vessel extraction process in step 402. Figure 8B In the example shown, where four vortex veins are present in the eye (typically there are four to six), the bulges 12E1, 12E2, 12E3, and 12E4 are clearly distinguished from other areas of the fundus and displayed. The bulges 12E1, 12E2, 12E3, and 12E4 can also be represented as inflow sites (choroidal entrances of vortex veins flowing outwards from the eyeball) into the scleral side of the vortex veins present in the peripheral fundus. The binarization processing of the entire choroidal vessel image CLA in step 402 can be used to extract choroidal bulges that could not be extracted using the linear portion extraction processing based on the first vessel extraction processing in step 400.

[0085] In addition, they can be interchanged. Figure 7 The flowchart shows the sequence of the first and second vessel extraction processes: the second vessel extraction process is performed in step 400, and the first vessel extraction process is performed in step 402. This is because the first and second vessel extraction processes are independent processes that do not interfere with each other.

[0086] In step 404, as Figure 8CAs shown, data integration is performed, combining the linear portion obtained as a result of the first vessel extraction process and the swollen portion obtained as a result of the second vessel extraction process. That is, an image of the linear portion obtained in step 400 and an image of the swollen portion obtained in step 402 are synthesized to generate a choroidal vessel image. Then, proceed to... Figure 7 Step 306 begins the process of parsing the synthesized image. Figure 8C The illustrated choroidal vessel image is equivalent to a “synthetic image” of the technology disclosed herein.

[0087] exist Figure 8C The image shows the combined state of the linear portions 12V1, 12V2, 12V3, 12V4 and the expanded portions 12E1, 12E2, 12E3, 12E4. Furthermore, Figure 8A , 8B Each of the 8C images is outlined in white as a background image for areas where the fundus is not displayed, clearly showing the image of the choroidal vessels extracted as a binary image. However, it is also possible to simply outline the binary image and the background image in white, thus enabling clear visual confirmation of the binary image. In the display screen 500 described later, the case where the background image for areas where the fundus is not displayed is shown in white is illustrated.

[0088] Figure 9 This is a schematic diagram showing a display screen 500 displayed on a monitor 256 of the management server 140, etc. The display image 500 can also be displayed on the monitor 156 of the observer 150, or on the input / display device 16E of the ophthalmic device 110, in addition to the monitor 256 of the management server 140.

[0089] like Figure 9 As shown, the display screen 500 has an information display area 502 and an image display area 504. The information display area 502 has a patient ID display area 512, a patient name display area 514, an age display area 516, a right eye / left eye display area 518, and an axial length display area 522.

[0090] Image display area 504 is used to display the latest image (in Figure 9 The latest image (image of the fundus taken on July 16, 2019) is displayed in area 550, while the previous image (image taken before the latest image) is displayed. Figure 9The image display area 560 (a fundus image taken on April 16, 2019), the process observation area 570 (displaying time-series changes in the fundus), and the notes column 580 (displaying records of treatment or diagnosis entered by the user) are all included. The latest image display area 550 has a date display area 552 at the top and displays a choroidal vessel extraction image 556, which combines the RG color fundus image 554 and the linear portions 12V1, 12V2, 12V3, 12V4 and the bulging portions 12E1, 12E2, 12E3, 12E4 of the choroidal vessels as a binary image.

[0091] The front image display area 560 has a photographic date display area 562 at the top, and displays a choroidal vessel extraction image 566 as a binary image, which combines the RG color fundus image 564 and the linear portions 12V1, 12V2, 12V3, 12V4 and the bulging portions 12E1, 12E2, 12E3, 12E4 of the choroidal vessels.

[0092] The areas of the latest image display area 550 and the front image display area 560 can also replace the RG color fundus images 554, 564 and the choroidal vessel extraction images 556, 566 to display choroidal angiography images (ICG) or optical interferometry angiography (OCTA). The RG color fundus images 554, 564, ICG, and OCTA images are not limited to 2D representation; they can also be displayed in 3D. The images displayed in the latest image display area 550 and the front image display area 560 can be selected from a menu displayed by opening the icons of the display switching icons 558, 568.

[0093] The process observation area 570 displays time-series-based changes in specific regions 12V3A of RG color fundus images 554 and 564 and specific regions 12V3B of choroidal vessel extraction images 556 and 566. The process observation area 570 includes a latest image display area 576 displaying the latest images of specific regions 12V3A and 12V3B, a previous image display area 574 displaying previous images taken before the latest images of specific regions 12V3A and 12V3B, and an image display area 574 displaying images taken before the previous images of specific regions 12V3A and 12V3B (in...). Figure 9 The image is an anterior image display area 572 (the fundus image captured on January 16, 2019). At the bottom, it has a time-series vessel diameter display area 578 that shows the vessel diameter of the bulge 12E3 and the peripheral portion (linear portion 12V3) based on the time-series changes when the latest image, the previous image, and the anterior image were captured. Figure 9In the process observation area 570, three images taken at three time intervals (latest, previous, and previous-previous) are displayed, but not limited to three images. Four or more fundus images taken at different dates and times can also be displayed in time sequence.

[0094] Figure 10 This is a schematic diagram showing the synthesis of each RG color fundus image of a specific region 12V3A displayed in a time-series manner in the process observation area 570, along with each choroidal vessel extraction image of the same specific region 12V3B from the same photographic date. It can also be displayed in the process observation area 570. Figure 10 This is a composite image.

[0095] Figure 11 This is a schematic diagram showing the synthesis of contour-enhanced images extracted from RG color fundus images of the same specific region 12V3A from each of the choroidal vessel extraction images of a specific region 12V3B displayed in a time-series manner in the process observation area 570. It can also be displayed in the process observation area 570. Figure 11 This synthesized image, displayed in the process observation area 570, can be selected from a menu displayed via the start display switch icon 582. The contour-enhanced image can be generated by applying known techniques such as the Sobel filter to an RG color fundus image or a choroidal vessel extraction image. By synthesizing the contour-enhanced image from the choroidal vessel extraction image, which is a binary image, the linear portion 12V3 and the dilated portion 12E3 of the choroidal vessels can be more clearly identified compared to the binary image. Furthermore, in the contour-enhanced image, by changing the color of the contour portion 12V3 and the dilated portion 12E3, the linear portion 12V3 and the dilated portion 12E3 can be clearly identified.

[0096] As explained above, in this embodiment, the linear portion of the choroidal vessels is enhanced from the analytical image using line enhancement processing, and the image is binarized, thereby enabling selective extraction of the linear portion of the choroidal vessels.

[0097] In addition, in this embodiment, the image for analysis is binarized or the convex portion of the image for analysis is detected using a Hessian matrix, thereby enabling selective extraction of the swollen portion of the choroidal vessels.

[0098] By utilizing the extraction of the linear and dilated portions of the choroidal vessels in this embodiment, choroidal vessels and vortex veins can be reliably extracted from fundus images. Therefore, the choroidal network data, including vortex veins, can be digitized and subjected to various analyses. For example, signs of arteriosclerosis can be easily and quickly detected, allowing ophthalmologists to infer early signs of vascular diseases.

[0099] The image processing described above is at least one example. Therefore, it is of course possible to delete unnecessary steps, add new steps, or change the processing order without departing from the main idea.

[0100] In the embodiments described above, image processing based on computer-based software is assumed, but the technology disclosed herein is not limited thereto. For example, instead of using computer-based software, image processing can be performed solely using hardware such as FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Alternatively, a portion of the image processing can be performed using software, while the remaining processing can be performed using hardware.

Claims

1. An image processing method, comprising: Acquire fundus images; Extract a first region with a first feature from the fundus image; Extract a second region from the fundus image that has a second feature different from the first feature; and Generate a composite image by combining the extracted first region and the second region.

2. The image processing method according to claim 1, wherein, The first feature is a linear feature. The second feature is a block-shaped feature.

3. The image processing method according to claim 2, wherein, The first feature is the linear characteristic of blood vessels. The second feature is the blocky feature of blood vessels.

4. The image processing method according to any one of claims 1 to 3, wherein, The fundus image is a red light photographic image taken using red light.

5. The image processing method according to claim 4, wherein, The fundus image is a binarized fundus image obtained by binarizing the red light photographic image.

6. The image processing method according to any one of claims 1 to 5, wherein, The synthesized image is an image showing the vascular structure of the fundus of the eye.

7. The image processing method according to any one of claims 1 to 6, wherein, The synthesized image is an image showing the vascular structure of the choroid in the fundus of the eye.

8. The image processing method according to any one of claims 1 to 7, wherein, The first region is a linear region of choroidal vessels, and the second region is a blocky portion of choroidal vessels.

9. The image processing method according to claim 8, wherein, The first region is a linear region of the vortex vein, and the second region is the expanded portion of the vortex vein.

10. An image processing method, comprising: Acquire fundus images; Extract the linear portion of blood vessels from the fundus image; Extract the blocky portions of blood vessels from the fundus image; as well as The extracted images of the linear portion and the block portion are combined to generate a vascular image that visualizes the blood vessels.

11. The image processing method according to claim 10, wherein, The fundus image is a grayscale image, and the blood vessel image is a binary image.

12. The image processing method according to claim 10 or 11, wherein, The vascular image is a choroidal vascular image after the choroidal vessels have been visualized.

13. The image processing method according to claim 12, wherein, The linear portion refers to the linear portion of the choroidal blood vessel.

14. The image processing method according to claim 12, wherein, The block-shaped portion is the expanded portion of the choroidal blood vessel.

15. The image processing method according to any one of claims 10 to 14, wherein, The linear portion is extracted from the fundus image using processing including line enhancement.

16. The image processing method according to any one of claims 10 to 15, wherein, The blocky region is extracted from the fundus image using binarization processing.

17. The image processing method according to any one of claims 10 to 15, wherein, The blocky region is extracted from the fundus image using an image processing filter that extracts only the blocky region.

18. An image processing method, comprising: Acquire fundus images, including choroidal vessels; Extract the linear portion of blood vessels from the fundus image; Extract the blocky portions of blood vessels from the fundus image; as well as The extracted images of the linear portion and the block portion are combined to generate a choroidal vascular image composed of the linear portion and the block portion.

19. The image processing method according to claim 18, wherein, The fundus image is an image obtained using a laser scanning ophthalmoscope.

20. The image processing method according to claim 18, wherein, The fundus image is a frontal image based on angiography using an optical coherence tomography.

21. The image processing method according to claim 18, wherein, It also includes detecting the location of vortex veins from the choroidal vascular image.

22. An image processing method, comprising: Acquire fundus images; Extract the linear portion of the vortex vein from the fundus image; The swollen portion of the vortex vein is extracted from the fundus image; as well as The linear portion and the expanded portion are synthesized, and the vascular structure of the vortex vein is extracted.

23. The image processing method according to claim 22, wherein, It also includes the analysis of the vascular structure of the vortex vein.

24. An image processing apparatus comprising: Image acquisition unit for acquiring fundus images; The first extraction section extracts the linear portion of blood vessels from the fundus image; A second extraction unit that extracts blocky portions of blood vessels from the fundus image; and The vascular visualization unit integrates the extracted images of the linear portion and the block portion to generate a vascular image that visualizes the blood vessels.

25. An image processing apparatus comprising: Image acquisition unit that acquires fundus images, including choroidal vessels; The first extraction section extracts the linear portion of blood vessels from the fundus image; A second extraction unit that extracts blocky portions of blood vessels from the fundus image; and The choroidal vessel image generation unit integrates the extracted images of the linear portion and the block portion to generate a choroidal vessel image composed of the linear portion and the block portion.

26. An image processing program that enables a computer to function as an image acquisition unit for acquiring a fundus image, a first extraction unit for extracting linear portions of blood vessels from the fundus image, a second extraction unit for extracting block portions of blood vessels from the fundus image, and a blood vessel visualization unit for integrating the extracted linear portions and block portions to generate a blood vessel image that visualizes the blood vessels.

27. An image processing program that enables a computer to function as an image acquisition unit for acquiring fundus images including choroidal vessels, a first extraction unit for extracting linear portions of vessels from the fundus images, a second extraction unit for extracting block portions of vessels from the fundus images, and a choroidal vessel image generation unit, wherein the choroidal vessel image generation unit integrates the extracted images of the linear portions and the block portions to generate a choroidal vessel image composed of the linear portions and the block portions.

Images