Ophthalmic apparatus, method for controlling ophthalmic apparatus, program, and recording medium

Abstract

An ophthalmic apparatus according to an embodiment includes an image acquisition unit, a blood vessel distribution image generation unit, a processed image generation unit, and a display control unit. The image acquisition unit acquires a first fundus image generated by applying optical coherence tomography angiography to the fundus of an eye to be examined, and a second fundus image generated by applying, to the fundus of the eye to be examined, an imaging modality different from the optical coherence tomography angiography. The blood vessel distribution image generation unit generates a blood vessel distribution image from the first fundus image. The processed image generation unit generates a processed image by applying a predetermined processing to the second fundus image. The display control unit causes a display device to display a composite image based on a plurality of images including the blood vessel distribution image and the processed image.

Description

Ophthalmic device, method for controlling an ophthalmic device, program, and recording medium

[0001] The present disclosure relates to an ophthalmic apparatus, a method for controlling an ophthalmic apparatus, a program, and a recording medium.

[0002] Various imaging modalities are used in ophthalmology practice, including fundus cameras, scanning laser ophthalmoscopy (SLO), slit lamp microscopes, and optical coherence tomography (OCT). OCT can be used for both structural and functional imaging.

[0003] Structural imaging using OCT is a technique for representing the spatial distribution of OCT signal intensity, which varies depending on the structure of a test object, as an image. Images generated by this technique are called OCT intensity images or simply intensity images.

[0004] Motion contrast imaging is one of the functional imaging techniques using OCT. Motion contrast imaging is a technique that expresses the temporal modulation of OCT signal intensity as an image (motion contrast image). In the field of ophthalmology, motion contrast imaging is mainly used to visualize blood flow in the fundus of the eye. This allows for the acquisition of images that express retinal and choroidal blood vessels. This technique is called OCT angiography (OCTA). Images constructed by OCT angiography are called OCT angiography images or simply angiography images.

[0005] U.S. Pat. No. 1,042,635

[0006] One objective of the present disclosure is to provide an integrated and detailed visualization of fundus structure and vascular status.

[0007] An ophthalmologic apparatus according to an exemplary embodiment of the present disclosure includes an image acquisition unit, a blood vessel distribution image generation unit, a processed image generation unit, and a display control unit. The image acquisition unit acquires a first fundus image generated by applying optical coherence tomography angiography to the fundus of the subject's eye, and a second fundus image generated by applying an imaging modality different from optical coherence tomography angiography to the fundus of the subject's eye. The blood vessel distribution image generation unit generates a blood vessel distribution image from the first fundus image. The processed image generation unit generates a processed image by applying a predetermined processing process to the second fundus image. The display control unit displays a composite image based on multiple images including the blood vessel distribution image and the processed image on a display device.

[0008] Exemplary embodiments of the present disclosure may provide an integrated and detailed visualization of fundus structure and vascular status.

[0009] FIG. 1 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 2 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 3 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 4 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 5 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 6 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 7 is a schematic diagram of a configuration of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 8 is a flowchart of an operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 9 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 10 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 11 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 12 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 13 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 14 is a flowchart of an operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 1 is a flowchart of the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 2 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 3 is a flowchart of the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 4 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 5 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment. FIG. 6 is a schematic diagram for explaining the operation of an ophthalmic apparatus according to a non-limiting embodiment.

[0010] Exemplary embodiments of the present disclosure are described below. Some embodiments provide an integrated and detailed visualization of the structure and vascular status of the fundus, which allows comprehensive, multi-angle, and precise fundus observation.

[0011] OCT angiography can visualize the location of blood flow in detail by detecting and enhancing the time-varying intensity of interference signals caused by blood flow. While OCT angiography can provide detailed visualization of vascularity, it cannot provide visualization of the location and structure of tissues other than blood vessels. Furthermore, OCT angiography cannot provide information about the relationship between blood vessels and other tissues.

[0012] Additionally, structural imaging modalities such as fundus cameras, SLO, and OCT structural imaging can visualize various tissues, including blood vessels, but cannot depict blood vessels with the same level of detail as OCT angiography.

[0013] As described above, conventional fundus imaging techniques have the problem of being unable to provide comprehensive and detailed visualization information of both the fundus structure and the state of blood vessels. This problem is solved by some embodiments of the present disclosure.

[0014] The present disclosure describes several non-limiting embodiments. The present disclosure provides an embodiment of an ophthalmic apparatus, an embodiment of a control method thereof, an embodiment of a program, and an embodiment of a recording medium. The ophthalmic apparatus of the embodiment may be an ophthalmic imaging apparatus having an imaging function, or an ophthalmic image processing apparatus without such an imaging function. The categories of the embodiments are not limited to these. For example, it will be understood by those skilled in the art that the present disclosure can provide aspects such as a method for processing ophthalmic images, a program for causing a computer to execute each step of the method, and a recording medium (computer-readable non-transitory recording medium) on which the program is recorded.

[0015] <Embodiments of Ophthalmic Device> Several non-limiting aspects of an ophthalmic device according to an embodiment will be described. The ophthalmic device according to an embodiment has one or both of a function for generating an OCT angiography image and a function for receiving an external input of an OCT angiography image, a function for generating an image using an imaging modality other than OCT angiography and a function for receiving an external input of a similar image, and a function for processing the image.

[0016] The ophthalmic device of the aspect mainly described in the present disclosure is configured to function as an OCT device capable of performing OCT angiography (OCT scanning and image construction processing). However, the ophthalmic device of another aspect may not have the function of applying OCT angiography to a living eye. In other words, the ophthalmic device of this other aspect may not have at least one of a configuration for applying OCT scanning to a living eye and a configuration for constructing an image from data collected by the OCT scanning.

[0017] The type of OCT used in the present disclosure may be any type, with some embodiments using either spectral domain OCT or swept source OCT.

[0018] Spectral domain OCT splits light from a low-coherence light source into measurement light and reference light, superimposes the return light of the measurement light from the test object with the reference light to generate interference light, detects the spectral distribution of the generated interference light with a spectroscope, and applies image construction processing such as Fourier transform to the detected spectral distribution.

[0019] Swept-source OCT is a technique in which light from a tunable light source is split into measurement light and reference light, return light from the test object is superimposed on the reference light to generate interference light, the generated interference light is detected by a balanced photodiode, and image construction processing such as Fourier transform is applied to the detection data collected in response to the wavelength sweep and scanning of the measurement light.

[0020] Thus, spectral domain OCT is an OCT method that acquires the spectral distribution of interference light in a spatially divided manner, and swept-source OCT is an OCT method that acquires the spectral distribution of interference light in a time-divided manner. Some embodiments may use another OCT method (e.g., time-domain OCT).

[0021] The ophthalmic device of the embodiment mainly described in this disclosure has a function as a fundus camera that generates digital fundus photographs. However, the ophthalmic device of another embodiment may have another fundus imaging modality. For example, the ophthalmic device of some embodiments may have a function as any of the fundus imaging modalities of SLO, slit lamp microscope, and surgical microscope. These fundus imaging modalities are non-limiting examples of imaging modalities different from OCT angiography. The ophthalmic device of the embodiment mainly described in this disclosure can acquire a three-dimensional OCT intensity image as another exemplary imaging modality.

[0022] Unless otherwise noted, in this disclosure, the terms "image data" to refer to a collection of pixel data and "image" to refer to visual information generated from this image data may be used interchangeably.

[0023] 1 to 3 show the configuration of an ophthalmologic apparatus according to a non-limiting embodiment. The ophthalmologic apparatus 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic and control unit 200. The fundus camera unit 2 includes elements of a fundus camera and elements of an OCT scanner. The OCT unit 100 includes elements of an OCT scanner. The arithmetic and control unit 200 includes one or more processors that perform various processes (such as calculation, analysis, and control) and a storage device.

[0024] At least a portion of the functionality of some elements of embodiments according to the present disclosure may be implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general purpose processor, a special purpose processor, an integrated circuit, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), a Field Programmable Gate Array (FPGA)), or a combination of these devices configured and / or programmed to perform at least some of the disclosed functions. The term "circuitry," "unit," "means," or the like may refer to hardware that performs at least some of the disclosed functions or that is configured and / or programmed to perform at least some of the disclosed functions. This hardware may be the hardware described in this disclosure or may include known hardware and / or hardware that is configured and / or programmed to perform at least some of the functions described in this disclosure. In the case of a processor, where the hardware can be considered as a certain type of circuitry, the term "circuitry," "unit," "means," or the like may refer to a combination of hardware and software. This software can be used to configure the hardware and / or the processor.

[0025] The fundus camera unit 2 will now be described. Fig. 1 shows a non-limiting configuration of the fundus camera unit 2. The fundus camera unit 2 is capable of photographing the fundus Ef and the anterior segment of the subject's eye E. The digital image generated by the fundus camera unit 2 is typically a front image. The fundus camera unit 2 acquires observation images by video recording using near-infrared fixed light as illumination light, and also acquires photographed images by photography using visible flash light as illumination light.

[0026] The fundus camera unit 2 includes an illumination optical system 10 and an imaging optical system 30. The illumination optical system 10 irradiates illumination light onto the subject's eye E. The imaging optical system 30 images the subject's eye E being irradiated with the illumination light. The fundus camera unit 2 guides measurement light provided from the OCT unit 100 to the subject's eye E, and also guides return light of the measurement light projected onto the subject's eye E to the OCT unit 100.

[0027] The observation illumination light output from the observation light source 11 of the illumination optical system 10 is reflected by the concave mirror 12, passes through the condenser lens 13, and passes through the visible cut filter 14 to become near-infrared light.It is then focused near the imaging light source 15, reflected by the mirror 16, and passed through the relay lens system 17, relay lens 18, aperture 19, and relay lens system 20 to be guided to the aperture mirror 21.It is reflected by the mirror portion around the central hole of the aperture mirror 21, passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. The return light of the observation illumination light projected onto the subject's eye E is refracted by the objective lens 22, passes through the dichroic mirror 46, passes through the central hole of the aperture mirror 21, passes through the dichroic mirror 55, passes through the photographing focusing lens 31, is reflected by the mirror 32, passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged on the light-receiving surface of the image sensor 35 by the imaging lens 34. The image sensor 35 detects the return light at regular time intervals. The focus of the photographing optical system 30 is adjusted according to the photographing region.

[0028] The imaging illumination light output from the imaging light source 15 is projected onto the fundus oculi Ef along the same path as the observation illumination light. The return light of the imaging illumination light from the subject's eye E is guided to the dichroic mirror 33 along the same path as the return light of the observation illumination light, passes through the dichroic mirror 33, is reflected by a mirror 36, and is imaged by an imaging lens 37 on the light-receiving surface of an image sensor 38.

[0029] The liquid crystal display (LCD) 39 displays a fixation target (fixation target image) used to guide and fixate the line of sight. The light beam output from the LCD 39 is reflected by the half mirror 33A, reflected by the mirror 32, passes through the photographing focusing lens 31 and the dichroic mirror 55, passes through the central hole of the aperture mirror 21, transmits through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the fundus Ef.

[0030] The alignment optical system 50 generates an alignment index used to align the ophthalmic apparatus 1 with the subject's eye E. Alignment light output from a light-emitting diode (LED) 51 passes through an aperture 52, an aperture 53, and a relay lens 54, is reflected by a dichroic mirror 55, passes through the central hole of the aperture mirror 21, transmits through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. Return light of the alignment light from the subject's eye E is guided to the image sensor 35 via the same path as the return light of the observation illumination light. By referring to the alignment index image generated by the image sensor 35, a user can perform manual alignment and / or the ophthalmic apparatus 1 can perform automatic alignment.

[0031] The focusing optical system 60 generates a split index used for focus adjustment of the subject's eye E. The focusing optical system 60 moves along the optical path (illumination optical path) of the illumination optical system 10 in conjunction with movement of the photographing focusing lens 31 along the optical path (photography optical path) of the photographing optical system 30. The reflecting rod 67 is inserted into and removed from the illumination optical path. When performing focus adjustment, the reflective surface of the reflecting rod 67 is tilted and positioned in the illumination optical path. The focusing light output from the LED 61 passes through the relay lens 62, is split into two beams by the split index plate 63, passes through the two-hole diaphragm 64, is reflected by the mirror 65, is once imaged and reflected by the condenser lens 66 on the reflective surface of the reflecting rod 67, passes through the relay lens 20, is reflected by the aperture mirror 21, passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. The return light of the focusing light from the subject's eye E is guided to the image sensor 35 through the same path as the return light of the alignment light. By referring to the split target image generated by the image sensor 35, manual focusing by the user and / or autofocusing by the ophthalmic apparatus 1 is performed.

[0032] The diopter correction lenses 70 and 71 are selectively inserted into the photographing optical path between the aperture mirror 21 and the dichroic mirror 55. The diopter correction lens 70 is a plus lens for correcting farsightedness, and the diopter correction lens 71 is a minus lens for correcting nearsightedness.

[0033] The dichroic mirror 46 combines the fundus imaging optical path and the OCT optical path (measurement arm). The dichroic mirror 46 reflects light in the wavelength band used for OCT and transmits light for fundus imaging. The measurement arm is provided with, in order from the OCT unit 100 side, a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45. The retroreflector 41 is movable along the optical path of the measurement light LS incident thereon and is used to correct the optical path length according to the axial length and adjust the interference state. The dispersion compensation member 42 is used for dispersion compensation between the measurement arm and the reference arm. The OCT focusing lens 43 is movable along the measurement arm and is used to adjust the focus of the measurement arm. The movement of the OCT focusing lens 43 is coordinated with the movement of the imaging focusing lens 31 and the focus optical system 60. The optical scanner 44 is aligned to be positioned substantially conjugate with the pupil of the subject's eye E, and changes the traveling direction of the measurement light LS. The optical scanner 44 includes, for example, a galvano scanner that deflects the measurement light LS in the x direction and a galvano scanner that deflects the measurement light LS in the y direction.

[0034] In the present disclosure, the direction of the optical axis of the fundus camera unit 2 (objective lens 22) is defined as the z-direction, one direction perpendicular to the z-direction is defined as the x-direction, and a direction perpendicular to both the z-direction and the x-direction is defined as the y-direction. In many ophthalmic examinations, alignment is performed so that the optical axis of the device (in this embodiment, the optical axis of the objective lens 22) coincides with the ocular axis of a subject in a sitting or standing position. In accordance with the conventions in the ophthalmic field, the present disclosure defines the direction of the axis of the subject's eye E as the z-direction, the horizontal direction as the x-direction, and the vertical direction (the subject's body axis direction) as the y-direction. When performing an examination on a subject in a different position, directions can be defined in accordance with these definitions.

[0035] The OCT unit 100 will now be described. FIG. 2 shows a non-limiting configuration of the OCT unit 100. The OCT unit 100 has a spectral domain OCT optical system. This OCT optical system includes an interference optical system. This interference optical system splits light from a low-coherence light source (broadband light source) into measurement light LS and reference light LR, and generates interference light LC by superimposing the return light of the measurement light LS projected onto the subject's eye E by the measurement arm and the reference light LR that has passed through the reference arm. A spectroscope 130 generates an electrical signal indicating the spectral distribution of the interference light LC.

[0036] The light source unit 101 outputs broadband low-coherence light L0, and includes an optical output device such as a superluminescent diode (SLD), an LED, or a semiconductor optical amplifier (SOA).

[0037] Low-coherence light L0 output from light source unit 101 is guided by optical fiber 102 to polarization controller 103, where its polarization state is adjusted, and then guided by optical fiber 104 to fiber coupler 105, where it is split into measurement light LS and reference light LR. The measurement light LS is guided by the measurement arm, and the reference light LR is guided by the reference arm.

[0038] The reference light LR is guided by an optical fiber 110 to a collimator 111 where it is converted into a parallel beam, passes through an optical path length correction member 112 for compensating for the optical distance between the measurement arm and the reference arm, passes through a dispersion compensation member 113 for compensating for dispersion between the measurement arm and the reference arm, and is then guided to a retroreflector 114. The retroreflector 114 is movable in a direction along the optical path of the reference light LR incident thereon. The movement of the retroreflector 114 is used to correct the optical path length based on the axial length of the subject's eye E and to adjust the interference state. The reference light LR that has passed through the retroreflector 114 passes through the dispersion compensation member 113 and the optical path length correction member 112, is converted from a parallel beam into a focused beam by the collimator 116, is guided through the optical fiber 117 to the polarization controller 118 where its polarization state is adjusted, is guided through the optical fiber 119 to the attenuator 120 where its light amount is adjusted, and reaches the fiber coupler 122 via the optical fiber 121.

[0039] On the other hand, the measurement light LS is guided through the optical fiber 127 to the collimator lens unit 40, where it is converted into a parallel beam, passes through the retroreflector 41, the dispersion compensation member 42, the OCT focusing lens 43, the optical scanner 44, and the relay lens 45, is reflected by the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the subject's eye E. The measurement light LS is scattered and reflected at various depth positions in the subject's eye E. Return light of the measurement light LS from the subject's eye E is guided by the measurement arm to the fiber coupler 105, and reaches the fiber coupler 122 via the optical fiber 128.

[0040] The fiber coupler 122 generates interference light LC by superimposing the measurement light LS incident from the optical fiber 128 and the reference light LR incident from the optical fiber 121. The interference light LC is guided to the spectroscope 130 via the optical fiber 129. The spectroscope 130 converts the incident interference light LC into a parallel beam using a collimator lens, resolves this parallel beam into multiple spectral components using a diffraction grating, and projects these spectral components onto an image sensor using a lens. This image sensor is, for example, a line sensor, and detects the multiple spectral components of the interference light LC to generate an electrical signal (detection signal). The detection signal, which includes information on the spectral distribution of the interference light LC, is sent to the arithmetic and control unit 200.

[0041] The OCT unit 100 of some embodiments may include a swept-source OCT optical system. In this case, the light source unit 101 includes, for example, a tunable light source (e.g., a near-infrared tunable laser) that rapidly changes the wavelength of emitted light. Furthermore, the swept-source OCT optical system splits the interference light LC at a predetermined ratio (e.g., 1:1) to generate a pair of interference lights, which are then detected by balanced photodiodes. The balanced photodiodes detect each of the pair of interference lights and output the difference between the pair of detection signals obtained. This difference signal is sent to a data acquisition system (DAQ). The light source unit 101 provides the data acquisition system with a clock synchronized with the wavelength sweep. The data acquisition system samples the difference signal input from the balanced photodiode based on the clock from the light source unit 101. Data obtained by sampling the difference signal is provided to subsequent processing (e.g., image construction).

[0042] In the embodiments shown in Figures 1 and 2, both the measurement arm and the reference arm are provided with optical path length changing elements (retroreflectors 41 and 114). Some embodiments may include an optical path length changing element in only one of the measurement arm or the reference arm. Also, the optical path length changing element is not limited to a retroreflector. In some embodiments, the optical path length changing element in the reference arm may be a movable reflective member (reference mirror). More generally, the optical path length changing element functions to change the relative lengths of the measurement arm and the reference arm, i.e., to move the coherence gate.

[0043] The moving mechanism 150 will be described. The moving mechanism 150 moves the optical system of the ophthalmologic apparatus 1. In some embodiments, the moving mechanism 150 moves at least the fundus camera unit 2 three-dimensionally. This three-dimensional movement is realized by, for example, a combination of movement in the x direction, movement in the y direction, and movement in the z direction.

[0044] The arithmetic and control unit 200 will now be described. The arithmetic and control unit 200 executes various processes such as control, calculation, analysis, etc. This disclosure will describe some non-limiting examples of processes that can be executed by the arithmetic and control unit 200.

[0045] The hardware elements of the arithmetic control unit 200 include, for example, a processor, random access memory (RAM), read-only memory (ROM), a hard disk drive (HDD), a solid-state drive (SSD), and a communication interface. Storage devices such as the HDD and SSD store various computer programs.

[0046] In some embodiments, the arithmetic and control unit 200 includes a user interface (also called a human-machine interface) such as an operation device, an input device, a display device, etc. In some embodiments, at least a part of the user interface is arranged as a peripheral device of the ophthalmologic apparatus 1.

[0047] 3 shows a non-limiting configuration of the arithmetic and control unit 200. The arithmetic and control unit 200 includes a control unit 210, an image construction unit 220, a data processing unit 230, and a user interface 240.

[0048] The control unit 210 includes a processor and controls each unit of the ophthalmic apparatus 1. The control unit 210 includes a main control unit 211 and a memory unit 212. The main control unit 211 includes a processor and controls each element of the ophthalmic apparatus 1 (including some of the elements shown in FIGS. 1 to 3 ). In some embodiments, the main control unit 211 controls external equipment (apparatus, device, system, etc.) connected to the ophthalmic apparatus 1. The functions of the main control unit 211 are realized by cooperation between hardware including circuits and control software. The memory unit 212 includes a storage device such as an HDD or SSD.

[0049] The image constructing unit 220 processes data collected by applying an OCT scan to the fundus Ef of the subject's eye E to generate OCT image data. The image constructing unit 220 includes a processor. The functions of the image constructing unit 220 are realized by cooperation between hardware including circuits and image constructing software. The processing performed by the image constructing unit 220 is similar to image construction in conventional spectral domain OCT. The image constructing unit 220 in some embodiments performs processing similar to image construction in conventional swept-source OCT.

[0050] The image construction unit 220 of this embodiment constructs cross-sectional image data by processing the data (signals including information on the spectral distribution of the interference light LC) generated by the spectroscope 130. This image construction process includes sampling (A / D conversion), denoising, filtering, fast Fourier transform (FFT), and the like, similar to conventional spectral domain OCT.

[0051] The OCT image data constructed by the image constructor 220 is a data set including a set of image data. This set of image data is a set of A-scan image data generated by imaging the reflection intensity profile of each of a plurality of A-lines arranged in the area where the OCT scan was applied. Such OCT image data is an example of an OCT intensity image.

[0052] The OCT image data may be stack data constructed by embedding multiple B-scan image data in a single three-dimensional coordinate system. The image constructor 220 can construct volume data (voxel data) by applying voxelization processing to the stack data. The stack data and volume data are examples of three-dimensional image data in which positions are expressed using a three-dimensional coordinate system (x-y-z coordinate system), and are examples of three-dimensional OCT intensity images.

[0053] The image constructing unit 220 can process the three-dimensional image data. For example, the image constructing unit 220 can generate new image data by applying rendering to the three-dimensional image data. This rendering method may be any method, such as volume rendering, surface rendering, multiplanar reconstruction (MPR), maximum intensity projection (MIP), minimum intensity projection (MIP), or average intensity projection (AIP).

[0054] As a non-limiting example of rendering, the image constructor 220 may generate projection image data by integrating 3D image data in the z direction. As another example, the image constructor 220 may construct shadowgram data by integrating a portion of the 3D image data (3D partial image data) in the z direction. The 3D partial image data is extracted from the 3D image data using a known segmentation method. The projection image data and shadowgram data are 2D image data whose positions are expressed using a 2D coordinate system (xy coordinate system). The direction in which the 3D image data is integrated is not limited to the z direction. Therefore, the image constructor 220 can generate 2D image data of any orientation from the 3D image data.

[0055] The ophthalmic apparatus 1 can apply OCT angiography to the fundus Ef of the subject's eye E. In OCT angiography, the ophthalmic apparatus 1 repeatedly performs OCT scans targeting the same region of the fundus Ef a predetermined number of times. The image construction unit 220 obtains difference data from the data sets collected by these repeated scans and constructs a motion contrast image based on this difference data. This motion contrast image is an image generated by emphasizing interference signals that change over time due to blood flow in the blood vessels of the fundus Ef, and is an angiography image that represents the distribution of blood vessels in the fundus Ef (actually, the distribution of blood flow). Typically, the ophthalmic apparatus 1 applies OCT angiography to a three-dimensional region of the fundus Ef to obtain multiple three-dimensional image data (data sets), and creates three-dimensional angiography image data based on the difference data of the multiple three-dimensional image data. This three-dimensional angiography image data includes information indicating the three-dimensional distribution of blood vessels in the fundus Ef.

[0056] The image constructing unit 220 can construct any two-dimensional angiographic image data and / or any pseudo-three-dimensional angiographic image data from the three-dimensional angiographic image data. For example, the image constructing unit 220 can generate two-dimensional angiographic image data representing any cross-section of the fundus Ef by applying multiplanar reconstruction or projection to the three-dimensional angiographic image data. The image constructing unit 220 can also apply segmentation to the three-dimensional angiographic image data to identify image regions (slabs) corresponding to specific tissues of the fundus Ef and generate shadowgram data (two-dimensional angiographic image data) from the identified slabs. The two-dimensional angiographic image data is typically en face image data whose position is expressed in an xy coordinate system. The image constructing unit 220 can generate en face image data for various depth areas, such as the superficial retina, deep retina, and choroid. In observing the blood vessels of the fundus using OCT angiography, any of the following slabs may be used: SCP (superfical capillary plexus), DCP (deep capillary plexus), OR (outer retina), ORCC (outer retina chorio-capillaries), CC (chorio-capillaries), and C (choroid). The image constructor 220 can identify one or more of these slabs from the three-dimensional angiographic image data and generate two-dimensional angiographic image data corresponding to each slab.

[0057] The data processing unit 230 performs various types of data processing. For example, the data processing unit 230 processes images (fundus images, anterior segment images, etc.) acquired by the fundus camera unit 2, and processes images (OCT images) acquired using OCT scanning. The data processing unit 230 includes a processor. The data processing unit 230 is realized by cooperation between hardware including circuits and data processing software. Some non-limiting examples of processing performed by the data processing unit 230 are described below.

[0058] The user interface 240 includes a display unit 241 and an operation unit 242. The display device 3 in Fig. 1 is an example of the display unit 241. The operation unit 242 includes an operation device and an input device. Examples of devices in the operation unit 242 include a switch, a lever, a mouse, a trackball, a keyboard, and an operation panel.

[0059] An exemplary configuration and operation of an ophthalmic apparatus 1 realized by the elements (hardware elements, software elements) shown in Figures 1 to 3 will be described. Figure 4 shows a non-limiting example of the configuration of the ophthalmic apparatus 1. Figures 5 to 7 show several non-limiting modifications of the ophthalmic apparatus 1 of Figure 4. The items related to the examples shown in Figures 4 to 7 can be combined in any manner.

[0060] 4 includes an image acquisition unit 1000, a blood vessel distribution image generation unit 1100, a processed image generation unit 1200, and a display control unit 1300. The display device 1400 may be an element of the ophthalmic apparatus 1 (the display unit 241) or an external device connected to the ophthalmic apparatus 1.

[0061] The image acquisition unit 1000 is configured to acquire an image (OCT angiography image, first fundus image) generated by applying OCT angiography to the fundus Ef of the subject's eye E, and an image (second fundus image) generated by applying an imaging modality different from OCT angiography to the fundus Ef. The image acquisition unit 1000 acquires one or more first fundus images and one or more second fundus images. Some non-limiting examples of the image acquisition unit 1000 are described below.

[0062] First, some non-limiting examples of images acquired by the image acquisition unit 1000 will be described. The data format of the first fundus image (OCT angiography image) may be arbitrary. In some non-limiting examples, the first fundus image may be a three-dimensional OCT angiography image representing a three-dimensional region of the fundus Ef. This three-dimensional OCT angiography image may include any of the following non-limiting examples: a data set consisting of multiple A-scan angiography images; stack data based on multiple A-scan angiography images; a data set consisting of multiple B-scan angiography images; stack data based on multiple B-scan angiography images; or volume data constructed from stack data based on multiple A-scan angiography images or multiple B-scan angiography images.

[0063] In some other non-limiting examples, the first fundus image may be a two-dimensional OCT angiography image, which may be any of the following non-limiting examples: a B-scan image representing a cross section along the z direction (depth direction); an enface image in which pixel positions are defined by an xy coordinate system; or an image representing an oblique cross section oriented at a non-zero angle relative to all of the x, y, and z directions.

[0064] The data format of the second fundus image may be any format. In some non-limiting examples, the second fundus image may be any of the following non-limiting examples: an image generated using a fundus camera (fundus camera image), an image generated using an SLO (SLO image), or an image generated using OCT (a three-dimensional or two-dimensional OCT intensity image).

[0065] The image acquisition unit 1000 is configured to perform any of the following processes: generate both a first fundus image and a second fundus image; generate a first fundus image and receive a second fundus image from outside; receive a first fundus image from outside and generate a second fundus image; receive both a first fundus image and a second fundus image from outside.

[0066] When generating a first fundus image, the image acquisition unit 1000 applies OCT angiography to the fundus Ef of the subject's eye E. According to the configuration shown in Figures 1 to 3, the image acquisition unit 1000 is realized by a combination of a fundus camera unit 2 (a group of elements forming a measurement arm), an OCT unit 100, an image construction unit 220, and a control unit 210 that controls these.

[0067] When a fundus camera image is generated as the second fundus image, the image acquisition unit 1000 includes a fundus camera. According to the configuration shown in FIGS. 1 to 3, the image acquisition unit 1000 is realized by combining a fundus camera unit 2 and a control unit 210. Similarly, when an SLO image is generated as the second fundus image, the image acquisition unit 1000 includes an SLO (not shown). When an OCT intensity image is generated as the second fundus image, the image acquisition unit 1000 includes a fundus camera unit 2 (a group of elements forming a measurement arm), an OCT unit 100, an image construction unit 220, and a control unit 210 that controls these. The same applies when the second fundus image is a different type of image.

[0068] When at least one of the first fundus image and the second fundus image is received from an external device, the image acquisition unit 1000 includes, for example, at least one of a communication device that receives data via a communication line and a reading device that reads data recorded on a recording medium. The communication device may be the communication interface (described above) of the arithmetic and control unit 200. The image acquisition unit 1000 having the communication device acquires images stored in an external device (e.g., an image archiving system, an ophthalmic imaging device, a storage device, etc.) via the communication line. The reading device includes a recording medium gate (not shown) and a control unit 210 that functions as a gate controller.

[0069] In some embodiments, the image acquisition unit 1000 includes a device (e.g., a communication interface, or a recording medium gate and gate controller) that accepts data acquired from the fundus Ef, and a device (e.g., an image construction unit 220) that constructs an image from the accepted data.

[0070] The following describes the blood vessel distribution image generating unit 1100. The blood vessel distribution image generating unit 1100 is configured to generate a blood vessel distribution image from the first fundus image (OCT angiography image) acquired by the image acquiring unit 1000.

[0071] The vascular distribution image generation unit 1100 is realized by cooperation between hardware including circuits and vascular distribution image generation software. In some non-limiting examples, the vascular distribution image generation unit 1100 executes at least some of the steps of the vascular distribution image generation process using a pre-created machine learning model. According to the configurations shown in FIGS. 1 to 3 , at least some of the functions of the vascular distribution image generation unit 1100 are realized by the image construction unit 220. In some non-limiting examples, some of the functions of the vascular distribution image generation unit 1100 may be realized by the data processing unit 230.

[0072] The OCT angiography image contains information resulting from blood flow in the fundus Ef, noise resulting from the OCT scan, noise resulting from data processing, etc. The blood vessel distribution image generation unit 1100 identifies information corresponding to the blood vessels of the fundus Ef from the OCT angiography image and generates a blood vessel distribution image showing the positions of the blood vessels (vascular distribution).

[0073] In some non-limiting examples, the vascular distribution image generating unit 1100 generates a vascular distribution image so as to reflect all vascular information included in the OCT angiography image without omission. In other non-limiting examples, the vascular distribution image generating unit 1100 selectively extracts information corresponding to blood vessels that satisfy predetermined conditions to generate a vascular distribution image. For example, the vascular distribution image generating unit 1100 selectively extracts information corresponding to blood vessels that belong to a predetermined size range (range of blood vessel diameter values) to generate a vascular distribution image. Some non-limiting examples of the vascular distribution image generating unit 1100 are described below.

[0074] First, some non-limiting examples of vascularity images generated by the vascularity image generating unit 1100 will be described. An OCT angiography image is an image obtained by visualizing and emphasizing areas where blood flow exists. The vascularity image may be an image in any data format that can be generated from an OCT angiography image. For example, the vascularity image may be a front image, a B-scan image, an MPR image of an arbitrary cross section, or a rendering image (e.g., a volume rendering image or a surface rendering image).

[0075] Some non-limiting examples of vascularity images are images that depict images of blood vessels (vascularity). Other non-limiting examples of vascularity images are images that depict, in addition to images of blood vessels, objects or events other than blood vessels. These objects or events can be depicted by visualizing blood flow. Examples of these include: vascular collapse, vascular detachment, blood leakage, vascular occlusion, ocular fundus ischemia, vascular aneurysm, vascular abnormality, vascular inflammation, and various lesions. The "status" of these items is evaluated and / or quantified using predetermined indicators (e.g., location, size, degree, etc.) and further visually represented.

[0076] The vascular distribution image generation unit 1100 performs processing according to the type of vascular distribution image to be generated. When the vascular distribution image to be generated is an enface image, the first fundus image is a 3D OCT angiography image. The vascular distribution image generation unit 1100 is configured to apply a projection in the z direction (depth direction, A-scan direction) to a portion or the entire 3D OCT angiography image. When generating a front image from a portion of the 3D OCT angiography image, the vascular distribution image generation unit 1100 extracts a partial image to which the projection is applied from the 3D OCT angiography image. This partial image extraction process is performed using any segmentation method. The segmentation method used may be a method using a machine learning model, a method not using a machine learning model, or a method that at least partially combines these methods.

[0077] When the generated vascular distribution image is a B-scan image, the first fundus image is, for example, a 3D OCT angiography image formed as stack data or volume data. When the first fundus image is stack data, the vascular distribution image generation unit 1100 selects a B-scan image corresponding to an automatically or manually specified cross section from multiple B-scan images included in the stack data. Alternatively, the vascular distribution image generation unit 1100 generates a B-scan image corresponding to an automatically or manually specified cross section based on the stack data. This B-scan image generation process includes, for example, selecting two or more B-scan images located near the specified cross section from the stack data and combining the selected two or more B-scan images to generate a B-scan image corresponding to the specified cross section. When the first fundus image is volume data, the vascular distribution image generation unit 1100 applies, for example, any rendering process to the volume data for generating a two-dimensional image from a three-dimensional image. This rendering process may be performed by any method, for example, a method similar to the above-mentioned front image generation method or MPR.

[0078] When the generated vascular distribution image is a cross-sectional image representing an arbitrary cross section, the first fundus image is, for example, a three-dimensional OCT angiography image formed as stack data or volume data. The vascular distribution image generating unit 1100 generates the cross-sectional image by applying an arbitrary rendering process for generating a two-dimensional image from a three-dimensional image to the volume data, similar to the above-described B-scan image generating method.

[0079] When the generated vascularity image is a rendering image, the first fundus image is a 3D OCT angiography image. The vascularity image generator 1100 generates the rendering image by applying a rendering process to the 3D OCT angiography image according to the type of rendering image to be generated. In some non-limiting examples, the vascularity image generator 1100 applies volume rendering or surface rendering to the 3D OCT angiography image formed as volume data.

[0080] The vascular distribution image generation unit 1100 may be configured to perform processing before the above processing (pre-processing) and / or processing after the above processing (post-processing) for the purpose of improving the quality of the processing content and output content according to the embodiment. Non-limiting examples of pre-processing include resizing, cropping, brightness correction, contrast correction, and denoising. Non-limiting examples of post-processing include edge detection, filtering, enhancement processing, pseudocolor processing, resizing, cropping, brightness correction, contrast correction, and denoising. The pre-processing and post-processing may be a method using a machine learning model, a method not using a machine learning model, or a method that at least partially combines these methods.

[0081] The vascular distribution image generating unit 1100 may perform masking. Masking is image processing that displays a specific portion (region of interest) of a target image and converts the remaining portion into a black image or a white image. The region of interest in the masking process performed by the vascular distribution image generating unit 1100 is a blood flow region where blood flow exists (typically, a vascular region corresponding to a blood vessel). Any method may be used to detect a vascular region, and may be, for example, a method that uses a machine learning model, a method that does not use a machine learning model, or a method that at least partially combines these methods. Non-limiting examples of vascular region detection methods include binarization, a matched filter method, a morphology method, a vascular tracking method, a clustering method, a dynamic model method, and a threshold method.

[0082] The vascularity image generator 1100 may apply a vascular enhancement filter to the OCT angiography image or the vascularity image to enhance the image of blood vessels depicted in the image. Non-limiting examples of the vascular enhancement filter include a multiscale Franzi filter, a Gabor filter, a non-local means filter, and a wavelet filter.

[0083] The vascular distribution image generation unit 1100 may use any of the following methods to enhance the vascular image: a method using vector concentration using a density gradient; a method using a black top-hat transform; a method using a double ring filter; a method combining a black top-hat transform and a double ring filter; a method of detecting ridges of gray values ​​from a green component image of a color image; a method using a Gabor wavelet; a method using a matched filter; a method using a Hough transform; a method using a Contourlet transform; a method using a Curvelet transform; a method based on ensemble learning; a method using a morphological filter bank; or a method using machine learning.

[0084] In some non-limiting examples, the vascular distribution image generation unit 1100 may be configured to generate either or both of an arterial distribution image and a venous distribution image. The arterial distribution image is an image showing the distribution of arteries in the fundus Ef, and the venous distribution image is an image showing the distribution of veins in the fundus Ef. Each of the arterial distribution image and the venous distribution image is a non-limiting example of a vascular distribution image generated by the vascular distribution image generation unit 1100.

[0085] An example of generating both an arterial distribution image and a venous distribution image will be described below. An example of generating either an arterial distribution image or a venous distribution image may also be employed. An example of generating either an arterial distribution image or a venous distribution image can be easily understood from the description of the example of generating both.

[0086] Figure 5 shows one non-limiting example of the vascularity image generator 1100 of Figure 4. The vascularity image generator 1110 of Figure 5 generates both arterial and venous distribution images.

[0087] The vascular distribution image generation unit 1110 includes a vascular classification unit 1111, an arterial distribution image generation unit 1112, and a venous distribution image generation unit 1113. The vascular distribution image generation unit 1110 generates a vascular distribution image to be input to the vascular classification unit 1111. This vascular distribution image is an image obtained in the middle of the process of generating an arterial distribution image and a venous distribution image, and may be referred to as an intermediate vascular distribution image. The vascular distribution image generation unit 1110 explicitly or implicitly includes an element for generating the intermediate vascular distribution image (an intermediate vascular distribution image generation unit; not shown). In the intermediate vascular distribution image, no distinction has been made between arterial images and venous images, or a provisional distinction has already been made.

[0088] A plurality of blood vessel images are depicted in the intermediate blood vessel distribution image. The blood vessel classification unit 1111 classifies each blood vessel image depicted in the intermediate blood vessel distribution image into an artery image or a vein image. In other words, the blood vessel classification unit 1111 identifies whether each blood vessel depicted in the intermediate blood vessel distribution image is an artery or a vein.

[0089] The method of the blood vessel image classification process (artery / vein discrimination process) may be arbitrary. A non-limiting first example of the blood vessel image classification process determines whether a target blood vessel is an artery, and if it is determined to be an artery, classifies the blood vessel as an artery, and if it is determined not to be an artery, classifies the blood vessel as a vein. A second example determines whether a target blood vessel is a vein, and if it is determined to be a vein, classifies the blood vessel as a vein, and if it is determined not to be a vein, classifies the blood vessel as an artery. A third example determines whether a target blood vessel is an artery and whether it is a vein, and classifies the blood vessel based on the results of the two determinations. A fourth example determines whether a target blood vessel is an artery (first determination), and if it is determined to be an artery, classifies the blood vessel as an artery, and if it is determined not to be an artery, determines whether it is a vein (second determination). In the fifth example, a determination is made as to whether or not a target blood vessel is a vein (first determination), and if it is determined to be a vein, the blood vessel is classified as a vein, and if it is determined not to be a vein, a determination is made as to whether or not the target blood vessel is an artery (second determination). If a negative determination result is obtained in the second determination in the fourth or fifth example, the blood vessel image classification process performs, for example, any of the following steps: assigning information indicating indeterminability to the blood vessel; generating an order for identification by a doctor; generating an order for identification by another classifier; and classifying the blood vessel as an artery or a vein based on at least the results of the first determination and the second determination.

[0090] In some non-limiting examples, the blood vessel classification unit 1111 applies a blood vessel image classification process to the intermediate blood vessel distribution image using a second fundus image acquired together with a first fundus image (OCT angiography image), which is the original image of the intermediate blood vessel distribution image. The second fundus image in this example is a type of image that can distinguish between arterial and venous images. Typically, the artery / vein identification process for the second fundus image in this example has higher quality (high accuracy, high precision, high reproducibility, etc.) than the identification process for the OCT angiography image and / or has different characteristics from the identification process for the OCT angiography image (e.g., the diameter of the blood vessels to be identified is different). The second fundus image in this example may be a fundus camera image or an SLO image. For artery / vein identification processes that can be applied to the second fundus image, see U.S. Patent Application Publication No. 2023 / 03036720 and U.S. Patent No. 9,924,867, etc. However, the artery / vein identification process applicable to the second fundus image is not limited to these. The blood vessel classification unit 1111 of this example may be configured to perform a first step of applying a blood vessel classification process to the second fundus image to obtain the distribution of arterial images and the distribution of vein images in the second fundus image, a second step of performing registration between the first fundus image and the second fundus image, a third step of identifying an area in the intermediate blood vessel distribution image corresponding to the arterial image distribution obtained in the first step based on the registration result, and a fourth step of identifying an area in the intermediate blood vessel distribution image corresponding to the vein image distribution obtained in the first step based on the registration result. Through the third and fourth steps, each blood vessel image depicted in the intermediate blood vessel distribution image is identified as either an artery image or a vein image. Note that some non-limiting examples do not perform the second step (registration). For example, if the first fundus image and the second fundus image are acquired substantially simultaneously and there is no positional misalignment between these fundus images, or if the positional misalignment can be ignored, the second step is omitted.

[0091] In some other non-limiting examples, the image acquisition unit 1000 acquires a third fundus image of the fundus oculi Ef along with the first and second fundus images. The third fundus image may be a fundus camera image or an SLO image. The second fundus image in this example is, for example, an OCT intensity image. The blood vessel classification unit 1111 in this example can perform the process according to this example in the same manner as the blood vessel image classification process using the second fundus image described above. That is, the blood vessel classification unit 1111 in this example may be configured to execute a first step of applying blood vessel classification processing to the third fundus image to determine the distribution of arterial images and the distribution of vein images in the third fundus image, a second step of performing alignment (registration) between the first fundus image and the third fundus image, a third step of identifying an area in the intermediate blood vessel distribution image corresponding to the arterial image distribution obtained in the first step based on the registration result, and a fourth step of identifying an area in the intermediate blood vessel distribution image corresponding to the vein image distribution obtained in the first step based on the registration result.

[0092] In still other non-limiting examples, the blood vessel classification unit 1111 analyzes an OCT angiography image to perform blood vessel image classification processing. The analyzed OCT angiography image may be a first fundus image (OCT angiography image) or a mid-vessel distribution image, and / or a third fundus image of the fundus Ef acquired by the image acquisition unit 1000 together with the first and second fundus images. For artery / vein identification processing in OCT images such as OCT angiography images, see U.S. Patent Application Publication No. 2020 / 0394789, etc. In addition, arterial and venous images can also be distinguished by detecting a capillary-free zone around the retinal artery from the OCT angiography image.

[0093] The artery distribution image generating unit 1112 will now be described. The artery distribution image generating unit 1112 generates an artery distribution image that represents the distribution of images of blood vessels (artery images) classified as arteries by the blood vessel classifying unit 1111.

[0094] In some non-limiting examples, the arterial distribution image generator 1112 generates an arterial distribution image (arterial mask image) by converting the brightness of areas other than the arterial image in the intermediate blood vessel distribution image to zero (black image).

[0095] The artery distribution image generation unit 1112 may convert the luminance of the artery image in the intermediate blood vessel distribution image. For example, the artery distribution image generation unit 1112 can convert the luminance of the artery image in the intermediate blood vessel distribution image to a predetermined value. The luminance of the converted artery image may be constant or non-constant. For example, the artery distribution image generation unit 1112 generates an artery mask image by converting the luminance of the artery image in the intermediate blood vessel distribution image to a predetermined non-zero value and converting the luminance of areas other than the artery image to zero (black image).

[0096] The vein distribution image generating unit 1113 will now be described. The vein distribution image generating unit 1113 generates a vein distribution image that represents the distribution of images of blood vessels (vein images) classified as veins by the blood vessel classifying unit 1111.

[0097] In some non-limiting examples, the vein distribution image generator 1113 generates a vein distribution image (vein mask image) by converting the brightness of areas other than the vein image in the intermediate vascular distribution image to zero (black image).

[0098] The vein distribution image generation unit 1113 may convert the brightness of the vein image in the intermediate blood vessel distribution image. For example, the vein distribution image generation unit 1113 can convert the brightness of the vein image in the intermediate blood vessel distribution image to a predetermined value. The brightness of the converted vein image may be constant or non-constant. For example, the vein distribution image generation unit 1113 converts the brightness of the vein image in the intermediate blood vessel distribution image to a predetermined non-zero value and converts the brightness of areas other than the vein image to zero (black image), thereby generating a vein mask image.

[0099] The brightness (non-zero value) of the vein image after conversion may be equal to or different from the brightness (non-zero value) of the artery image after conversion. When different brightness values ​​are used, a user observing the displayed image can visually distinguish between the artery image and the vein image based on their brightness.

[0100] Visual information other than brightness can be added to arterial images in the arterial distribution image (e.g., arterial mask image) and venous images in the venous distribution image (e.g., venous mask image). In some non-limiting examples, the vascular distribution image generator 1110 can apply pseudocolor processing to one or both of the arterial distribution image and the venous distribution image. For example, the vascular distribution image generator 1110 applies pseudocolor processing to the arterial distribution image, which assigns a first color to the arterial image, and applies pseudocolor processing to the venous distribution image, which assigns a second color different from the first color. For example, the first color is red and the second color is blue. By assigning different colors to the arterial image and the venous image, a user viewing the displayed image can visually distinguish between the arterial image and the venous image by color. Note that the pseudocolor processing may be performed by an element other than the vascular distribution image generator 1110 (e.g., the display control unit 1300).

[0101] The processed image generating unit 1200 will now be described. The processed image generating unit 1200 applies a predetermined processing to the second fundus image acquired by the image acquiring unit 1000. The image generated in this way is called a processed image.

[0102] The processed image generation unit 1200 is realized by cooperation between hardware including a circuit and processed image generation software. In some non-limiting examples, the processed image generation unit 1200 executes at least some of the steps of the processed image generation process using a pre-created machine learning model. According to the configurations shown in FIGS. 1 to 3 , at least some of the functions of the processed image generation unit 1200 are realized by the image construction unit 220. In some non-limiting examples, some of the functions of the processed image generation unit 1200 may be realized by the data processing unit 230.

[0103] The second fundus image is an image generated by applying an imaging modality different from that of the OCT angiography image to the fundus Ef. The type of the second fundus image may be any type, such as a fundus camera image, an SLO image, or an OCT intensity image. When two or more second fundus images are acquired by the image acquisition unit 1000, the processed image generation unit 1200 generates a processed image by applying a predetermined processing process to each of the second fundus images, or by applying a predetermined processing process to one or more fundus images selected from the second fundus images. When generating a processed image from two or more second fundus images, the same processing process may be applied to the second fundus images, or different processing processes may be applied to the second fundus images. Furthermore, two or more processing processes may be applied to one second fundus image.

[0104] In some non-limiting examples, the processed image generating unit 1200 generates a processed image having a data format corresponding to a vascularity image generated from a first fundus image (OCT angiography image) acquired together with the second fundus image, and also performs image processing to improve the image quality of the second fundus image.

[0105] Non-limiting examples of image processing performed by the processed image generation unit 1200 include resizing, cropping, brightness correction, contrast correction, denoising, enhancement processing, segmentation, registration, rendering (e.g., generation of any of a front image, a cross-sectional image, a volume rendering image, a surface rendering image, and other types of rendering images), etc. Image processing may be processing using a machine learning model, processing without using a machine learning model, or processing that at least partially combines these processes.

[0106] In the above example, the processed image is generated based on the data format of the vascular distribution image. However, the relationship between the vascular distribution image generation process and the processed image generation process is not limited to this. In some non-limiting examples, the vascular distribution image generation process is performed based on the data format of the processed image, or the vascular distribution image generation process and the processed image generation process are performed based on a predetermined data format.

[0107] In some non-limiting examples, the second fundus image acquired by the image acquisition unit 1000 is provided to the display control unit 1300 without image processing being applied thereto. Non-limiting examples of cases in which the processed image generation process is omitted include when the data format of the first fundus image and the data format of the second fundus image are the same, when adjustment of the data format is not required, when improvement of the image quality of the second fundus image is not required, etc.

[0108] In some non-limiting examples, the processed image generation process may not always be performed, in which case the processed image generation unit 1200 is omitted. Alternatively, the processed image generation unit 1200 in this example may be considered to apply an identity transformation to the second fundus image.

[0109] In some non-limiting examples, the processed image generating unit 1200 may be configured to determine whether to execute the processed image generating process on the input second fundus image. This determination may be made based on, for example, any one or more of the data format of the first fundus image, the data format of the second fundus image, the image quality of the second fundus image, the content of subsequent processing, and a user instruction.

[0110] In some non-limiting examples, the second fundus image includes a two-dimensional image (e.g., a fundus camera image, an SLO image, or a two-dimensional OCT intensity image). The processed image generation unit 1200 can generate a processed image corresponding to an area including at least a portion of the area of ​​the fundus Ef depicted in the vascular distribution image by applying a predetermined processing process to this two-dimensional image. The processed image generation process in this example may include resizing, cropping, registration, etc. The range of the area of ​​the fundus Ef depicted in the vascular distribution image (first fundus area) and the range of the area of ​​the fundus Ef depicted in the processed image (second fundus area) may or may not match each other. When the first fundus area and the second fundus area do not match, at least a portion of the first fundus area and at least a portion of the second fundus area match. In other words, the first fundus image and the second fundus image are images obtained by respectively capturing at least a partially common area of ​​the fundus Ef.

[0111] In some non-limiting examples, the second fundus image includes a three-dimensional image (e.g., a three-dimensional OCT intensity image). The processed image generation unit 1200 can generate a processed image corresponding to an area including at least a portion of the area of ​​the fundus Ef depicted in the blood vessel distribution image by applying a predetermined processing process to this three-dimensional image. The processed image generation process in this example may include rendering (e.g., generation of a front image), resizing, cropping, registration, etc. The relationship between the range of the area of ​​the fundus Ef depicted in the blood vessel distribution image (first fundus area) and the range of the fundus Ef depicted in the processed image (second fundus area) may be the same as when the second fundus image is a two-dimensional image.

[0112] Here, registration will be explained. Registration is the alignment of two (or more) images. There are various registration methods, which can be broadly divided into methods that use machine learning models and methods that do not. A typical example of a method that does not use a machine learning model includes a process of detecting common landmarks from each image and a process of relatively moving these images so that the landmarks in each image match. Typically, high-contrast areas in each image are used as landmarks.

[0113] Non-limiting examples of registrations that can be performed by the ophthalmologic apparatus 1 of this embodiment include: registration between a first fundus image and a second fundus image; registration between at least one of the first fundus image and the second fundus image and another image; registration between a first fundus image and a processed image; registration between a processed image and another image; registration between a vascular distribution image and a second fundus image; registration between a vascular distribution image and another image; registration between at least one of an arterial distribution image and a venous distribution image and the second fundus image; registration between at least one of an arterial distribution image and a venous distribution image and another image; registration between a vascular distribution image and a processed image. Any method may be used for each registration.

[0114] If there is no misalignment between the first and second fundus images or if the misalignment is negligible, registration between the first fundus image (or an image generated from the first fundus image) and the second fundus image (or an image generated from the second fundus image) is not required. Non-limiting examples of such cases include when the photographing to generate the first fundus image and the photographing to generate the second fundus image are performed substantially simultaneously, when the first and second fundus images are generated from the same data, when the first and second fundus images have already been registered, etc. Here, the term "substantially simultaneously" is intended to allow a time difference such that the amount of misalignment due to eye movement is equal to or less than a predetermined threshold.

[0115] The display control unit 1300 will be described. The display control unit 1300 performs visualization based on the blood vessel distribution image generated from the first fundus image by the blood vessel distribution image generation unit 1100 and the processed image generated from the second fundus image by the processed image generation unit 1200. The image provided by this visualization is called a composite image. The display control unit 1300 controls the display device 1400 to display the composite image.

[0116] The display control unit 1300 is realized by the cooperation of hardware including circuits and display control software. According to the configuration shown in Figures 1 to 3, at least some of the functions of the display control unit 1300 are realized by the control unit 210.

[0117] In some non-limiting examples, the display controller 1300 generates a composite image based on a vascularity image and a processed image. In another example, the display controller 1300 generates a composite image based on three or more images including a vascularity image, a processed image, and another image. More generally, the display controller 1300 generates a composite image based at least on one or more vascularity images and one or more processed images.

[0118] The composite image may have any data format or representation, i.e., may be any visualization that can be generated using both the vascularity image and the processed image.

[0119] In some non-limiting examples, the display control unit 1300 uses a display interface with a layer function. The display control unit 1300 in this example executes a process of preparing multiple layers, a process of displaying images on each layer, and a process of presenting multiple layers in an overlapping manner. The process of presenting multiple layers in an overlapping manner is a process of arranging multiple layers in an overlapping manner in the display space. According to this example, the user can simultaneously observe the vascular distribution image and the processed image, and can easily grasp the positional relationship between the objects shown in the vascular distribution image and the objects shown in the processed image.

[0120] A specific example will be described. The display control unit 1300 displays a vascular distribution image on a first layer, a processed image on a second layer, and presents the first and second layers in an overlapping manner. More generally, the display control unit 1300 displays k1 vascular distribution images on layers N1(1) to N1(k1), k2 processed images on layers N2(1) to N2(k2), and k3 separate images on layers N3(1) to N3(k3), and presents these (k1+k2+k3) layers in an overlapping manner. Here, k1 is an integer equal to or greater than 1, k2 is an integer equal to or greater than 1, and k3 is an integer equal to or greater than 0.

[0121] In some non-limiting examples, the display control unit 1300 selectively displays a plurality of images including at least a vascular distribution image and a processed image on the display device 1400. For example, the display control unit 1300 selects an image based on an instruction using the operation unit 242. In another example, the display control unit 1300 selects an image according to a display control program. The display control unit 1300 may select two or more images and display them overlapping or side by side.

[0122] In some non-limiting examples, the display control unit 1300 sequentially displays a plurality of images including at least a blood vessel distribution image and a processed image on the display device 1400. The order in which the plurality of images are displayed may be determined in advance or may be determined by the user or the ophthalmologic apparatus 1. The display control unit 1300 may switch between and display the plurality of images according to, for example, a display control program. Furthermore, the display control unit 1300 may switch the image to be displayed in response to an instruction from the user.

[0123] In some non-limiting examples, the display control unit 1300 combines a plurality of images including at least a vascular distribution image and a processed image, and displays the generated composite image on the display device 1400. The composite image generation function of the display control unit 1300 in this example is realized by at least one of the image construction unit 220 and the data processing unit 230. Any method may be used for the composite image generation process, and may be, for example, either alpha blending or Poisson image synthesis.

[0124] FIG. 6 shows one non-limiting example of the display control unit 1300. The display control unit 1310 in FIG. 6 includes an opacity determination unit 1311. The display control unit 1310 prepares an alpha channel layer for which an alpha value can be set. This alpha value is used as opacity. Furthermore, the display control unit 1310 prepares a layer (referred to as a lower layer) to be placed below this alpha channel layer.

[0125] The display control unit 1310 displays the blood vessel distribution image generated from the first fundus image by the blood vessel distribution image generation unit 1100 on the alpha channel layer. Furthermore, the display control unit 1310 displays the processed image generated from the second fundus image by the processed image generation unit 1200 on the lower layer. As a result, an overlay image in which the blood vessel distribution image is superimposed on the processed image is displayed on the display device 1400. This overlay image is an example of a composite image.

[0126] The opacity determination unit 1311 determines an alpha value (opacity) of each pixel of the alpha channel layer on which the blood vessel distribution image is displayed. The opacity determination unit 1311 may be configured to determine the opacity of each pixel based on the blood vessel distribution image.

[0127] In some non-limiting examples, the opacity determination unit 1311 sets the opacity of each pixel in a region corresponding to a blood vessel in the blood vessel distribution image (vascular region, blood vessel image) to a relatively large value, and sets the opacity of each pixel in a region other than the blood vessel region (background region, non-vascular region) to a relatively small value. In other words, the opacity determination unit 1311 in this example sets the opacity of each pixel (first pixel group) constituting the blood vessel region to a first value, and sets the opacity of each pixel (second pixel group) constituting the background region to a second value. Here, the opacity of the background region (second value) is smaller than the opacity of the blood vessel region (first value). For example, the opacity of the background region (second value) is set to zero.

[0128] The configuration shown in Fig. 5 can be combined with the configuration shown in Fig. 6. The ophthalmologic apparatus 1 of this example includes the image acquisition unit 1000 shown in Fig. 4 to Fig. 6, the blood vessel distribution image generation unit 1110 shown in Fig. 5, the processed image generation unit 1200 shown in Fig. 4 to Fig. 6, the display control unit 1310 shown in Fig. 6, and the display device 1400 shown in Fig. 4 to Fig. 6.

[0129] The vascular distribution image generating unit 1110 of the ophthalmologic apparatus 1 of this example generates at least one of an arterial distribution image and a venous distribution image from the first fundus image (OCT angiography image) acquired by the image acquiring unit 1000. The processed image generating unit 1200 applies a predetermined processing process to the second fundus image acquired by the image acquiring unit 1000 together with the first fundus image to generate a processed image.

[0130] When an arterial distribution image is generated, the display control unit 1310 can execute at least a process of displaying the arterial distribution image on an alpha channel layer and a process of displaying a processed image on a lower layer.

[0131] When a vein distribution image is generated, the display control unit 1310 can execute at least a process of displaying the vein distribution image on an alpha channel layer and a process of displaying a processed image on a lower layer.

[0132] When both the arterial distribution image and the venous distribution image are generated, the display control unit 1310 can execute a process of displaying the arterial distribution image on a first alpha channel layer, a process of displaying the venous distribution image on a second alpha channel layer, and a process of displaying the processed image on a lower layer, which is a layer disposed below both the first alpha channel layer and the second alpha channel layer.

[0133] In this example, in order to make the arterial distribution image and the venous distribution image visually distinguishable from each other, the display mode of the arterial distribution image and the venous distribution image can be made different from each other. For example, the display control unit 1310 assigns a first color (typically red) to each pixel forming an artery image in the arterial distribution image, and assigns a second color (typically blue) to each pixel forming a vein image in the venous distribution image.

[0134] The display control unit 1310 of this example can selectively display a first alpha channel layer displaying an arterial distribution image and a second alpha channel layer displaying a venous distribution image together with a lower layer displaying a processed image. This makes it possible to selectively provide or switch between a visualization in which an arterial distribution image is overlaid on a processed image and a visualization in which a venous distribution image is overlaid on a processed image. The selection or switching of visualizations can be performed automatically or manually.

[0135] In the above example, the arterial distribution image and the venous distribution image are displayed on separate alpha channel layers in order to present the arterial image and the venous image in different ways. However, the method for presenting the arterial image and the venous image in a manner that allows them to be distinguished from each other is not limited to the above example. Several non-limiting examples that achieve a similar effect will be described below.

[0136] In one example, the vascular distribution image generation unit 1110 generates a vascular distribution image depicting both arterial and venous images, identifies the arterial and venous images in the vascular distribution image, assigns a first color to each pixel of the arterial image, and assigns a second color to each pixel of the venous image. The display control unit 1310 displays the vascular distribution image on an alpha channel layer and an edited image on a lower layer. This makes it possible to present the arterial and venous images in a manner that allows them to be distinguished from each other using a single alpha channel layer and lower layer.

[0137] In another example, the vascular distribution image generation unit 1110 generates a vascular distribution image in which both an arterial image and a vein image are depicted, identifies the arterial image and the vein image in the vascular distribution image, assigns first incidental information to the arterial image, and assigns second incidental information to the vein image. The assignment of the first incidental information to the arterial image may be performed, for example, by assigning the first incidental information to each pixel of the arterial image or by assigning the first incidental information to a range of the arterial image. The first incidental information may be, for example, a label indicating that the image is an arterial image or a label indicating a display mode. Similarly, the assignment of the second incidental information to the vein image may be performed, for example, by assigning the second incidental information to each pixel of the vein image or by assigning the second incidental information to a range of the vein image. The second incidental information may be, for example, a label indicating that the image is a vein image or a label indicating a display mode. The display control unit 1310 assigns a first color to the artery image based on the first incidental information and a second color to the vein image based on the second incidental information, and then displays the resulting vascular distribution image on the alpha channel layer. Furthermore, the display control unit 1310 displays the processed image on the lower layer. This makes it possible to present the artery image and the vein image in a manner that allows them to be distinguished from each other using a single alpha channel layer and lower layer.

[0138] In ophthalmologic examinations, attention may be focused on specific blood vessels among the many blood vessels in the fundus. FIG. 7 shows a non-limiting example of the configuration of an ophthalmologic apparatus 1 useful in such cases. The ophthalmologic apparatus 1 of this example includes an image acquisition unit 1000, a blood vessel distribution image generation unit 1100, a processed image generation unit 1200, a display control unit 1300, and a display device 1400, as well as an attention area identification unit 1150. At least one element of the image acquisition unit 1000, the blood vessel distribution image generation unit 1100, the processed image generation unit 1200, the display control unit 1300, and the display device 1400 in FIG. 7 may have the same function as the corresponding element in any of the ophthalmologic apparatuses 1 shown in FIGS. 4 to 6 .

[0139] The attention area specifying unit 1150 is configured to specify an image area corresponding to a specific blood vessel in the fundus Ef of the subject's eye E from the blood vessel distribution image generated by the blood vessel distribution image generating unit 1100. This specific blood vessel is, for example, a blood vessel that is the subject of observation in medical examination. This blood vessel is called a blood vessel of interest. Furthermore, the image area corresponding to the blood vessel of interest is called a attention area.

[0140] The number of blood vessels of interest may be any number, and may be one or more. The type of blood vessels of interest may also be any number. Non-limiting examples of blood vessels of interest include blood vessels with lesions or blood vessels associated with lesions, such as disrupted blood vessels, shed blood vessels, blood leaking blood vessels, blocked blood vessels, blood vessels passing through an ischemic portion of the fundus, blood vessels with aneurysms, blood vessels with abnormal shapes, blood vessels with inflammation, neovascularization, and blood vessels connected to neovascularization. Other examples include blood vessels that are the target of an examination such as fundus blood flow measurement, blood vessels that are the target of treatment such as laser photocoagulation, blood vessels of a specific size, and blood vessels associated with a specific region (e.g., the optic disc, the macula, etc.).

[0141] The attention region identification unit 1150 is realized by cooperation between hardware including a circuit and attention region identification software. In some non-limiting examples, the attention region identification unit 1150 executes at least some of the steps of the attention region identification process using a pre-created machine learning model. According to the configuration shown in Figures 1 to 3, at least some of the functions of the attention region identification unit 1150 are realized by the data processing unit 230.

[0142] In some non-limiting examples, the user specifies a blood vessel of interest using the operation unit 242. For example, the user specifies a blood vessel of interest for any of the displayed images, including the first fundus image, the blood vessel distribution image based on the first fundus image, the second fundus image, the processed image based on the second fundus image, and another image. The region-of-interest identifying unit 1150 identifies a region of interest in the blood vessel distribution image that corresponds to the specified blood vessel of interest. When a blood vessel of interest is specified for an image other than the first fundus image and the blood vessel distribution image, the region-of-interest identifying unit 1150 identifies a region of interest in the blood vessel distribution image that corresponds to the blood vessel of interest in the other image by performing registration between the other image and the blood vessel distribution image (or the first fundus image).

[0143] In some other non-limiting examples, a blood vessel of interest is determined based on medical information of the subject (e.g., electronic medical records, medical images, reports, etc.). This blood vessel of interest determination process may be performed by the ophthalmic apparatus 1 (e.g., the region of interest specifying unit 1150, etc.) or an external device (e.g., an electronic medical record system, a computer connected to the ophthalmic apparatus 1, etc.). The region of interest specifying unit 1150 specifies a region of interest corresponding to the determined blood vessel of interest from the blood vessel distribution image.

[0144] Any method may be used to identify a region of interest from the vascular distribution image, such as a method using a machine learning model, a method not using a machine learning model, or a method that at least partially combines these methods. By using such a known method, a desired region of interest (e.g., an image of a blood vessel having a lesion, an image of a blood vessel associated with a lesion, an image of a blood vessel to be inspected, an image of a blood vessel to be treated, an image of a blood vessel of a specific size, an image of a blood vessel associated with a specific location, etc.) can be detected from the vascular distribution image. The region of interest identification unit 1150 has a configuration corresponding to the type of region of interest to be detected. For example, the region of interest identification unit 1150 may include a machine learning model trained to detect a specific lesion and / or software that causes a processor to operate according to an algorithm including a series of steps for detecting a specific lesion.

[0145] The operation of the ophthalmic apparatus 1 according to the embodiment will be described. In the present disclosure, several non-limiting operation examples will be described. Any feature according to the present disclosure may be at least partially combined with any of the operation examples. Furthermore, two or more operation examples may be at least partially combined. The ophthalmic apparatus 1 may be configured to perform some of the steps in any of the operation examples.

[0146] Fig. 8 is a flowchart showing a first operation example. The first operation example can be performed, for example, by the ophthalmologic apparatus 1 having the configuration shown in Fig. 4. The first operation example includes steps S1 to S4. In describing the first operation example, Figs. 9A to 9C are referenced.

[0147] In step S1, the ophthalmologic apparatus 1 acquires a first fundus image and a second fundus image of the fundus oculi Ef by the image acquisition unit 1000. For example, the ophthalmologic apparatus 1 applies OCT angiography to the fundus oculi Ef to generate an OCT angiography image (first fundus image), and applies fundus photography to the fundus oculi Ef to generate a fundus camera image (second fundus image).

[0148] In step S2, the ophthalmologic apparatus 1 generates a blood vessel distribution image from the first fundus image acquired in step S1 using the blood vessel distribution image generating unit 1100. Fig. 9A shows an exemplary blood vessel distribution image 301 generated in step S2.

[0149] In step S3, the ophthalmologic apparatus 1 applies a predetermined processing to the second fundus image acquired in step S1 to generate a processed image using the processed image generation unit 1200. Fig. 9B shows an exemplary processed image 302 generated in step S3.

[0150] Steps S2 and S3 may be executed in reverse order, or steps S2 and S3 may be executed in parallel.

[0151] In step S4, the ophthalmologic apparatus 1 causes the display control unit 1300 to display a composite image of the vascular distribution image generated in step S2 and the processed image generated in step S3 on the display device 1400. Fig. 9C shows an exemplary composite image 303 displayed in step S4.

[0152] The blood vessel distribution image 301 in Fig. 9A is an image based on an OCT angiography image, and the processed image 302 in Fig. 9B is an processed image based on a fundus camera image. Therefore, the blood vessels of the fundus Ef are depicted more clearly in the blood vessel distribution image 301 in Fig. 9A than in the processed image 302 in Fig. 9B. Conversely, parts other than the blood vessels are depicted more clearly in the processed image 302 in Fig. 9B than in the blood vessel distribution image 301 in Fig. 9A.

[0153] The composite image 303 in Fig. 9C, which is a composite image of the blood vessel distribution image 301 in Fig. 9A and the processed image 302 in Fig. 9B, represents a blood vessel image derived from the blood vessel distribution image 301 in Fig. 9A and a fundus image derived from the processed image 302 in Fig. 9B (excluding at least the blood vessel image derived from the blood vessel distribution image 301). Therefore, this operation example can provide an integrated and detailed visualization of the structure of the fundus oculi Ef and the state of the blood vessels.

[0154] Fig. 10 is a flowchart showing a second operation example. The second operation example can be performed, for example, by the ophthalmologic apparatus 1 having the configuration shown in Fig. 5. The second operation example includes steps S11 to S16. In describing the second operation example, Figs. 11A to 11D are referred to.

[0155] In step S11, the ophthalmologic apparatus 1 acquires a first fundus image and a second fundus image of the fundus oculi Ef by the image acquisition unit 1000.

[0156] In step S12, the ophthalmologic apparatus 1 causes the blood vessel distribution image generating unit 1110 (intermediate blood vessel distribution image generating unit) to generate an intermediate blood vessel distribution image from the first fundus image acquired in step S11.

[0157] In step S13, the ophthalmologic apparatus 1 generates an arterial distribution image from the intermediate blood vessel distribution image acquired in step S12 using the blood vessel distribution image generation unit 1110 (blood vessel classification unit 1111 and arterial distribution image generation unit 1112). Fig. 11A shows an exemplary arterial distribution image 311 generated in step S13.

[0158] In step S14, the ophthalmologic apparatus 1 generates a vein distribution image from the intermediate vascular distribution image acquired in step S12 using the vascular distribution image generation unit 1110 (the vascular classification unit 1111 and the vein distribution image generation unit 1113). Fig. 11B shows an exemplary vein distribution image 312 generated in step S14.

[0159] Steps S13 and S14 may be executed in the reverse order, or steps S13 and S14 may be executed in parallel.

[0160] In step S15, the ophthalmologic apparatus 1 generates a processed image by applying a predetermined processing process to the second fundus image acquired in step S11 using the processed image generation unit 1200. Fig. 11C shows an exemplary processed image 313 generated in step S15.

[0161] Steps S12 to S14 and step S15 may be executed in reverse order, or steps S12 to S14 and step S15 may be executed in parallel.

[0162] In step S16, the ophthalmologic apparatus 1 causes the display control unit 1300 to display a composite image of the arterial distribution image generated in step S13, the venous distribution image generated in step S14, and the processed image generated in step S15 on the display device 1400. The display control unit 1300 can change the display mode of the arterial image in the arterial distribution image and the display mode of the venous image in the venous distribution image. This display mode is, for example, a color display. Figure 11D shows an exemplary composite image 314 displayed in step S16.

[0163] The artery distribution image 311 in Fig. 11A and the vein distribution image 312 in Fig. 11B are images based on an OCT angiography image, and the processed image 313 in Fig. 11C is a processed image based on a fundus camera image. Therefore, the arteries of the fundus Ef are depicted more clearly in the artery distribution image 311 in Fig. 11A than in the processed image 313 in Fig. 11C. Furthermore, the veins of the fundus Ef are depicted more clearly in the vein distribution image 312 in Fig. 11B than in the processed image 313 in Fig. 11C. On the other hand, parts other than blood vessels (arteries and veins) are depicted more clearly in the processed image 313 in Fig. 11C than in the artery distribution image 311 in Fig. 11A and the vein distribution image 312 in Fig. 11B.

[0164] The composite image 314 in Fig. 11D, which is a composite image of the arterial distribution image 311 in Fig. 11A, the venous distribution image 312 in Fig. 11B, and the processed image 313 in Fig. 11C, represents an arterial image derived from the arterial distribution image 311 in Fig. 11A, a venous image derived from the venous distribution image 312 in Fig. 11B, and a fundus image derived from the processed image 313 in Fig. 11C (excluding at least the arterial image derived from the arterial distribution image 311 and the venous image derived from the venous distribution image 312). Thus, this operation example can provide an integrated and detailed visualization of the structure of the fundus Ef and the state of the blood vessels.

[0165] Fig. 12 is a flowchart showing a third operation example. The third operation example can be performed, for example, by the ophthalmologic apparatus 1 having the configuration shown in Fig. 6. The third operation example includes steps S21 to S28. Fig. 13 is referenced in the description of the third operation example.

[0166] In step S21, the ophthalmologic apparatus 1 acquires a first fundus image and a second fundus image of the fundus oculi Ef by the image acquisition unit 1000.

[0167] In step S22, the ophthalmologic apparatus 1 prepares an alpha channel layer and a lower layer for overlay display using the display control unit 1310. In Fig. 13, reference numeral 320a denotes the alpha channel layer, and reference numeral 320b denotes the lower layer.

[0168] In step S23, the ophthalmologic apparatus 1 generates a blood vessel distribution image from the first fundus image acquired in step S21 using the blood vessel distribution image generating unit 1100. Reference numeral 321 in Fig. 13 denotes an exemplary blood vessel distribution image 321 generated in step S23.

[0169] In step S24, the ophthalmologic apparatus 1 applies a predetermined processing to the second fundus image acquired in step S21 to generate a processed image using the processed image generation unit 1200. Reference numeral 322 in Fig. 13 indicates an exemplary processed image 322 generated in step S24.

[0170] In step S25, the ophthalmologic apparatus 1 causes the opacity determination unit 1311 of the display control unit 1310 to determine the opacity of each pixel of the alpha channel layer prepared in step S22.

[0171] In step S26, the ophthalmologic apparatus 1 causes the display control unit 1310 to display the blood vessel distribution image generated in step S23 on the alpha channel layer for which the opacity of each pixel has been determined in step S25. Fig. 13 shows the process of displaying the blood vessel distribution image 321 on the alpha channel layer 320a.

[0172] In step S27, the ophthalmologic apparatus 1 displays the processed image generated in step S24 on the lower layer prepared in step S22 by the display control unit 1310. Fig. 13 shows the process of displaying the processed image 322 on the lower layer 320b.

[0173] In step S28, the ophthalmologic apparatus 1 causes the display control unit 1310 to superimpose the alpha channel layer on which the blood vessel distribution image is displayed in step S26 and the lower layer on which the processed image is displayed in step S27 on the display device 1400. Fig. 13 shows a composite image 323 presented by overlaying the blood vessel distribution image 321 and the processed image 322.

[0174] According to this operation example, similar to the first operation example, it is possible to provide an integrated and detailed visualization of the structure of the fundus oculi Ef and the state of blood vessels.

[0175] Fig. 14 is a flowchart showing a fourth operation example. The fourth operation example can be performed, for example, by an ophthalmologic apparatus 1 having both the configurations shown in Fig. 5 and Fig. 6. The fourth operation example includes steps S31 to S41. Fig. 15 is referenced in the description of the fourth operation example.

[0176] In step S31, the ophthalmologic apparatus 1 acquires a first fundus image and a second fundus image of the fundus oculi Ef by the image acquisition unit 1000.

[0177] In step S32, the ophthalmologic apparatus 1 prepares a first alpha channel layer, a second alpha channel layer, and a lower layer for overlay display by the display control unit 1310. In Fig. 15, reference numeral 330a denotes the first alpha channel layer, reference numeral 330b denotes the second alpha channel layer, and reference numeral 330c denotes the lower layer.

[0178] In step S33, the ophthalmologic apparatus 1 generates an intermediate blood vessel distribution image from the first fundus image acquired in step S31 using the blood vessel distribution image generation unit 1110 (intermediate blood vessel distribution image generation unit). Reference numeral 331 in Fig. 15 indicates an exemplary intermediate blood vessel distribution image generated in step S33.

[0179] In step S34, the ophthalmologic apparatus 1 generates an arterial distribution image from the intermediate blood vessel distribution image acquired in step S33 using the blood vessel distribution image generation unit 1110 (the blood vessel classification unit 1111 and the arterial distribution image generation unit 1112). Reference numeral 332 in Fig. 15 indicates an exemplary arterial distribution image generated in step S34.

[0180] In step S35, the ophthalmologic apparatus 1 generates a vein distribution image from the intermediate vascular distribution image acquired in step S33 using the vascular distribution image generation unit 1110 (the vascular classification unit 1111 and the vein distribution image generation unit 1113). Reference numeral 333 in Fig. 15 indicates an exemplary vein distribution image generated in step S35.

[0181] Steps S34 and S35 may be executed in the reverse order, or steps S34 and S35 may be executed in parallel.

[0182] In step S36, the ophthalmologic apparatus 1 generates a processed image by applying a predetermined processing process to the second fundus image acquired in step S31 using the processed image generation unit 1200. Reference numeral 334 in Fig. 15 indicates an exemplary processed image generated in step S36.

[0183] Steps S33 to S35 and step S36 may be executed in reverse order, or steps S33 to S35 and step S36 may be executed in parallel.

[0184] In step S37, the ophthalmologic device 1 determines the opacity of each pixel of the first alpha channel layer prepared in step S32, and also determines the opacity of each pixel of the second alpha channel layer, using the opacity determination unit 1311 of the display control unit 1310.

[0185] In step S38, the ophthalmologic apparatus 1 causes the display control unit 1310 to display the arterial distribution image generated in step S34 on the first alpha channel layer for which the opacity of each pixel was determined in step S37. Fig. 15 shows the process of displaying the arterial distribution image 332 on the first alpha channel layer 330a.

[0186] In step S39, the ophthalmologic apparatus 1 causes the display control unit 1310 to display the vein distribution image generated in step S35 on the second alpha channel layer, the opacity of each pixel of which has been determined in step S37. Fig. 15 shows the process of displaying the vein distribution image 333 on the second alpha channel layer 330b.

[0187] The display control unit 1310 can differentiate the display mode of the artery image in the artery distribution image displayed on the first alpha channel layer in step S38 from the display mode of the vein image in the vein distribution image displayed on the second alpha channel layer in step S39. For example, in step S38, the display control unit 1310 assigns a first color to each pixel constituting the artery image and displays the artery distribution image on the first alpha channel layer, and in step S39, assigns a second color different from the first color to each pixel constituting the vein image and displays the vein distribution image on the second alpha channel layer.

[0188] In step S40, the ophthalmologic apparatus 1 displays the processed image generated in step S36 on the lower layer prepared in step S32 by the display control unit 1310. Fig. 15 shows the process of displaying the processed image 334 on the lower layer 330c.

[0189] In step S41, the ophthalmologic apparatus 1 causes the display control unit 1310 to superimpose the first alpha channel layer on which the arterial distribution image is displayed in step S38, the second alpha channel layer on which the venous distribution image is displayed in step S39, and the lower layer on which the processed image is displayed in step S40, and displays them on the display device 1400. Fig. 15 shows a composite image 335 presented by overlaying an arterial distribution image 332, a venous distribution image 333, and a processed image 334.

[0190] According to this operation example, which is implemented by combining the second and third operation examples, it is possible to provide an integrated and detailed visualization of the structure and vascular condition of the fundus oculi Ef.

[0191] Fig. 16 is a flowchart showing a fifth operation example. The fifth operation example can be performed, for example, by the ophthalmologic apparatus 1 having the configuration shown in Fig. 7. The fifth operation example includes steps S51 to S56. In describing the fifth operation example, Figs. 17A to 17D are referred to.

[0192] In step S51, the ophthalmologic apparatus 1 acquires a first fundus image and a second fundus image of the fundus oculi Ef by the image acquisition unit 1000.

[0193] In step S52, the ophthalmologic apparatus 1 generates a blood vessel distribution image from the first fundus image acquired in step S51 using the blood vessel distribution image generating unit 1100. Fig. 17A shows an exemplary blood vessel distribution image 341 generated in step S52.

[0194] In step S53, the ophthalmologic apparatus 1 applies a predetermined processing to the second fundus image acquired in step S51 to generate a processed image using the processed image generation unit 1200. Fig. 17B shows an exemplary processed image 342 generated in step S53.

[0195] Steps S52 and S53 may be executed in the reverse order, or steps S52 and S53 may be executed in parallel.

[0196] In step S54, the ophthalmologic apparatus 1 specifies a region of interest corresponding to the blood vessel of interest in the fundus oculi Ef from the blood vessel distribution image generated in step S52 by the region of interest specifying unit 1150. The region of interest may indicate all blood vessels connected to the blood vessel of interest in the fundus oculi (for example, a main trunk such as the central retinal artery and all its branches), or may indicate a specific blood vessel (for example, a specific branch).

[0197] In step S55, the ophthalmologic apparatus 1 sets pixel values ​​of the attention region identified in step S54, for example, by using the attention region identification unit 1150 or the display control unit 1300. For example, the ophthalmologic apparatus 1 can represent the attention region in a display mode different from other parts by assigning a predetermined color to each pixel of the attention region.

[0198] If the ophthalmologic apparatus 1 is capable of distinguishing between an artery image and a vein image, the color used when the region of interest is an artery image or a part thereof can be different from the color used when the region of interest is a vein image or a part thereof, thereby enabling the user to easily recognize whether the region of interest is an artery or a vein.

[0199] 17C shows a blood vessel distribution image 343 obtained by specifying a region of interest (S54) from the blood vessel distribution image 342 generated in step S52 and setting pixel values ​​of this region of interest (S55). Reference numeral 343a indicates the region of interest.

[0200] In step S56, the ophthalmologic apparatus 1 causes the display control unit 1300 to display a composite image of the vascular distribution image in which the display mode of the region of interest has been changed in step S55 and the processed image generated in step S53 on the display device 1400. Fig. 17D shows an exemplary composite image 344 displayed in step S56. Reference numeral 344a denotes the region of interest.

[0201] According to this operation example, similar to the first to fourth operation examples, it is possible to provide an integrated and detailed visualization of the structure of the fundus oculi Ef and the state of blood vessels.

[0202] Furthermore, according to this operation example, the user can comprehensively and in detail observe the region of interest corresponding to the blood vessel of interest in the fundus Ef, along with the structure of the fundus Ef and the state of the blood vessel. This makes it possible to understand the state of the blood vessel of interest in consideration of the relationship between the structure of the fundus Ef and the blood vessel. Therefore, this operation example contributes to speeding up medical treatment and improving the quality.

[0203] In another example of operation, the image acquisition unit 1000 acquires a 3D OCT angiography image. The vascularity image generation unit 1100 of this example identifies at least one of the aforementioned slabs SCP, DCP, OR, ORCC, CC, and C from the 3D OCT angiography image and generates a 2D angiography image (front image) from each identified slab. As a non-limiting example, a case will be described in which four 2D angiography images corresponding to the SCP, DCP, OR, and CC, respectively, are generated. The display control unit 1300 of this example causes the display device 1400 to display a composite image of at least one of these four 2D angiography images and the processed image generated by the processed image generation unit 1200. This display control is realized, for example, by the display control unit 1310 (opacity determination unit 1311) preparing four alpha channel layers corresponding to the four 2D angiographic images, and overlaying and displaying at least one of the four alpha channel layers, each displaying one of the four 2D angiographic images, on a lower layer displaying a processed image. When two or more alpha channel layers are overlaid on a lower layer, that is, when two or more 2D angiographic images are overlaid on a processed image, the display color of the blood vessel images may be different for each 2D angiographic image. For example, the blood vessel images in the SCP 2D angiographic image are displayed in a first color, the blood vessel images in the DCP 2D angiographic image are displayed in a second color, the blood vessel images in the OR 2D angiographic image are displayed in a third color, and the blood vessel images in the CC 2D angiographic image are displayed in a fourth color. The allocation of display colors to each two-dimensional angiographic image may be performed by the vascular distribution image generating unit 1100 or the display control unit 1300 (1310). For example, when generating two-dimensional angiographic images from each of the SCP slab, the DCP slab, the OR slab, and the CC slab, the vascular distribution image generating unit 1100 can assign corresponding color information (RGB values) to pixel groups corresponding to blood vessel images in the two-dimensional angiographic images.Alternatively, the display control unit 1300 (1310) can assign corresponding color information (RGB values) to pixel groups corresponding to blood vessel images in each 2D angiographic image by using a layer having both an alpha channel (opacity) and a color channel (RGB values) (a layer having at least an alpha channel, therefore referred to as an alpha channel layer). Selection of the 2D angiographic image to be displayed may be performed, for example, according to a user instruction or by the display control unit 1300 (1310). This operational example not only provides an integrated and detailed visualization of the fundus structure and blood vessel condition, but also improves visibility by displaying blood vessel images in different colors for each depth region. This operational example can be combined with each of the above-mentioned operational examples.

[0204] An ophthalmologic apparatus according to a non-limiting embodiment includes an image acquisition unit, a blood vessel distribution image generation unit, a processed image generation unit, and a display control unit. The image acquisition unit is configured to acquire a first fundus image generated by applying OCT angiography to the fundus of the subject's eye, and a second fundus image generated by applying an imaging modality different from OCT angiography to the fundus. The blood vessel distribution image generation unit is configured to execute processing to generate a blood vessel distribution image from the first fundus image. The processed image generation unit is configured to execute processing to generate a processed image by applying a predetermined processing to the second fundus image. The display control unit is configured to execute processing to display a composite image based on a plurality of images including the blood vessel distribution image and the processed image on a display device.

[0205] In some non-limiting embodiments, the display control unit is configured to perform a process of displaying a vascular distribution image on an alpha channel layer whose opacity can be set, and a process of displaying a processed image on a layer below the alpha channel layer.

[0206] In some non-limiting embodiments, the display control unit is configured to perform a process for determining the opacity of the alpha channel layer based on the vascularity image.

[0207] In some non-limiting embodiments, the display controller is configured to perform a process of setting the opacity of a first group of pixels corresponding to blood vessels in the vascular distribution image to a first value, and a process of setting the opacity of a second group of pixels other than the first group of pixels to a second value smaller than the first value.

[0208] In some non-limiting embodiments, the vascular distribution image generation unit is configured to perform a process of generating at least one of an arterial distribution image and a venous distribution image as the vascular distribution image, and the display control unit is configured to perform a process of displaying the generated at least one image on an alpha channel layer and a process of displaying a processed image on a lower layer.

[0209] In some non-limiting embodiments, the vascularity image generation unit is configured to perform a process of generating both an arterial distribution image and a venous distribution image. Further, the display control unit is configured to perform a process of preparing a first alpha channel layer and a second alpha channel layer as alpha channel layers, a process of displaying an arterial distribution image in which a first color is assigned to an arterial image on the first alpha channel layer, a process of displaying a venous distribution image in which a second color different from the first color is assigned to a venous image on the second alpha channel layer, and a process of displaying a processed image on a lower layer.

[0210] In some non-limiting embodiments, the display control unit is configured to perform processing to selectively display a first alpha channel layer displaying an arterial distribution image and a second alpha channel layer displaying a venous distribution image together with a lower layer displaying a processed image.

[0211] In some non-limiting embodiments, the ophthalmologic apparatus further includes a region-of-interest specifying unit configured to specify a region of interest corresponding to a blood vessel of interest in the fundus of the subject's eye from the blood vessel distribution image. Furthermore, the display control unit is configured to execute processing to display a composite image based on a plurality of images including the blood vessel distribution image in which pixel values ​​of the region of interest differ from pixel values ​​of regions other than the region of interest and the processed image.

[0212] In some non-limiting embodiments, the second fundus image includes a 3D OCT intensity image generated by applying an OCT scan to a 3D region of the fundus of the test eye. The processed image generation unit is further configured to perform processing to generate a processed image corresponding to an area including at least a portion of the fundus area depicted in the blood vessel distribution image by applying a predetermined processing process to the 3D OCT intensity image. Additionally, the display control unit is configured to perform processing to display a composite image based on a plurality of images including the blood vessel distribution image and the processed image generated from the 3D OCT intensity image.

[0213] In some non-limiting embodiments, the second fundus image includes a two-dimensional image of the fundus of the subject's eye. The processed image generation unit is configured to perform processing to generate a processed image corresponding to an area including at least a portion of the area of ​​the fundus depicted in the blood vessel distribution image by applying a predetermined processing process to the two-dimensional image of the fundus. Additionally, the display control unit is configured to perform processing to display a composite image based on a plurality of images including the blood vessel distribution image and the processed image generated from the two-dimensional image.

[0214] Other Embodiments Those skilled in the art will appreciate that the present disclosure provides embodiments in categories other than ophthalmic devices.

[0215] The present disclosure provides an embodiment of a method for controlling an ophthalmic device. A non-limiting embodiment is a method for controlling an ophthalmic device including a processor and a storage device. The method causes the processor to execute a storing step, a blood vessel distribution image generating step, a processed image generating step, and a display control step. The storing step causes the processor to store in the storage device a first fundus image generated by applying OCT angiography to the fundus of the subject's eye and a second fundus image generated by applying an imaging modality different from OCT angiography to the fundus. The blood vessel distribution image generating step generates a blood vessel distribution image from the first fundus image. The processed image generating step generates the processed image by applying a predetermined processing process to the second fundus image. The display control step causes a display device to display a composite image based on a plurality of images including the blood vessel distribution image and the processed image. Any feature related to the present disclosure may be combined with this method.

[0216] The present disclosure provides an embodiment of a program. A non-limiting embodiment is a program that causes a computer to execute each step of the above method. Any feature of the present disclosure can be combined with this program.

[0217] The present disclosure provides an embodiment of a recording medium. A non-limiting embodiment is a computer-readable non-transitory recording medium on which the above-described program is recorded. This recording medium may take any form. For example, the recording medium may be any of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory. Any aspect of the present disclosure may be combined with this recording medium.

[0218] The present disclosure is merely an example of how to implement the present invention, and those who intend to implement the present invention can make any modifications (omissions, substitutions, additions, etc.) within the scope of the gist of the present invention.

[0219] REFERENCE SIGNS LIST 1 Ophthalmic apparatus 1000 Image acquisition unit 1100, 1110 Blood vessel distribution image generation unit 1111 Blood vessel classification unit 1112 Artery distribution image generation unit 1113 Venous distribution image generation unit 1150 Attention area identification unit 1200 Processed image generation unit 1300, 1310 Display control unit 1311 Opacity determination unit 1400 Display device

Claims

1. An ophthalmologic device comprising: an image acquisition unit that acquires a first fundus image generated by applying optical coherence tomography angiography to the fundus of a subject's eye, and a second fundus image generated by applying an imaging modality different from optical coherence tomography angiography to the fundus; a blood vessel distribution image generation unit that generates a blood vessel distribution image from the first fundus image; a processed image generation unit that generates a processed image by applying a predetermined processing process to the second fundus image; and a display control unit that displays a composite image based on multiple images including the blood vessel distribution image and the processed image on a display device.

2. The ophthalmologic device according to claim 1, wherein the display control unit displays the blood vessel distribution image on an alpha channel layer whose opacity can be set, and displays the processed image on a layer below the alpha channel layer.

3. The ophthalmologic apparatus according to claim 2, wherein the display control unit determines the opacity of the alpha channel layer based on the blood vessel distribution image.

4. The ophthalmologic device of claim 3, wherein the display control unit sets the opacity of a first group of pixels corresponding to blood vessels in the blood vessel distribution image to a first value, and sets the opacity of a second group of pixels other than the first group of pixels to a second value smaller than the first value.

5. An ophthalmologic device according to any one of claims 2 to 4, wherein the vascular distribution image generation unit generates at least one of an arterial distribution image and a venous distribution image as the vascular distribution image, and the display control unit displays the at least one image on the alpha channel layer, and displays the processed image on the lower layer.

6. An ophthalmologic device according to claim 5, wherein the blood vessel distribution image generation unit generates both the arterial distribution image and the venous distribution image, and the display control unit prepares a first alpha channel layer and a second alpha channel layer as the alpha channel layers, displays the arterial distribution image in which a first color is assigned to an arterial image on the first alpha channel layer, displays the venous distribution image in which a second color different from the first color is assigned to a vein image on the second alpha channel layer, and displays the processed image on the lower layer.

7. The ophthalmologic device of claim 6, wherein the display control unit selectively displays the first alpha channel layer on which the arterial distribution image is displayed and the second alpha channel layer on which the venous distribution image is displayed together with the lower layer on which the processed image is displayed.

8. An ophthalmologic device according to any one of claims 1 to 7, further comprising an attention area specifying unit that specifies an attention area corresponding to an attention blood vessel of the fundus from the blood vessel distribution image, and the display control unit displays the composite image based on the multiple images including the blood vessel distribution image in which pixel values ​​of the attention area differ from pixel values ​​of areas other than the attention area and the processed image.

9. An ophthalmologic device according to any one of claims 1 to 8, wherein the second fundus image includes a three-dimensional optical coherence tomography intensity image generated by applying an optical coherence tomography scan to a three-dimensional region of the fundus, the processed image generation unit applies the predetermined processing to the three-dimensional optical coherence tomography intensity image to generate the processed image corresponding to an area including at least a portion of the area of ​​the fundus depicted in the blood vessel distribution image, and the display control unit displays the composite image based on the multiple images including the blood vessel distribution image and the processed image generated from the three-dimensional optical coherence tomography intensity image.

10. An ophthalmologic device according to any one of claims 1 to 9, wherein the second fundus image includes a two-dimensional image of the fundus, the processed image generation unit applies the predetermined processing to the two-dimensional image to generate the processed image corresponding to an area including at least a portion of the area of ​​the fundus depicted in the blood vessel distribution image, and the display control unit displays the composite image based on the multiple images including the blood vessel distribution image and the processed image generated from the two-dimensional image.

11. A method for controlling an ophthalmologic device having a processor and a storage device, the method causing the processor to execute the following steps: a storage step for storing in the storage device a first fundus image generated by applying optical coherence tomography angiography to the fundus of a subject's eye, and a second fundus image generated by applying an imaging modality different from optical coherence tomography angiography to the fundus; a vascular distribution image generation step for generating a vascular distribution image from the first fundus image; a processed image generation step for generating a processed image by applying a predetermined processing process to the second fundus image; and a display control step for displaying on a display device a composite image based on multiple images including the vascular distribution image and the processed image.

12. A program for causing a computer to execute each step of the method of claim 11.

13. A computer-readable non-transitory recording medium on which the program of claim 12 is recorded.

Images