IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, AND PROGRAM

The image processing method aligns three-dimensional MRI eyeball images with two-dimensional UWF fundus images by identifying reference points, enhancing diagnostic accuracy and visualization for ophthalmologists.

JP7585800B2Active Publication Date: 2024-11-19NIKON CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2021005188
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-01-15
Publication Date
2024-11-19
Estimated Expiration
2041-01-15

AI Technical Summary

Technical Problem

Existing methods for constructing eyeball models from MRI images lack efficient techniques to accurately align and superimpose two-dimensional and three-dimensional eye images for enhanced diagnostic accuracy.

Method used

An image processing method that involves acquiring both three-dimensional MRI eyeball images and two-dimensional UWF fundus images, identifying reference points in both images, and aligning them to generate a superimposed image for improved diagnostic visualization.

Benefits of technology

Enables precise alignment and superimposition of two-dimensional and three-dimensional eye images, facilitating more accurate diagnosis and disease evaluation by ophthalmologists.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585800000001
    Figure 0007585800000001
  • Figure 0007585800000002
    Figure 0007585800000002
  • Figure 0007585800000003
    Figure 0007585800000003
Patent Text Reader

Abstract

SOLUTION: To provide an image processing method which comprises the steps of: acquiring a three-dimensional image of a subject eye and a two-dimensional image of the subject eye; specifying a first reference point in the three-dimensional image; specifying a second reference point in the two-dimensional image; and making the first reference point in the three-dimensional image correspond to the second reference point in the two-dimensional image and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image.EFFECT: Since a two-dimensional fundus image is superimposed, it is helpful to grasp the position of retinal detachment when viewed from the outside of an eyeball.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The technology disclosed herein relates to an image processing method, an image processing device, and a program. [Background technology]

[0002] It is known to construct an eyeball model from MRI (Magnetic Resonance Imaging) images (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2013-031536 Summary of the Invention

[0004] An image processing method of a first aspect of the technology disclosed herein includes the steps of acquiring a three-dimensional image of a test eye and a two-dimensional image of the test eye, identifying a first reference point in the three-dimensional image, identifying a second reference point in the two-dimensional image, and matching the first reference point in the three-dimensional image with the second reference point in the two-dimensional image, and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image.

[0005] An image processing device of a second aspect of the technology disclosed herein includes a memory and a processor connected to the memory, and the processor executes an image processing method including the steps of acquiring a three-dimensional image of a test eye and a two-dimensional image of the test eye, identifying a first reference point in the three-dimensional image, identifying a second reference point in the two-dimensional image, and matching the first reference point in the three-dimensional image with the second reference point in the two-dimensional image, and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image.

[0006] A program of a third aspect of the technology disclosed herein causes a computer to execute an image processing method including the steps of acquiring a three-dimensional image of a test eye and a two-dimensional image of the test eye, identifying a first reference point in the three-dimensional image, identifying a second reference point in the two-dimensional image, and matching the first reference point in the three-dimensional image with the second reference point in the two-dimensional image, and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image.

[0007] An image processing method according to a fourth aspect of the technology disclosed herein includes the steps of: acquiring a first image, which is a three-dimensional image, of a test eye captured with a first modality; acquiring a second image of the test eye captured with a second modality different from the first modality; identifying a first structural feature in the first image; identifying a second structural feature in the second image that corresponds to the first structural feature; and generating an image in which at least a portion of the second image is superimposed on at least a portion of the first image based on the first structural feature and the second structural feature. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an ophthalmologic system 100 according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing the configuration of an electrical system of the UWF ophthalmic device 110. FIG. [Diagram 3] 2 is a block diagram showing the configuration of an electrical system of a management server 140. FIG. [Figure 4] 2 is a block diagram showing the configuration of an electrical system of an image viewer 150. FIG. [Diagram 5] 2 is a functional block diagram of a CPU 162 of the management server 140. FIG. [Figure 6] 13 is a flowchart of an image processing program executed by the management server 140. [Figure 7] 1 is a diagram showing the appearance of an optic disc 352 and a macula 356 detected in a UWF two-dimensional fundus image 350. FIG. [Figure 8] 1 is a diagram showing the appearance of an optic disc 352, a macula 356, and a pupil 354 detected in a UWF two-dimensional fundus image 350. FIG. [Figure 9] FIG. 3 is a diagram showing the state in which an optic disc 373 is detected in an MRI three-dimensional eye image 370. [Figure 10] FIG. 3 is a diagram showing the state in which an optic disc 373 is detected in an MRI three-dimensional eye image 370. [Figure 11A] FIG. 3 shows how the corneal center 376 is detected in an MRI three-dimensional eye image 370. [Figure 11B] FIG. 3 shows how the corneal center 376 and pupil 378 are detected in an MRI three-dimensional eye image 370. [Figure 12] 13 is a diagram showing a state in which no pupil is present in a UWF two-dimensional fundus image 350 obtained by photographing a fundus 351. FIG. [Figure 13A] FIG. 13 is a diagram showing how a UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image. [Figure 13B] 13 is a diagram for explaining how a UWF two-dimensional fundus image 350 is superimposed. FIG. [Figure 14] A diagram showing a display screen 400. [Figure 15] 13 is a diagram showing how an image obtained by superimposing a UWF two-dimensional fundus image on an MRI three-dimensional eye image and an OCT tomographic image of the entire test eye including an anterior segment OCT image and a posterior segment OCT image are superimposed. FIG. [Figure 16] 13 is a diagram showing how images 382 and 384 of a partial region of a fundus image are superimposed on an MRI three-dimensional eyeball image 370. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the technology of the present disclosure will be described in detail with reference to the drawings.

[0010] The configuration of an ophthalmic system 100 according to an embodiment of the technique of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the ophthalmic system 100 according to the present embodiment. As shown in Fig. 1, the ophthalmic system 100 includes a UWF (Ultra Wide Field) ophthalmic device 110, an MRI image acquisition device 120, a network 130, a management server device (hereinafter referred to as "management server") 140, and an image display device (hereinafter referred to as "image viewer") 150.

[0011] The UWF ophthalmic device 110, the MRI image acquisition device 120, the management server 140, and the image viewer 150 are connected to each other via a network 130. The network 130 is any network such as a LAN, a WAN, the Internet, or a wide area Ethernet network. For example, when the ophthalmic system 100 is constructed in one hospital, a LAN can be adopted as the network 130.

[0012] The MRI image acquisition device 120 is a device that places the subject's eye in a strong static magnetic field (by a superconducting, normal conducting, or permanent magnet), applies various pulse sequences of electromagnetic waves to the subject's eye, and then processes biochemical information of hydrogen nuclei as signals by applying the nuclear magnetic resonance phenomenon to generate an image (Non-Patent Document 1). Image components are relaxation time (T, T2) and hydrogen nucleus density. By selecting an appropriate pulse sequence, an image (MRI three-dimensional eye image) in which the relaxation times T1, T2 and hydrogen nucleus density are emphasized can be obtained. As the pulse sequence, a saturation recovery method (SR), an inversion recovery method (IR), a spin echo method (SE), a STIR method, or the like can be used. The MRI three-dimensional eye image obtained by the MRI image acquisition device 120 is a surface image (spherical image) obtained by modeling a three-dimensional shape. The MRI three-dimensional eye image has the respective structural parts of the eyeball and the optic nerve. (Non-Patent Document 1) Yuzo Nakao, "Magnetic Resonance Imaging (MRI) Examination in Ophthalmology", page 1, [online], 5th Lecture of the Japanese Association of Orthoptists, Internet<https: / / www.jstage.jst.go.jp / article / jorthoptic1977 / 18 / 0 / 18_0_1 / _pdf / -char / ja>

[0013] The management server 140 receives various UWF fundus images and OCT (Optical Coherence Tomography) images, which will be described later, obtained by photographing the fundus by the UWF ophthalmic device 110, and MRI three-dimensional eye images of the subject's eye acquired by the MRI image acquisition device 120, in correspondence with the patient ID, via the network 130, and stores them in the memory 164 (see also FIG. 3). Thus, the memory 164 of the management server 140 stores various UWF fundus images, OCT images, and MRI three-dimensional eye images in correspondence with the patient ID of each patient. The management server 140 also transmits various images to the image viewer 150 via the network 130.

[0014] The image viewer 150 is a touch panel, a display, etc., and has a communication function. The image viewer 150 displays a fundus image acquired by the management server 140.

[0015] In addition, other ophthalmic equipment (examination equipment for visual field measurement, intraocular pressure measurement, etc.) and a diagnostic support device that performs image analysis using artificial intelligence may be connected to the UWF ophthalmic device 110, the MRI image acquisition device 120, the management server 140, and the image viewer 150 via the network 130.

[0016] Next, the configuration of the UWF ophthalmic device 110 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the UWF ophthalmic device 110 according to this embodiment. As shown in Fig. 2, the UWF ophthalmic device 110 includes an imaging device 14 and a control device 16.

[0017] When the UWF ophthalmic device 110 is placed on a horizontal plane, the horizontal direction is defined as the "X direction", the vertical direction to the horizontal plane is defined as the "Y direction", and the optical axis direction of the photographing optical system 19 is defined as the "Z direction". The device is placed with respect to the subject's eye 12 so that the center of the pupil 27 of the subject's eye is located on the optical axis in the X direction. The X direction, Y direction, and Z direction are perpendicular to each other.

[0018] The control device 16 includes a computer having a CPU (Central Processing Unit) 16A, a RAM (Random Access Memory) 16B, a ROM (Read-Only memory) 16C, an input / output (I / O) port 16D, an input / display device 16E, and a communication interface (I / F) 16F. Each component of the control device 16 is connected to each other via a bus so as to be able to communicate with each other.

[0019] The CPU 16A is a central processing unit, and executes various programs and controls each part. That is, the CPU 16A reads out the programs from the ROM 16C and executes the programs using the RAM 16B as a working area. The CPU 16A controls each component and performs various arithmetic processing according to the programs stored in the ROM 16C. In this embodiment, the ROM 16C stores a scan program for executing a scan process.

[0020] The RAM 16B temporarily stores programs or data as a working area. The ROM 16C stores various programs and various data. The control device 16 may further include a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). In this case, the storage stores various programs including an operating system and various data.

[0021] The input / display device 16E is connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphical user interface (GUI) that displays an image of the subject's eye 12 and receives various instructions from a user. A touch panel or a display can be used as the GUI. The control device 16 also includes an image processing device 17 connected to the I / O port 16D.

[0022] The control device 16 connects to the network 130 via a communication interface 16F. The communication interface 16F is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0023] 2, the control device 16 of the UWF ophthalmic device 110 includes an input / display device 16E, but the technology of the present disclosure is not limited to this. For example, the control device 16 of the UWF ophthalmic device 110 may not include the input / display device 16E, but may include a separate input / display device that is physically independent from the UWF ophthalmic device 110. In this case, the display device includes an image processing processor unit that operates under the control of the CPU 16A of the control device 16. The image processing processor unit may display an acquired image, etc., based on an image signal instructed to be output by the CPU 16A.

[0024] The image processing device 17 generates an image of the subject's eye 12 based on the data obtained by the photographing device 14. Note that the image processing device 17 may be omitted, and the CPU 16A may generate an image of the subject's eye 12 based on the data obtained by the photographing device 14.

[0025] The imaging device 14 captures an image of the subject's eye 12. The imaging device 14 operates under the control of the control device 16. For ease of explanation, a scanning laser ophthalmoscope (SLO) is referred to as "SLO." The imaging device 14 includes an imaging optical system 19, an SLO unit 18, and an OCT unit 20. The SLO unit 18 captures an image of the fundus 12A of the subject's eye 12. The OCT unit 20 captures a tomographic image of the subject's eye 12.

[0026] Hereinafter, a front view image of the retina created based on the SLO data acquired by the SLO unit 18 will be referred to as an SLO image, and a tomographic image or a front image (en-face image) of the retina created based on the OCT data acquired by the OCT unit 20 will be referred to as an OCT image. The SLO image may also be referred to as a two-dimensional fundus image. The OCT image may also be referred to as a fundus tomographic image or an anterior segment tomographic image depending on the imaging site of the subject's eye 12.

[0027] The photographing optical system 19 itself is moved in the X, Y, and Z directions by a photographing optical system driving unit (not shown) under the control of the CPU 16A. Alignment between the photographing device 14 and the subject's eye 12 may be performed, for example, by moving not only the photographing device 14 but also the entire UWF ophthalmic apparatus 110 in the X, Y, and Z directions.

[0028] The photographing optical system 19 includes an optical path combining element 21, a first scanner 22, a second scanner 24, and an objective optical system 26. The optical path combining element 21 is a half mirror or a beam splitter, and the first scanner 22 and the second scanner 24 are optical scanners.

[0029] The objective optical system 26 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric optical system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph the retina not only in the center of the fundus but also in the peripheral part of the fundus.

[0030] When a system including an elliptical mirror is used, the system using the elliptical mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may be used. The disclosures of International Publication WO2016 / 103484 and International Publication WO2016 / 103489 are each incorporated herein by reference in their entirety.

[0031] The photographing optical system 19 realizes observation of the fundus in a wide field of view (FOV). The FOV indicates a range that can be photographed by the photographing device 14. The FOV can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal irradiation angle and an external irradiation angle. The external irradiation angle is an irradiation angle of a light beam irradiated from the UWF ophthalmic device 110 to the subject's eye, the irradiation angle being defined based on the pupil. The internal irradiation angle is an irradiation angle of a light beam irradiated to the fundus 12A, the irradiation angle being defined based on the eyeball center O. The external irradiation angle and the internal irradiation angle are in a corresponding relationship. For example, when the external irradiation angle is 120 degrees, the internal irradiation angle corresponds to about 160 degrees. In this embodiment, the internal irradiation angle is 200 degrees.

[0032] Here, an SLO fundus image captured at an internal illumination angle of 160 degrees or more is referred to as a UWF-SLO fundus image.

[0033] First, the SLO image acquisition will be described.

[0034] The SLO unit 18 includes a plurality of light sources. As shown in FIG. 2, the SLO unit 18 includes a light source 40 of B light (blue light), a light source 42 of G light (green light), a light source 44 of R light (red light), and a light source 46 of IR light (infrared light (e.g., near-infrared light)), and optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light emitted from each of the light sources 40, 42, 44, and 46 and guide it to one optical path. The optical systems 48, 50, and 56 are mirrors, and the optical systems 52 and 54 are beam splitters. The B light is reflected by the optical system 48, passes through the optical system 50, and is reflected by the optical system 54, the G light is reflected by the optical systems 50 and 54, the R light is transmitted through the optical systems 52 and 54, and the IR light is reflected by the optical systems 52 and 56, and is guided to one optical path, respectively.

[0035] The light sources 40, 42, 44, and 46 may be LED light sources or laser light sources. An example using a laser light source will be described below. The optical systems 48 and 56 may be total reflection mirrors. The optical systems 50, 52, and 54 may be dichroic mirrors, half mirrors, etc.

[0036] The SLO unit 18 is configured to be able to switch between various light emission modes, such as a light emission mode in which G light, R light, B light, and IR light are emitted individually, and a light emission mode in which all or some of them are emitted simultaneously. In the example shown in FIG. 2, four light sources are provided: a light source 40 of B light (blue light), a light source 42 of G light, a light source 44 of R light, and a light source 46 of IR light, but the technology of the present disclosure is not limited to this. For example, the SLO unit 18 may further include a light source of white light. In this case, in addition to the various light emission modes described above, a light emission mode in which only white light is emitted may be set.

[0037] The laser light incident on the photographing optical system 19 from the SLO unit 18 is scanned in the X direction and the Y direction by the first scanner 22 and the second scanner 24 of the photographing optical system 19. The scanning light passes through the pupil 27 and is irradiated onto the posterior segment (e.g., the fundus 12A) of the subject's eye 12. The reflected light reflected by the fundus 12A is incident on the SLO unit 18 via the photographing optical system 19.

[0038] The light reflected by the fundus 12A is detected by light detection elements 70, 72, 74, and 76 provided in the SLO unit 18. In this embodiment, the SLO unit 18 includes a B light detection element 70, a G light detection element 72, an R light detection element 74, and an IR light detection element 76 corresponding to a plurality of light sources, i.e., the B light source 40, the G light source 42, the R light source 44, and the IR light source 46. The B light detection element 70 detects the B light reflected by the beam splitter 64. The G light detection element 72 detects the G light that passes through the beam splitter 64 and is reflected by the beam splitter 58. The R light detection element 74 detects the R light that passes through the beam splitters 64 and 58 and is reflected by the beam splitter 60. The IR light detection element 76 detects the IR light that passes through the beam splitters 64, 58, and 60 and is reflected by the beam splitter 62. The light detection elements 70, 72, 74, and 76 include, for example, APDs (avalanche photodiodes).

[0039] Under the control of the CPU 16A, the image processing device 17 generates an SLO image corresponding to each color using signals detected by the B light detection element 70, the G light detection element 72, the R light detection element 74, and the IR light detection element 76. The SLO images corresponding to each color include a B-SLO image generated using a signal detected by the B light detection element 70, a G-SLO image generated using a signal detected by the G light detection element 72, an R-SLO image generated using a signal detected by the R light detection element 74, and an IR-SLO image generated using a signal detected by the IR light detection element 76. In addition, in the case of a light emission mode in which the B light source 40, the G light source 42, and the R light source 44 emit light simultaneously, an RGB-SLO image may be synthesized from the B-SLO image, the G-SLO image, and the R-SLO image generated using the signals detected by the R light detection element 74, the G light detection element 72, and the B light detection element 70, respectively. In addition, in the case of a light emission mode in which the G light source 42 and the R light source 44 emit light simultaneously, an RG-SLO image may be synthesized from a G-SLO image and an R-SLO image generated using the respective signals detected by the R light detection element 74 and the G light detection element 72.

[0040] The beam splitters 58, 60, 62, and 64 may be dichroic mirrors, half mirrors, or the like.

[0041] Next, the OCT image acquisition will be described. The OCT system is a three-dimensional image acquisition device realized by the control device 16, the OCT unit 20, and the photographing optical system 19 shown in Fig. 2. The OCT unit 20 includes a light source 20A, a sensor (detection element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0042] The light source 20A generates light for optical coherence tomography. For example, a super luminescent diode (SLD) can be used as the light source 20A. The light source 20A generates low-coherence light of a broadband light source having a wide spectral width. The light emitted from the light source 20A is split by the first optical coupler 20C. One of the split lights is collimated by the collimating lens 20E as a measurement light, and then enters the photographing optical system 19. The measurement light is scanned in the X direction and the Y direction by the first scanner 22 and the second scanner 24 of the photographing optical system 19. The scanning light is irradiated to the anterior segment of the subject's eye or the posterior segment via the pupil 27. The measurement light reflected by the anterior segment or the posterior segment enters the OCT unit 20 via the photographing optical system 19, and enters the second optical coupler 20F via the collimating lens 20E and the first optical coupler 20C. In this embodiment, SD-OCT using an SLD as the light source 20A is exemplified, but the present invention is not limited to this, and SS-OCT using a wavelength swept light source instead of an SLD may be adopted.

[0043] The other light beam emitted from the light source 20A and branched by the first optical coupler 20C is incident as reference light on the reference optical system 20D, and passes through the reference optical system 20D to be incident on the second optical coupler 20F.

[0044] The measurement light (return light) reflected and scattered by the subject's eye 12 and the reference light are combined by the second optical coupler 20F to generate interference light. The interference light is detected by the sensor 20B. The image processing device 17 generates a tomographic image of the subject's eye 12 based on the detection signal (OCT data) from the sensor 20B.

[0045] The OCT system generates a tomographic image of the anterior or posterior segment of the subject's eye 12 .

[0046] The anterior segment of the subject's eye 12 includes, as an anterior segment, the cornea, the iris, the angle, the crystalline lens, the ciliary body, and a part of the vitreous body. The posterior segment of the subject's eye 12 includes, as a posterior segment, the remaining part of the vitreous body, the retina, the choroid, and the sclera. The vitreous body belonging to the anterior segment is the part of the vitreous body on the cornea side with respect to the XY plane passing through the point of the crystalline lens closest to the center of the eyeball as a boundary, and the vitreous body belonging to the posterior segment is the part of the vitreous body other than the vitreous body belonging to the anterior segment.

[0047] When the anterior segment of the subject's eye 12 is the imaging target site, the OCT system generates, for example, a tomographic image of the cornea. When the posterior segment of the subject's eye 12 is the imaging target site, the OCT system generates, for example, a tomographic image of the retina. Hereinafter, a tomographic image of the cornea may be referred to as an anterior segment OCT image, and a tomographic image of the retina may be referred to as a posterior segment OCT image. An OCT image including an anterior segment OCT image and a posterior segment OCT image may be referred to as a Whole-Eye OCT image (OCT image of the entire subject's eye).

[0048] The light emitted from the OCT unit 20 and passing through the light path combining element 21 is scanned in the X-axis direction by the first scanner 22. The second scanner 24 scans the light emitted from the OCT unit 20 in the Y-axis direction. The first scanner 22 and the second scanner 24 may be optical elements capable of deflecting a light beam, such as a polygon mirror or a galvanometer mirror. Alternatively, a combination of these may be used. The first scanner 22 and the second scanner 24 may be configured as one optical scanner.

[0049] The objective optical system 26 is an optical system that guides the light guided by the first scanner 22 and the second scanner 24 to the subject's eye 12. The objective optical system 26 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric optical system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph the retina not only in the center of the fundus but also in the peripheral part of the fundus.

[0050] In addition, there is provided a focus adjustment mechanism 28 that adjusts the focus of the measurement light from the OCT unit 20. The focus adjustment mechanism 28 is one of the optical adjustment mechanisms described later.

[0051] Light emitted from the light source 20A is branched by the first optical coupler 20C. One of the branched lights is collimated by the collimating lens 20E as measurement light, and then enters the photographing optical system 19. The measurement light is scanned in the X and Y directions by the first scanner 22 and the second scanner 24. The scanning light passes through the objective optical system 26 and the pupil 27 and is irradiated onto the fundus. The measurement light reflected by the fundus is entered into the OCT unit 20 via the objective optical system 26, the second scanner 24, and the first scanner 22, and then enters the second optical coupler 20F via the collimating lens 20E and the first optical coupler 20C.

[0052] The other light emitted from the light source 20A and branched by the first optical coupler 20C is incident as a reference light to the reference optical system 20D, and passes through the reference optical system 20D to the second optical coupler 20F. These lights incident on the second optical coupler 20F, i.e., the measurement light reflected from the fundus and the reference light, interfere with each other in the second optical coupler 20F to generate interference light. The interference light is received by the sensor 20B. The image processing device 17, which operates under the control of the image processing unit 182 (see also FIG. 5), generates OCT images such as tomographic images and en-face images based on the OCT data detected by the sensor 20B.

[0053] The configuration of the management server 140 will be described with reference to Fig. 3. As shown in Fig. 3, the management server 140 includes a control unit 160 and a display / operation unit 170. The control unit 160 includes a computer including a CPU 162, a memory 164 which is a storage device, and a communication interface (I / F) 166. The memory 164 stores an image processing program shown in Fig. 6. The display / operation unit 170 is a graphic user interface which displays images and accepts various instructions, and includes a display 172 and a touch panel 174.

[0054] The control unit 160 is an example of a "computer program product" of the disclosed technology.

[0055] The memory 164 is an example of a "memory" in the technology of the present disclosure. The CPU 162 is an example of a "processor" in the technology of the present disclosure. The image processing program is an example of a "program" in the technology of the present disclosure.

[0056] The configuration of the electrical system of the image viewer 150 will be described with reference to Fig. 4. As shown in Fig. 4, the image viewer 150 includes a computer main body 252.

[0057] The computer main body 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268, which are connected to each other via a bus 270. A storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258 are connected to the input / output (I / O) port 268. The storage device 254 is, for example, a non-volatile memory. The input / output (I / O) port 268 is connected to the network 130 via the communication interface (I / F) 258. Therefore, the image viewer 150 can communicate with the UWF ophthalmic device 110 and the management server 140. The storage device 254 stores a data creation program, which will be described later.

[0058] As shown in FIG. 5, the CPU 162 of the management server 140 functions as an image processing unit 182, a display control unit 184, and an output unit 186.

[0059] The image processing unit 182 is an example of an "acquisition unit," a "determination unit," and a "generation unit" of the technology of the present disclosure.

[0060] Next, image processing by management server 140 will be described in detail with reference to Fig. 6. Image processing (image processing method) shown in the flowchart of Fig. 6 is realized by CPU 162 of management server 140 executing an image processing program. The image processing program starts when an instruction to start is given by the operator via touch panel 174.

[0061] When issuing an instruction to start the image processing program, the operator inputs the patient ID via the touch panel 174 .

[0062] In step 302 , the image processor 182 identifies the patient ID entered via the touch panel 174 .

[0063] As described above, the memory 164 stores UWF two-dimensional fundus images, OCT images, and MRI three-dimensional eye images in correspondence with the patient ID. The UWF two-dimensional fundus image is an image capturing blood vessels in the posterior pole and peripheral parts of the fundus, i.e., an image in which the blood vessels in the fundus are visualized. The UWF two-dimensional fundus image is an image in which the blood vessels of the retina and / or choroid are visualized. Since the image is captured up to the peripheral parts, the UWF two-dimensional fundus image has an image of the vortex vein. The UWF two-dimensional fundus image is an example of a "two-dimensional image" of the technology of the present disclosure. The MRI three-dimensional eye image is an example of a "three-dimensional image" of the technology of the present disclosure.

[0064] In step 304, the image processor 182 acquires from the memory 164 a UWF two-dimensional fundus image stored in association with the specified patient ID, and in step 306, the image processor 182 acquires an MRI three-dimensional eye image stored in association with the specified patient ID. Note that the order of execution of steps 304 and 306 is not limited to this, and may be reversed. In either step 304 or step 306, the image processor 182 acquires an OCT image.

[0065] In step 308, the image processor 182 identifies reference points in the UWF two-dimensional fundus image and the MRI three-dimensional eye image that correspond to each other. As an example, the reference points are structural features in the fundus of the test eye that are commonly captured. Thus, the reference points are structural features common to the UWF two-dimensional fundus image and the MRI three-dimensional eye image. There may be one or more reference points. When there are multiple reference points, the positions of multiple structural features located at different positions in one of the UWF two-dimensional fundus image and the MRI three-dimensional eye image are identified as the respective reference points. Then, in the other image, multiple reference points are identified for multiple structural features for which multiple reference points have been identified in the UWF two-dimensional fundus image. The structural features include, first, anatomical features of the test eye such as the optic disc, pupil, and macula, second, vascular structures such as vortex veins, and third, structural elements such as non-uniform fundus depression structures seen in uveitis, etc.

[0066] Here, the multiple reference points are set on the first structural feature and the second structural feature located at a position different from the first structural feature. The first structural feature and the second structural feature are selected from, for example, the optic disc, the pupil, and the macula. For example, the first structural feature is the optic disc, and the second structural feature is, for example, the macula. The multiple reference points may be two or more, for example, three, four, or five. The multiple reference points may be set on the first structural feature, the second structural feature located at a position different from the first structural feature, and a third structural feature located at a position different from the first structural feature and the second structural feature. For example, the first structural feature is the optic disc, the second structural feature is the macula, and the third structural feature is the pupil. In this case, for the optic disc, which is the first structural feature, a first reference point is specified in the MRI three-dimensional eye image, and a second reference point is specified in the UWF two-dimensional fundus image. For a second structural feature, the macula, a second first fiducial point is identified in the MRI 3D eye image and a second second fiducial point is identified in the UWF 2D fundus image, and for a third structural feature, the pupil, a third first fiducial point is identified in the UWF 2D fundus image and a third second fiducial point is identified in the MRI 3D eye image.

[0067] The method of identifying the plurality of reference points in the UWF two-dimensional fundus image and the MRI three-dimensional eye image includes a first method of identifying a plurality of reference points in either the UWF two-dimensional fundus image or the MRI three-dimensional eye image, and then identifying points corresponding to the identified plurality of reference points in the other of the UWF two-dimensional fundus image and the MRI three-dimensional eye image, and a second method of identifying a plurality of reference points independently in each of the UWF two-dimensional fundus image and the MRI three-dimensional eye image. Note that, in the second method, the order of identifying the plurality of reference points may be first identifying from the UWF two-dimensional fundus image and then identifying from the MRI three-dimensional eye image, or the reverse order.

[0068] First, the first method will be described. As shown in Fig. 7, the image processor 182 detects the optic disc 352 by detecting the brightest point in the UWF two-dimensional fundus image 350. The image processor 182 also detects the macula 356 by detecting the darkest point in the UWF two-dimensional fundus image 350.

[0069] Next, the image processor 182 detects points corresponding to the optic disc 352 and the macula 356 detected in the UWF two-dimensional fundus image 350 from the MRI three-dimensional eyeball image 370, as shown in Fig. 8. The detection method will be described below.

[0070] For example, first, the image processor 182 projects the UWF two-dimensional fundus image 350 onto a three-dimensional virtual sphere based on the relational equation of the stereo image projection method. This results in a spherical image onto which the UWF two-dimensional fundus image 350 is projected. In this case, the image processor 182 positions the optic disc at, for example, the north pole on the three-dimensional virtual sphere. Then, the image processor 182 calculates the latitude and longitude coordinates of the macula on the three-dimensional virtual sphere.

[0071] The image processing unit 182 detects the position of the optic disc 373 on the MRI three-dimensional eyeball image 370. Specifically, the image processing unit 182 uses a three-dimensional shape of a predetermined optic nerve structure portion to detect a three-dimensional shape corresponding to the predetermined three-dimensional shape as the optic nerve structure portion in the MRI three-dimensional eyeball image 370 by analyzing a three-dimensional volume image. The image processing unit 182 converts the optic nerve structure portion into a binary image, performs a thinning process to calculate a center line, and obtains an intersection portion between the center line and the MRI three-dimensional eyeball image as the position of the optic disc 373. The image processing unit 182 sets the north pole as the position of the optic disc 373 in the MRI three-dimensional eyeball image 370. The image processing unit 182 detects the latitude and longitude positions calculated in the spherical image onto which the UWF two-dimensional fundus image 350 is projected in the MRI three-dimensional eyeball image 370, and sets the detected position as the position of the macula 375.

[0072] As described above, the technology of the present disclosure is not limited to using a method of detecting the optic disc 352 and the macula 356 in the UWF two-dimensional fundus image 350, and then detecting points corresponding to the optic disc 352 and the macula 356 from the MRI three-dimensional eyeball image 370. The opposite method may also be used. That is, the optic disc 373 and the macula 375 may be detected from the MRI three-dimensional eyeball image 370, and then points corresponding to the optic disc 373 and the macula 375 may be detected from the UWF two-dimensional fundus image 350.

[0073] Specifically, as shown in Fig. 9 and Fig. 10, the image processor 182 uses the predetermined three-dimensional shapes of the optic nerve path and the eyeball to extract a three-dimensional shape corresponding to the three-dimensional shapes in the MRI three-dimensional eyeball image 370 by analyzing the three-dimensional volume image. The extracted three-dimensional shapes are the optic nerve path 374 and the eyeball surface. The image processor 182 detects the intersection of the optic nerve path 374 and the eyeball as the optic disc 373.

[0074] The image processor 182 uses a predetermined three-dimensional shape of the retina to extract a three-dimensional shape corresponding to the predetermined three-dimensional shape in the MRI three-dimensional eye image 370 by analyzing the three-dimensional volume image. The extracted three-dimensional shape is the retina. The image processor 182 detects the macula by detecting the center of the retina. This is because the macula is generally located in the center of the retina.

[0075] Next, the image processing unit 182 detects points corresponding to the optic disc 373 and the macula extracted in the MRI three-dimensional eyeball image 370 from the UWF two-dimensional fundus image 350. Specifically, the image processing unit 182 detects the optic disc 352 in the UWF two-dimensional fundus image 350 as described above, and projects the UWF two-dimensional fundus image 350 onto a sphere based on the relational equation of the stereo image projection method. In this case, the image processing unit 182 positions the position of the optic disc at, for example, the north pole on the sphere. The positional relationship of the points corresponding to the optic disc 373 and the macula extracted in the MRI three-dimensional eyeball image 370 is applied to the sphere. This application detects the position of the macula on the sphere. The image processing unit 182 detects a position corresponding to the position of the macula detected on the sphere in the UWF two-dimensional fundus image 350 using the inverse relational equation of the relational equation of the stereo image projection method.

[0076] In addition, the method of detecting the optic disc from the MRI three-dimensional eye image 370 may be performed using the brightness of the MRI three-dimensional eye image 370, similar to the method of detecting the optic disc 352 in the UWF two-dimensional fundus image 350.

[0077] Next, the second method will be described. By the method described in the first method, the image processor 182 detects the optic disc 352 and the macula 356 in the UWF two-dimensional fundus image 350, and detects the position of the optic disc in the MRI three-dimensional eyeball image. The image processor 182 detects the apex of the eyeball convexity as the position of the cornea center in the MRI three-dimensional eyeball image, and detects the point opposite to the apex as the macula.

[0078] Next, a method for detecting the pupil as a reference point in the MRI 3D eye image 370 will be described. 8, the image processing unit 182 detects the macula 375 and the eyeball center in the MRI three-dimensional eyeball image 370. Specifically, the eyeball center is found by a method of calculating a sphere inscribed in the MRI three-dimensional eyeball image 370 and calculating its center of gravity, etc. Then, the image processing unit 182 detects the pupil 354 by detecting a point on the opposite side of the eyeball center with respect to the macula 375. Alternatively, the image processor 182 detects a locally convex maximum portion as the corneal center 376, as shown in FIG. 11A and FIG. 11B. Usually, the position of the corneal center 376 on the MRI three-dimensional eyeball image 370 (three-dimensional surface spherical image) can be detected as the apex of a convex structure portion having a smaller radius of curvature than the vitreous body. Next, a virtual surface corresponding to the crystalline lens is extrapolated from the MRI three-dimensional eyeball image 370, and the intersection point between the virtual surface and a perpendicular line 377 drawn from the corneal center 376 on the extrapolated surface is set as the pupil 354. Note that the apex of the cornea may not be located at the corneal center, for example, when the shape of the cornea 379 is flattened by corneal surgery or when there is a corneal shape abnormality such as irregular astigmatism or keratoconus. In such a case, the corneal shape in a normal case may be estimated from the obtained MRI three-dimensional eyeball image 370 to identify the virtual corneal apex, or the center point of the flat part of the cornea may be identified as the corneal center.

[0079] Incidentally, in the MRI three-dimensional eyeball image 370 (see FIG. 9 and FIG. 10), a staphyloma 375B or the like may be detected as a convex part corresponding to the cornea. However, the staphyloma 375B and the pupil 354 (see FIG. 8) have different curvatures. After the convex parts are detected, the image processing unit 182 calculates the curvature of the curved surface of each convex part, and removes the part with a curvature greater than a predetermined threshold value from the cornea as the staphyloma 375B. In addition, the staphyloma 375B and the pupil 354 have different coordinates in the MRI three-dimensional eyeball image 370. After the convex parts are detected, the image processing unit 182 specifies the three-dimensional coordinates of each convex part, determines whether the staphyloma 375B is a convex part based on the relative positional relationship with other structural features such as the macula and the optic disc, and removes the part determined to be a staphyloma 375B.

[0080] Next, a method of detecting the pupil as a reference point in the UWF two-dimensional fundus image 350 will be described. As shown in FIG. 12, the pupil does not exist in the UWF two-dimensional fundus image 350 obtained by photographing the fundus 351. Therefore, the position of the pupil on the UWF two-dimensional fundus image 350 is determined as the position of the pupil 378 at a point on the opposite side between the macula and the center of the eyeball after projecting the UWF two-dimensional fundus image 350 onto a spherical surface based on the relational equation of the stereo image projection method. Alternatively, the position of the pupil is determined by extrapolating from an approximate spherical surface obtained by projecting the UWF two-dimensional fundus image 350 onto a spherical surface based on the relational equation of the stereo image projection method.

[0081] The position of the pupil on the UWF two-dimensional fundus image 350 and the position of the pupil 354 on the MRI three-dimensional eye image 370 are made to correspond to each other using the latitude and longitude information on the spherical surfaces of both images. The position of the pupil can be used as a reference point for alignment.

[0082] The reference point may be a vortex vein. The vortex vein is located in the peripheral part of the fundus, and when the two-dimensional image of the subject eye is a UWF fundus image, the peripheral part of the fundus is also imaged, so the vortex vein is also imaged. To detect the vortex vein in the UWF two-dimensional fundus image, the image processing unit 182 first detects blood vessels. Then, since the blood vessels are radial, the center position of the radiation is detected as the vortex vein. The UWF two-dimensional fundus image is an image in which the vortex vein is visualized. In addition, the contents described in International Publication (WO) 2019 / 203309 Pamphlet are incorporated by reference into this specification as if specifically and individually described in this specification.

[0083] In the MRI three-dimensional eye image, the vortex vein may be identified by identifying the outflow route of the choroidal vein to the outside of the eye. In addition, to detect the vortex vein, the MRI three-dimensional eye image may be analyzed to detect the part where blood vessels are concentrated as the vortex vein. Specifically, the vascular concentration structure existing around the equator on the MRI three-dimensional eye image (three-dimensional surface spherical image) can be detected as the position of the vortex vein. The position can be detected by converting the three-dimensional structure part of the vascular concentration structure into a binary image, performing a thinning process to calculate the center line, and determining the intersection part of the center line and the MRI three-dimensional eye image (three-dimensional surface spherical image). In addition, when the three-dimensional image is an OCT volume image rather than an MRI image, the area where blood vessels are concentrated can be detected as a vortex vein.

[0084] In step 310, the image processor 182 matches a common reference point between the UWF two-dimensional fundus image and the MRI three-dimensional eye image, and superimposes the UWF two-dimensional fundus image on at least a part of, for example, the entire MRI three-dimensional eye image, as shown in Fig. 13A. As described above, it is preferable to have at least two reference points.

[0085] Specifically, first, as described above, the image processor 182 stereoscopically projects the UWF two-dimensional fundus image onto a three-dimensional virtual sphere 392 as shown in Fig. 13B. The three-dimensional virtual sphere 392 is, for example, a sphere having a diameter of 24 mm. In the example shown in Fig. 13B, a macula 356, a pupil 378, and a vortex vein 357 are detected.

[0086] Next, using the above-mentioned multiple reference points in each of the UWF two-dimensional fundus image and the MRI three-dimensional eye image, for example, the macula 356, 375, the pupil 378, 354, and the vortex vein 357, 379, the relative positions are made to correspond in each of the three-dimensional virtual sphere 392 onto which the UWF two-dimensional fundus image is projected and the MRI three-dimensional eye image 370. More specifically, the image processor 182 makes the latitude and longitude of the three-dimensional virtual sphere correspond to the latitude and longitude of the MRI three-dimensional eye image. In FIG. 13B, the eye surface of the MRI three-dimensional eye image 370 is highlighted.

[0087] Then, the image processing unit 182 superimposes the UWF two-dimensional fundus image on the MRI three-dimensional eye image so that the points on the three-dimensional virtual sphere 392 onto which the UWF two-dimensional fundus image is projected correspond to the points on the MRI three-dimensional eye image 370. That is, each point of the UWF two-dimensional fundus image projected on the three-dimensional virtual sphere 392 is reprojected to a correct position using information on the position coordinates of each pixel present on the eye surface of the MRI three-dimensional eye image 370. More specifically, the image processing unit 182 searches for a point in the MRI three-dimensional eye image having the same latitude and longitude as the latitude and longitude of each point on the UWF spherical projection image sphere, and projects the pixel of the point on the UWF spherical projection image sphere to the searched point. The above is executed for all points of the UWF spherical projection image.

[0088] The information required for the process of matching the relative positions in each of the three-dimensional virtual sphere and the MRI three-dimensional eye image in step 310 above is information on latitude and longitude in the three-dimensional virtual sphere and the MRI three-dimensional eye image, and information on length, size, and scale is not necessarily required. Therefore, the lengths in each of the three-dimensional virtual sphere and the MRI three-dimensional eye image may be different from each other. For example, the distance between the macula 356 and the pupil 354 may be different in the three-dimensional virtual sphere and the MRI three-dimensional eye image. However, the size of the three-dimensional virtual sphere and the MRI three-dimensional eye image may be made the same using a value of the axial length measured by axial length measurement or a value of the equatorial diameter measured by an ultrasound device.

[0089] In step 312, the display control unit 184 creates a display screen 400 for displaying information and images for a GUI (Graphical User Interface), which will be described later.

[0090] In step 314, the output unit 186 outputs the data to the display screen 400. Specifically, the data is stored in the memory 164 in association with the patient ID, and is transmitted to the image viewer 150 together with the patient ID. Once step 314 is completed, the image processing program ends.

[0091] Next, the display screen 400 will be described with reference to Fig. 14. As shown in Fig. 14, the display screen 400 has an information area 402 and an image display area 404. The information area 402 has a patient ID display field 406, a patient name display field 408, an age display field 410, a visual acuity display field 412, a right eye / left eye display field 414, and an axial length display field 416. In each of the display fields from the patient ID display field 406 to the axial length display field 416, the image viewer 150 displays each piece of information based on information received from the management server 140.

[0092] The layout of the image display area 404 is a screen layout that is likely to be of interest to ophthalmologists. Specifically, it is as follows.

[0093] The image display area 404 has a first section 420 for displaying a UWF two-dimensional fundus image and a second section 422 for displaying an image in which the UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image.

[0094] The image displayed in the second section 422, in which the UWF two-dimensional fundus image is superimposed on the MRI three-dimensional eye image, can be rotated, for example, based on the center of the eye, according to the operation of an attendant.

[0095] The image display area 404 has a third section 424 for displaying a posterior segment tomographic image composed of a UWF two-dimensional fundus image and an OCT tomographic image, a fourth section 426 for displaying an OCT tomographic image of the fundus, and a fifth section 428 for displaying a panoramic image (i.e., a two-dimensional frontal image) of the en-face of the choroid (a plane perpendicular to the measurement optical axis by OCT).

[0096] The image display area 404 has a sixth section 430 for displaying comments.

[0097] As described above, in this embodiment, an image is generated by superimposing a UWF two-dimensional fundus image on an MRI three-dimensional eye image, so that an image can be generated that allows the state of the fundus, for example, the state of blood vessels and vortex veins of the fundus, to be recognized three-dimensionally. This is useful for ophthalmologists in making diagnoses. Therefore, ophthalmologists can make highly accurate diagnoses and evaluate diseases.

[0098] For example, a UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image, so that when diagnosing strabismus, it can be diagnosed in conjunction with the state of related retinal diseases.

[0099] In addition, since a UWF two-dimensional fundus image is obtained by photographing the fundus through a wide-angle optical system, the appearance of a wider area of ​​the fundus can be recognized in three dimensions.

[0100] Next, a modification of this embodiment will be described.

[0101] A first modified example will be described. In the above embodiment, an image in which a UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image is displayed in the second section 422 of the image display area 404 of the display screen 400 in FIG. 14. The technology of the present disclosure is not limited to this. For example, as shown in FIG. 15, an image in which a UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image may be superimposed on an OCT tomographic image of the entire subject's eye, including the anterior and posterior segments. The state of the fundus can be recognized three-dimensionally in relation to the state of the tomography.

[0102] A second modified example will be described. In the above embodiment, the second section 422 in FIG. 14 displays an image in which the entire UWF two-dimensional fundus image is superimposed on the MRI three-dimensional eye image. The technology of the present disclosure is not limited to this. For example, as shown in FIG. 16, the UWF two-dimensional fundus image superimposed on the MRI three-dimensional eye image 370 is a partial region image 382, ​​384 of the UWF two-dimensional fundus image. The partial region may be, for example, an image obtained by trimming a part of the two-dimensional image, an image of a lesion, or a fundus vascular image in which only blood vessels are extracted from the fundus image. The lesion image is, for example, an image obtained by cutting out a region of a fundus image that is specified by a doctor or AI. An example of a lesion is a retinal detachment. An image 382 of the retinal detachment is displayed in green, for example. An image 384 of the blood vessel portion is displayed in red, for example. The MRI three-dimensional eye image 370 is displayed semi-transparently. Either the image of the lesion or the image of the blood vessel portion may be selectively displayed in sequence. Thus, only the image of the lesion or only the image of the blood vessel portion may be displayed on the MRI three-dimensional eye image 370 that is displayed semi-transparently.

[0103] A third modified example will be described. As described above, the memory 164 stores a UWF two-dimensional fundus image and an MRI three-dimensional eye image in correspondence with the patient ID. The technology of the present disclosure is not limited to this. For example, the memory 164 stores a UWF two-dimensional fundus image, a posterior segment OCT image, and an anterior segment OCT image in correspondence with the patient ID. In step 304, the image processor 182 acquires the UWF two-dimensional fundus image, the posterior segment OCT image, and the anterior segment OCT image stored in correspondence with the patient ID from the memory 164.

[0104] In step 310, the image processor 182 generates a first superimposed image by superimposing the UWF two-dimensional fundus image on at least a part of the MRI three-dimensional eye image, and generates a second superimposed image by superimposing the first superimposed image on the posterior segment OCT image. The second section 422 displays the first superimposed image and the second superimposed image. Also, image processing may be performed to obtain shape data of the posterior segment of the subject's eye from the posterior segment OCT image, and to deform the first superimposed image based on the shape data, for example, so as to correspond to the shape of the posterior segment of the subject's eye. Alternatively, in step 310, the image processor 182 generates an image in which an UWF two-dimensional fundus image is superimposed on an MRI three-dimensional eye image, and the image is superimposed on an anterior segment OCT image. In the second section, an image in which an UWF two-dimensional fundus image is superimposed on at least a part of an MRI three-dimensional eye image, and the image is superimposed on an anterior segment OCT image is displayed. Also, image processing may be performed to obtain shape data of the cornea of ​​the subject's eye from the anterior segment OCT image, and to deform the first superimposed image based on the shape data.

[0105] A fourth modified example will be described. The two-dimensional image stored in the memory 164 in correspondence with the patient ID as described above is one two-dimensional image obtained at a certain time. The technology of the present disclosure is not limited to this. For example, the memory 164 may store a plurality of fundus images of the test eye, including a first fundus image of the test eye and a second fundus image of the test eye obtained by imaging the fundus at a time different from the first fundus image, in correspondence with the patient ID as the two-dimensional image. A predetermined event occurs between the time when the first fundus image of the test eye is obtained and the time when the second fundus image of the test eye is obtained. The predetermined event is, for example, surgery on the test eye. Hereinafter, as an example, a case where the two-dimensional image is a first UWF two-dimensional fundus image and a second UWF two-dimensional fundus image will be described. The first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image are examples of the "first two-dimensional image" and the "second two-dimensional image" of the technology of the present disclosure.

[0106] In step 304, the image processor 182 acquires the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image corresponding to the patient ID from the memory 164.

[0107] In step 310, the image processor 182 superimposes the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image on the MRI three-dimensional eye image. In this case, the image processor 182 may superimpose the difference between the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image on the MRI three-dimensional eye image. Alternatively, the image processor 182 may superimpose each of the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image on the MRI three-dimensional eye image, and the image processor 182 may extract the difference between the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image superimposed on the MRI three-dimensional eye image.

[0108] In this way, the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image are superimposed on the MRI three-dimensional eye image, so that the state of the fundus before and after surgery on the subject eye can be recognized three-dimensionally. Note that the difference between the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image superimposed on the MRI three-dimensional eye image, and the first UWF two-dimensional fundus image and the second UWF two-dimensional fundus image superimposed on the MRI three-dimensional eye image may each be an image of only a predetermined area including the surgical site.

[0109] An example of such surgery is scleral buckling surgery, that is, a treatment to prevent the progression of rhegmatogenous retinal detachment by pressing silicone from the outside of the eyeball (the sclera).

[0110] The first UWF two-dimensional fundus image taken before surgery is superimposed on the MRI three-dimensional eye image, which helps to understand where the retinal detachment is located when viewed from the outside of the eye. In addition, the second UWF two-dimensional fundus image taken after surgery is superimposed on the MRI three-dimensional eye image, which allows you to confirm that the silicone is being pressed against the retinal detachment.

[0111] Various further modifications will be described below. In the above example, an MRI three-dimensional eye image is used as the three-dimensional image of the subject's eye, but the technology of the present disclosure is not limited thereto. The three-dimensional image may be an image obtained using a device that generates three-dimensional image data of the subject's eye. Other examples of the three-dimensional image include an X-ray CT (Computed Tomography) three-dimensional eye image, an ultrasonic three-dimensional eye image, an OCT image (Whole-eye OCT image) of the entire subject's eye, and an OCT Angiography (OCTA) image of the entire subject's eye. The three-dimensional image of the subject's eye can also be referred to as a three-dimensional image of the subject's eye obtained using an MRI device, an X-ray CT device, an ultrasonic imaging device, an OCT device, or the like, which is a first device of a first modality (medical image diagnostic device) that generates three-dimensional image data of the subject's eye. When an X-ray CT (Computed Tomography) three-dimensional eye image or an ultrasonic three-dimensional eye image is used as the three-dimensional image, the optic disc, vortex vein, pupil, or the like can be used as the reference point. In addition, when the 3D image of the subject's eye is an OCT image including the posterior pole, the optic disc and vortex vein on the image can be used. In addition, when the OCT image is an OCT image of only the posterior pole, the position of the pupil estimated from the actual shape display can be used as the reference point. However, the above-mentioned three-dimensional images are images obtained using a device that generates three-dimensional image data of the subject's eye (such as an MRI device, an X-ray CT device, an ultrasonic imaging device, or an OCT device), and therefore include images that are not displayed three-dimensionally (stereoscopically). The technology of the present disclosure is not limited to this. For example, a three-dimensional image of the subject's eye may be an image that is drawn on a flat surface so as to have a three-dimensional impression, and that is reconstructed and recognized as a three-dimensional image in the brain by utilizing binocular parallax. The three-dimensional image of the subject's eye may be a stereoscopic image obtained by performing image processing (such as segmentation) on the three-dimensional image. The stereoscopic image of the subject's eye is an image that stereoscopically (three-dimensionally) displays at least a part of the subject's eye.

[0112] In the above example, the UWF two-dimensional fundus image is used as the two-dimensional image of the subject's eye superimposed on the MRI three-dimensional eye image, but the technology of the present disclosure is not limited to this. Other examples of the two-dimensional image include a fundus camera image, an en-face OCT, and an en-face OCTA image. The two-dimensional image may be a two-dimensional tomographic image created using OCT volume data. In addition, the fundus image can also be referred to as a two-dimensional image (also called a planar image or a front image) of the subject's eye obtained using an SLO device, an OCT device (which generates an en-face image from OCT volume data), a fundus camera, or the like, which is a second device of the second modality that generates two-dimensional image data of the subject's eye. In addition, the image of the subject's eye superimposed on the MRI three-dimensional eye image may be an image acquired by a device of a modality different from the MRI, and may not necessarily be a two-dimensional image. For example, it may be a three-dimensional image such as OCT volume data of the entire subject's eye, or may be image data in which time information is added to a three-dimensional image such as an OCT angiography image of the entire subject's eye. By superimposing the OCT volume data on the MRI three-dimensional eye image, the three-dimensional information of the subject's eye can be mutually complemented. Specifically, by superimposing the OCT volume data on the MRI three-dimensional eye image, rough information from the MRI three-dimensional eye image is obtained for the area not optically scanned by the OCT device, and detailed vascular information is obtained for the area optically scanned by the OCT device, thereby obtaining a three-dimensional image of the subject's eye. When the OCT volume image is superimposed on the MRI image, the vortex vein, pupil, and ONH can also be used as reference points. Similarly, the OCT volume image may be superimposed on a CT image or an ultrasound image of the subject's eye.

[0113] In the above examples, the management server 140 executes the image processing program, but the technology of the present disclosure is not limited to this and may be executed, for example, by any of the UWF ophthalmic device 110, the MRI image acquisition device 120, the image viewer 150, other ophthalmic equipment (examination equipment for visual field measurement, intraocular pressure measurement, etc.), and a diagnostic support device that performs image analysis using artificial intelligence.

[0114] The UWF ophthalmic apparatus 110 can omit the OCT unit 20. In this case, the focus adjustment mechanism 28 of the imaging optical system 19 and the optical path synthesis element 21 are also omitted.

[0115] In the present disclosure, each component (device, etc.) may be present in only one or in two or more instances, unless a contradiction arises.

[0116] In each of the above-described examples, the image processing is realized by a software configuration using a computer, but the technology of the present disclosure is not limited to this. For example, instead of a software configuration using a computer, the image processing may be performed only by a hardware configuration such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A part of the image processing may be performed by a software configuration, and the remaining processing may be performed by a hardware configuration.

[0117] As described above, the technology of the present disclosure includes cases in which image processing is realized by a software configuration using a computer and cases in which it is not realized, and therefore includes the following technologies.

[0118] (First Technology) an acquisition unit that acquires a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; An identification unit that identifies a first reference point in the three-dimensional image and a second reference point in the two-dimensional image; a generating unit that generates an image in which the first reference point in the three-dimensional image corresponds to the second reference point in the two-dimensional image and the two-dimensional image is superimposed on at least a portion of the three-dimensional image; An image processing device comprising:

[0119] (Second Technology) An acquisition unit acquires a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; A step in which an identification unit identifies a first reference point in the three-dimensional image and a second reference point in the two-dimensional image; A generation unit generates an image in which the first reference point in the three-dimensional image corresponds to the second reference point in the two-dimensional image and the two-dimensional image is superimposed on at least a portion of the three-dimensional image; An image processing method comprising:

[0120] Based on the above disclosure, the following techniques are proposed. (Third Technology) 1. A computer program product for processing an image, comprising: The computer program product comprises a computer-readable storage medium that is not itself a transitory signal; The computer-readable storage medium stores a program, The program is On the computer, acquiring a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; A step of generating an image in which the first reference point in the three-dimensional image corresponds to the second reference point in the two-dimensional image and the two-dimensional image is superimposed on at least a portion of the three-dimensional image; A computer program product that causes the

[0121] Based on the above disclosure, the following techniques are proposed. (Fourth Technology) acquiring a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; A step of generating an image in which the first reference point in the three-dimensional image corresponds to the second reference point in the two-dimensional image and the two-dimensional image is superimposed on at least a portion of the three-dimensional image; diagnosing the subject's eye based on an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image; A method for diagnosing an eye to be examined comprising the steps of:

[0122] The image processing described above is merely an example. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed without departing from the spirit of the invention.

[0123] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0124] 140 Management Server 160 Control Unit 162 CPU 164 Memory 182 Image Processing Unit 184 Display control section 186 Output section

Claims

1. On the computer, acquiring a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image by associating the first reference point in the three-dimensional image with the second reference point in the two-dimensional image. The two-dimensional image superimposed on the three-dimensional image is a difference between a first image of the subject's eye and a second image of the subject's eye captured at a time different from that of the first image of the subject's eye.

2. On the computer, acquiring a three-dimensional image of an entire subject's eye including an anterior segment and a posterior segment of the subject's eye, and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image by associating the first reference point in the three-dimensional image with the second reference point in the two-dimensional image. the two-dimensional image includes a first subject's eye image and a second subject's eye image captured at a time different from that of the first subject's eye image, the step of generating a superimposed image includes generating an image in which the first subject's eye image and the second subject's eye image are superimposed on the three-dimensional image, A program comprising: a step of extracting a difference between the first image of the subject's eye and the second image of the subject's eye from an image superimposed on the three-dimensional image.

3. On the computer, 3. The program according to claim 1, further comprising: a step of outputting the superimposed image.

4. The program according to claim 1 , wherein the two-dimensional image superimposed on at least a portion of the three-dimensional image is an image of at least a partial area of ​​a fundus.

5. The program according to claim 1 , wherein the two-dimensional image superimposed on at least a portion of the three-dimensional image is an image in which blood vessels in a fundus are visualized.

6. The program according to claim 1 , wherein the two-dimensional image superimposed on at least a portion of the three-dimensional image is an image in which a vortex vein is visualized.

7. The program according to claim 1 , wherein the two-dimensional image is a fundus image obtained by imaging the fundus through a wide-angle optical system.

8. The computer-readable medium according to claim 1 , wherein the first reference point and the second reference point include a structural feature common to the three-dimensional image and the two-dimensional image.

9. Identifying a plurality of first reference points located at different positions in the three-dimensional image; Identifying a plurality of second reference points located at different positions in the two-dimensional image; the plurality of second reference points are structural features corresponding to the plurality of first reference points, The program according to any one of claims 1 to 8.

10. The first reference point and the second reference point are any one of a macula, an optic disc, and a pupil. The program according to any one of claims 1 to 9.

11. The step of generating the superimposed image includes: projecting the two-dimensional image onto a three-dimensional virtual sphere; superimposing the two-dimensional image and the three-dimensional image projected onto the three-dimensional virtual sphere such that a second reference point of the two-dimensional image projected onto the three-dimensional virtual sphere coincides with a first reference point of the three-dimensional image; The program according to any one of claims 1 to 10, comprising:

12. The step of generating the superimposed image includes: generating a first superimposed image by superimposing the first image of the subject's eye on at least a portion of the three-dimensional image; generating a second superimposed image by superimposing the second test eye image on at least a portion of the first superimposed image; The program according to claim 2 .

13. The computer-readable medium according to claim 1 , wherein the three-dimensional image is a three-dimensional image of the entire subject's eye including an anterior eye segment and a posterior eye segment.

14. the two-dimensional image is an SLO image and a posterior segment OCT image, The step of generating the superimposed image includes: Superimposing the SLO image and the posterior segment OCT image on at least a portion of the three-dimensional image. The program according to any one of claims 1 to 13, comprising:

15. the two-dimensional image is a fundus image and an anterior segment OCT image, The step of generating the superimposed image includes: The program according to claim 1 , further comprising a step of generating an image in which the fundus image and the anterior segment OCT image are superimposed on at least a portion of the three-dimensional image.

16. the two-dimensional image is a fundus image and an OCT image of the entire test eye, The program according to any one of claims 1 to 13, wherein the step of generating the superimposed image includes a step of generating an image in which the fundus image and an OCT image of the entire test eye are superimposed on at least a portion of the three-dimensional image.

17. The program according to claim 1 , wherein the two-dimensional image superimposed on the three-dimensional image is an image including a region indicating a lesion.

18. The program according to claim 1 , wherein the three-dimensional image is an MRI three-dimensional eye image.

19. a memory; and a processor coupled to the memory; The processor, acquiring a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image by associating the first reference point in the three-dimensional image with the second reference point in the two-dimensional image, The two-dimensional image superimposed on the three-dimensional image is a difference between a first image of the subject's eye and a second image of the subject's eye captured at a time different from that of the first image of the subject's eye.

20. a memory; and a processor coupled to the memory; The processor, acquiring a three-dimensional image of an entire subject's eye including an anterior segment and a posterior segment of the subject's eye, and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image by associating the first reference point in the three-dimensional image with the second reference point in the two-dimensional image, the two-dimensional image includes a first subject's eye image and a second subject's eye image captured at a time different from that of the first subject's eye image, the step of generating a superimposed image includes generating an image in which the first subject's eye image and the second subject's eye image are superimposed on the three-dimensional image, An image processing apparatus comprising: extracting a difference between the first image of the subject's eye and the second image of the subject's eye from an image superimposed on the three-dimensional image.

21. acquiring a three-dimensional image of a subject's eye and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; and generating an image in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image by making the first reference point in the three-dimensional image correspond to the second reference point in the two-dimensional image, an image processing method, wherein the two-dimensional image superimposed on the three-dimensional image is a difference between a first image of the subject's eye and a second image of the subject's eye captured at a time different from that of the first image of the subject's eye;

22. acquiring a three-dimensional image of an entire subject's eye including an anterior segment and a posterior segment of the subject's eye, and a two-dimensional image of the subject's eye; identifying a first reference point in the three-dimensional image; identifying a second reference point in the two-dimensional image; The first reference point in the three-dimensional image is made to correspond to the second reference point in the two-dimensional image, and an image is generated in which the two-dimensional image is superimposed on at least a portion of the three-dimensional image. and the two-dimensional image includes a first subject's eye image and a second subject's eye image captured at a time different from that of the first subject's eye image, the step of generating a superimposed image includes generating an image in which the first subject's eye image and the second subject's eye image are superimposed on the three-dimensional image, An image processing apparatus comprising: extracting a difference between the first image of the subject's eye and the second image of the subject's eye from an image superimposed on the three-dimensional image.

Citation Information

Patent Citations

  • Constructing / displaying device for three-dimensional eye fundus image

    JP2006061196A

  • Ophthalmologic observation system

    JP2012075640A

  • Image processing method, image processing device, and program

    JP2018075229A

  • Ophthalmic imaging apparatus and ophthalmic information processing apparatus

    US20170273558A1

  • Image analysis device for ophthalmic disease, image analysis method for ophthalmic disease, and image analysis program for ophthalmic disease

    WO2013031536A1