Image processing method, image processing apparatus, and image processing program

Abstract

To provide an image processing method for more accurately diagnosing an eye to be examined of a patient based on a montage image in which a region of a fundus equatorial portion and a fundus peripheral portion is captured.SOLUTION: An image processing method includes: obtaining a first direction fundus image captured in a state where an eye to be examined is directed in a first direction and a second direction fundus image captured in a state where the eye to be examined is directed in a second direction different from the first direction; generating a composite image for analyzing a fundus peripheral portion of the eye to be examined by composing the first direction fundus image and the second direction fundus image; and outputting the composite image.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, and an image processing program. [Background technology]

[0002] US Patent Application Publication No. 2009 / 0136100 discloses an apparatus and method for synthesizing a panoramic fundus image, and there is a need for an image synthesis method suitable for analyzing the structure of the peripheral part of the fundus. Summary of the Invention

[0003] An image processing method according to a first aspect of the disclosed technology includes acquiring a first direction fundus image captured with the subject's eye facing in a first direction and a second direction fundus image captured with the subject's eye facing in a second direction different from the first direction, combining the first direction fundus image and the second direction fundus image to generate a composite image for analyzing the peripheral portion of the fundus of the subject's eye, and outputting the composite image.

[0004] An image processing device according to a second aspect of the disclosed technology includes an acquisition unit that acquires a first direction fundus image captured with the subject's eye facing in a first direction and a second direction fundus image captured with the subject's eye facing in a second direction different from the first direction, a generation unit that generates a composite image for analyzing the peripheral area of ​​the fundus of the subject's eye by combining the first direction fundus image and the second direction fundus image, and an output unit that outputs the composite image.

[0005] The image processing program of the third aspect of the technology disclosed herein causes a computer to function as an acquisition unit that acquires a first direction fundus image taken with the test eye facing in a first direction and a second direction fundus image taken with the test eye facing in a second direction different from the first direction, a generation unit that generates a composite image for analyzing the peripheral area of ​​the fundus of the test eye by combining the first direction fundus image and the second direction fundus image, and an output unit that outputs the composite image. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of an ophthalmology system 100. [Figure 2] 1 is a schematic diagram showing the overall configuration of an ophthalmologic apparatus 110. FIG. [Figure 3A] 1 is a first diagram showing a photographing range of the fundus of the subject's eye 12 by the ophthalmologic apparatus 110. FIG. [Figure 3B] 10 is a second diagram showing the photographing range of the fundus of the subject's eye 12 by the ophthalmologic apparatus 110, and is an image of the fundus obtained by the photographing. FIG. [Figure 3C] 10 is a third diagram showing the photographing range of the fundus of the subject's eye 12 by the ophthalmologic apparatus 110. FIG. [Figure 4] FIG. 2 is a functional block diagram of a CPU 22 of the ophthalmologic apparatus 110. [Figure 5] 10 is a flowchart showing processing for photographing the fundus of the subject's eye 12 executed by a CPU 22 of the ophthalmologic apparatus 110. [Figure 6A] 1 is a diagram showing the fundus imaging range (U1 to D1) in a plane parallel to the vertical direction passing through the pupil and the center of the eyeball when the optical axis of the eye 12 to be examined is directed obliquely upward with the ophthalmologic system 100 as the reference. [Figure 6B] 1 is a diagram showing the fundus photography range (U2 to D2) in a plane parallel to the vertical direction passing through the pupil and the center of the eyeball when the optical axis of the eye 12 to be examined is directed diagonally downward with the ophthalmologic system 100 as the reference. [Figure 7A] FIG. 6 is a diagram showing a UWF top-view fundus image GU obtained by top-view photography in step 304 of FIG. 5. [Figure 7B] FIG. 6 is a diagram showing a UWF downward-looking fundus image GD obtained by downward-looking photography in step 308 of FIG. 5. [Figure 8] FIG. 2 is a block diagram of the electrical system configuration of the server 140. [Figure 9] FIG. 2 is a functional block diagram of a CPU 262 of the server 140. [Figure 10] 10 is a flowchart showing a montage image creation process executed by a CPU 262 of the server 140. [Figure 11] 11 is a flowchart showing the process of aligning images in step 324 of FIG. 10. [Figure 12A] FIG. 10 is a diagram showing a state in which a line segment LGU is set on a UWF upward-viewing fundus image GU. [Figure 12B] FIG. 10 is a diagram showing a state in which a line segment LGD is set on a UWF downward-looking fundus image GD. [Figure 13] FIG. 10 is a diagram for explaining the generation of a montage image GM. [Figure 14] 4 is a diagram showing a first display mode of the display screen 400 of the display 256 of the viewer 150. FIG. [Figure 15] 10 is a diagram showing a second display mode of the display screen 400 of the display 256 of the viewer 150. FIG. [Figure 16] FIG. 10 is a diagram showing a third display mode of the display screen 400 of the display 256 of the viewer 150. [Figure 17] FIG. 10 is a diagram showing a fourth display mode of the display screen 400 of the display 256 of the viewer 150. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0008] The configuration of an ophthalmologic system 100 will be described with reference to Fig. 1. As shown in Fig. 1, the ophthalmologic system 100 includes an ophthalmologic apparatus 110, an axial length measuring device 120, a server apparatus (hereinafter referred to as "server") 140, and a display device (hereinafter referred to as "viewer") 150. The ophthalmologic apparatus 110 acquires fundus images. The axial length measuring device 120 measures the axial length of a patient. The server 140 stores a plurality of fundus images and axial lengths obtained by photographing the funduses of a plurality of patients using the ophthalmologic apparatus 110, corresponding to the patient IDs. The viewer 150 displays the fundus images and analysis results acquired by the server 140. The server 140 is an example of the "image processing device" of the technology of the present disclosure.

[0009] The ophthalmologic apparatus 110, the axial length measuring device 120, the server 140, and the viewer 150 are connected to one another via a network 130.

[0010] In addition, other ophthalmic devices (examination devices for visual field measurement, intraocular pressure measurement, etc.) and diagnostic support devices that perform image analysis using AI (Artificial Intelligence) may be connected to the ophthalmic device 110, the axial length measuring device 120, the server 140, and the viewer 150 via the network 130.

[0011] Next, the configuration of the ophthalmologic apparatus 110 will be described with reference to Fig. 2. As shown in Fig. 2, the ophthalmologic apparatus 110 includes a control unit 20, a display / operation unit 30, and an SLO unit 40. The posterior segment (fundus) of the subject's eye 12 is photographed. The apparatus may further include an OCT unit (not shown) that acquires OCT data of the fundus. Here, "SLO" stands for scanning laser ophthalmoscope. "OCT" stands for optical coherence tomography.

[0012] The control unit 20 includes a computer having a CPU 22, a memory 24, and a communication interface (I / F) 26. The display / operation unit 30 is a graphic user interface that displays captured images and receives various instructions including instructions for capturing images, and includes a display 32 and an input / instruction device 34. As CPU 22 executes the photographing processing program, CPU 22 functions as an SLO control unit 180 (including a fixation light control unit 1802, an SLO light source control unit 1804, and a scanner control unit 1806), an image processing unit 182, a display control unit 184, and a processing unit 186, as shown in FIG. 4.

[0013] The memory 24 stores an imaging processing program for imaging the fundus of the subject's eye 12, which will be described later.

[0014] The SLO unit 40 includes a light source 42 of G light (green light: wavelength 530 nm), a light source 44 of R light (red light: wavelength 650 nm), and a light source 46 of IR light (infrared (near-infrared light): wavelength 800 nm). The light sources 42, 44, and 46 emit their respective lights in response to commands from the control unit 20. The light sources 42, 44, and 46 may be LED light sources or laser light sources. An example using a laser light source will be described below.

[0015] The SLO unit 40 includes optical systems 50, 52, 54, and 56 that reflect or transmit light from the light sources 42, 44, and 46 to guide the light into a single optical path. The optical systems 50 and 56 are mirrors, and the optical systems 52 and 54 are beam splitters, specifically, dichroic mirrors, half mirrors, or the like.

[0016] The G light is reflected by the optical systems 50 and 54, the R light is transmitted through the optical systems 52 and 54, and the IR light is reflected by the optical systems 52 and 56, and each is guided to one optical path.

[0017] The SLO unit 40 includes a wide-angle optical system 80 that two-dimensionally scans light from the light sources 42, 44, and 46 over the posterior segment (fundus) of the subject's eye 12. The SLO unit 40 includes a beam splitter 58 that reflects G light from the posterior segment (fundus) of the subject's eye 12 and transmits all light other than G light. The SLO unit 40 includes a beam splitter 60 that reflects R light from the light that has passed through the beam splitter 58 and transmits all light other than R light. The SLO unit 40 includes a beam splitter 62 that reflects IR light from the light that has passed through the beam splitter 60. A dichroic mirror, a half mirror, or the like can be used as the beam splitters 58, 60, and 62.

[0018] The SLO unit 40 includes a G light detecting element 72 that detects the G light reflected by the beam splitter 58, an R light detecting element 74 that detects the R light reflected by the beam splitter 60, and an IR light detecting element 76 that detects the IR light reflected by the beam splitter 62. The light detecting elements 72, 74, 76 may be, for example, an APD (avalanche photodiode). The SLO unit 40 also includes a fixation target control device 90 that is controlled by the control unit 20 and turns on an upper fixation light 92U and a lower fixation light 92D (and further a central fixation light, not shown). By turning on any one of the central fixation light, the upper fixation light 92U, and the lower fixation light 92D, the direction (gaze direction) of the subject's eye 12 can be changed.

[0019] The wide-angle optical system 80 includes an X-direction scanning device 82 made up of a polygon mirror that scans the light from the light sources 42, 44, and 46 in the X direction, a Y-direction scanning device 84 made up of a galvanometer mirror that scans the light in the Y direction, and an ultra-wide-angle (UWF: Ultra Wide Field) scanning device that converts the scanned light into a The ophthalmic apparatus 110 is equipped with an optical system 86 that includes a lens system consisting of a concave mirror such as an elliptical mirror that can irradiate light in a wide angle and a plurality of lenses. Each of the scanning devices, the X-direction scanning device 82 and the Y-direction scanning device 84, may use a MEMS (Micro Electro Mechanical System) mirror. Furthermore, two-dimensional scanning may be performed with a single MEMS mirror without providing separate scanners in the X and Y directions. Note that the horizontal direction when the ophthalmic apparatus 110 is placed on a horizontal plane is defined as the "X direction," the direction perpendicular to the horizontal plane as the "Y direction," and the direction connecting the center of the pupil of the anterior segment of the subject's eye 12 and the center of the eyeball as the "Z direction." Therefore, the X direction, Y direction, and Z direction are perpendicular to each other.

[0020] The wide-angle optical system 80 makes the field of view (FOV) of the fundus an ultra-wide angle, and can capture an image of the area from the posterior pole of the fundus of the subject's eye 12 beyond the equator.

[0021] The equatorial region 178 will be explained using Figure 3A. The eyeball (examined eye 12) is a spherical structure with a diameter of approximately 24 mm and an eyeball center 170. The straight line connecting the anterior pole 175 and posterior pole 176 is called the ocular axis 172, and the line where a plane perpendicular to the ocular axis 172 intersects with the surface of the eyeball is called a latitude line, with the longest latitude line being the equator 174. The part of the retina and choroid corresponding to the position of the equator 174 is called the equatorial region 178. The equatorial region 178 is part of the peripheral part of the fundus. The ophthalmologic apparatus 110 can capture an image of an area with an internal illumination angle of 200°, with the center 170 of the eyeball of the subject's eye 12 as the reference position. The internal illumination angle of 200° corresponds to an external illumination angle of 167° with the pupil of the eyeball of the subject's eye 12 as the reference. In other words, the wide-angle optical system 80 irradiates the laser light from the pupil at an angle of view of an external illumination angle of 167°, and captures an image of the fundus area with an internal illumination angle of 200°.

[0022] 3B shows an SLO image 179 obtained by capturing an image using an ophthalmic apparatus 110 capable of scanning at an internal illumination angle of 200°. As shown in FIG. 3B, an equator 178 corresponds to an internal illumination angle of 180°, and the area indicated by a dotted line 178a in the SLO image 179 corresponds to the equator 178. In this way, the ophthalmic apparatus 110 can capture an image of the fundus peripheral region from the posterior pole including the posterior pole 176 to beyond the equator 178 all at once (in a single capture or a single scan). In other words, the ophthalmic apparatus 110 can capture an image of the fundus from the center to the peripheral region all at once.

[0023] FIG. 3C is a diagram showing the positional relationship between the choroid 12M and vortex veins 12V1 and V2 in the eyeball. In FIG. 3C, the mesh-like pattern represents the choroidal blood vessels of the choroid 12M. The choroidal blood vessels circulate blood throughout the entire choroid. Blood flows out of the eyeball through multiple (usually four to six) vortex veins present in the subject's eye 12. FIG. 3C shows the superior vortex vein V1 and the inferior vortex vein V2 present on one side of the eyeball. The vortex veins are often present near the equator 178. Therefore, to photograph the vortex veins present in the subject's eye 12 and the choroidal blood vessels around the vortex veins, the ophthalmic apparatus 110 described above is used, which has an internal illumination angle of 200° and can scan a wide area around the fundus.

[0024] An SLO fundus image obtained by photographing the eye of the subject eye 12 using an ophthalmic apparatus 110 capable of scanning at an internal illumination angle of 200° is referred to as a UWF fundus image.

[0025] The ophthalmic apparatus 110 equipped with such a wide-angle optical system may be configured as described in International Application PCT / EP2017 / 075852. The disclosure of International Application PCT / EP2017 / 075852 (International Publication WO2018 / 069346), filed on October 10, 2017, is incorporated herein by reference in its entirety.

[0026] The following describes how the ophthalmologic apparatus 110 captures UWF fundus images in upward and downward gaze. First, patient attribute information such as a patient ID and a patient name, information as to whether the eye to be photographed is the right eye or the left eye, and the like are input to the ophthalmologic apparatus 110 via the input / instruction device 34. If the patient has a medical history, the patient ID is input and the patient attribute information recorded in the server 140 is read out. Next, the ophthalmologic apparatus 110 displays a menu screen on the display 32 to allow the user to select between a normal imaging mode in which a central fixation light is turned on to image a wide area of ​​the fundus, and a montage image imaging mode in which the structure of the peripheral part of the fundus (for example, vortex veins and choroidal blood vessels around the vortex veins) is analyzed. The user can select a mode from the menu screen displayed on the display 32 via the input / instruction device 34.

[0027] When the user selects the montage image capturing mode, the CPU 22 of the ophthalmic apparatus 110 executes a capturing processing program, thereby realizing the capturing processing shown in the flowchart of FIG.

[0028] When the user selects the montage image capture mode, the SLO control unit 180 performs alignment and focus adjustment. 5, the fixation light control unit 1802 controls the fixation target control device 90 to turn on the upper fixation light 92U in order to direct the patient's gaze diagonally upward. As a result, the patient's gaze is directed diagonally upward, that is, in a direction from the center of the eyeball toward the upper fixation light 92U, as shown in FIG. 6A. In addition to turning on the upper fixation light 92U, the operator of the ophthalmologic apparatus 110 may issue an instruction to direct the patient's gaze diagonally upward, such as "Please look up," to direct the gaze of the subject's eye diagonally upward. The diagonally upward direction is an example of the "first direction" of the technology of the present disclosure.

[0029] In step 304, the fundus is photographed with the patient's line of sight directed diagonally upward. Specifically, the SLO light source control unit 1804 causes the G light source 42 and the R light source 44 to generate G light and R light. The scanner control unit 1806 controls the X-direction scanning device 82 and the Y-direction scanning device 84. The G light and R light scanned in the X-direction and Y-direction are reflected by the fundus of the subject's eye.

[0030] G light, which has a wavelength corresponding to green, is reflected by the retina and therefore contains structural information about the retina. The G light reflected from the subject's eye 12 is detected by the G light detection element 72. The image processing unit 182 generates image data of a UWF upward-viewed fundus image G from the signal from the G light detection element 72. Similarly, R light reflected from the subject's eye 12 is detected by the R light detection element 74. The red laser light (R light) is reflected by the choroid, which is deeper than the retina, and therefore contains structural information about the choroid. The image processing unit 182 generates image data of a UWF upward-viewed fundus image R from the signal from the R light detection element 74. The image processing unit 182 generates image data of a UWF upward-viewed fundus image RG by combining the UWF upward-viewed fundus image and the UWF upward-viewed fundus image R at a predetermined mixing ratio. When there is no need to distinguish between the UWF up-view fundus image G, the UWF up-view fundus image R, and the UWF up-view fundus image RG, they will be referred to as UWF up-view fundus images. The UWF upward-viewing fundus image is an example of a "first direction fundus image" of the technology of the present disclosure.

[0031] Incidentally, the UWF frontal view fundus image of the fundus of the subject eye 12 obtained by photographing the fundus with the optical axis of the subject eye 12 aligned with the optical axis of the ophthalmologic system 100 captures the area between the upper position U0 and the lower position D0 of the fundus in the YZ plane shown in Figure 6A.

[0032] In each of the fundus images obtained by photographing the fundus in the upward viewing state, as shown in FIG. 6A, the patient's line of sight is directed obliquely upward, so an image of the area between the upper position U1 above the upper position U0 and the lower position D1 is present. Therefore, each of the fundus images obtained by photographing the fundus in the upward viewing state includes an image of the area between the upper position U1 above the upper position U0 and the lower position D1. There is an image of the region MU01 where there is no vein. Figure 7A shows a UWF upper fundus image of the right eye. In the UWF upper fundus image GU, the nasal (right side of the page) vortex vein 12V1 and the temporal (left side of the page) vortex vein 12V3, which are located near the equator on the upper side of the eyeball, are located above the UWF upper fundus image GU. Furthermore, the optic disc ONH and macula M, which are located in the center of the fundus, are located below the UWF upper fundus image GU.

[0033] In step 306, the fixation light control unit 1802 controls the fixation target control device 90 to turn on the lower fixation light 92D in order to direct the patient's gaze diagonally downward. As a result, the patient's gaze is directed diagonally downward, that is, from the center of the eyeball toward the lower fixation light 92D, as shown in Fig. 6B. In addition to turning on the upper fixation light 92D, the operator of the ophthalmologic apparatus 110 may issue an instruction to direct the patient's gaze diagonally downward, such as "Please look down," to direct the gaze of the subject's eye diagonally downward. The diagonally downward direction is an example of the "second direction" of the technology of the present disclosure.

[0034] In step 308, the fundus is photographed in a downward gaze state in which the patient's line of sight is directed diagonally downward. As in step 304, the SLO light source control unit 1804 generates G light and R light from the G light source 42 and the R light source 44, and the scanner control unit 1806 controls the X-direction scanning device 82 and the Y-direction scanning device 84 to scan the G light and R light in the X and Y directions. As in step 304, the image processing unit 182 generates image data of a UWF downward gaze fundus image G, a UWF downward gaze fundus image R, and a UWF downward gaze fundus image RG. When there is no need to distinguish between the UWF downward-viewing fundus image G, the UWF downward-viewing fundus image R, and the UWF downward-viewing fundus image RG, they will be referred to as UWF downward-viewing fundus images. The UWF downward-looking fundus image is an example of a "second direction fundus image" of the technology of the present disclosure.

[0035] In each downward-looking fundus image obtained by photographing the fundus while looking downward, as shown in Figure 6B, the patient's gaze is directed diagonally downward, so an image of the area between lower position D2, which is below lower position D0, and upper position U2 is included. Therefore, the downward-looking fundus image includes an image of area MD02, which is not included in the UWF front-view fundus image. Figure 7B shows a UWF downward-looking fundus image GD of the right eye. In the UWF downward-looking fundus image GD, the nasal side (right side of the page) vortex vein 12V2 and the temporal side (left side of the page) vortex vein 12V4, which are located near the equator on the upper side of the eyeball, are located below the UWF downward-looking fundus image GD. Furthermore, the optic disc ONH and macula M, which are located in the center of the fundus, are located above the UWF downward-looking fundus image GD.

[0036] In step 310, the processing unit 186 transmits image data of the UWF upward-looking fundus image and the UWF downward-looking fundus image to the server 140 via the communication interface (I / F) 26. When transmitting the image data to the server 140, the processing unit 186 also transmits the patient ID and patient attribute information (patient name, age, information on whether each fundus image is from the right eye or the left eye, visual acuity, photographing date and time, etc.) to the server 140.

[0037] The server 140 stores the received image data, patient ID, and patient attribute information in association with each other in the storage device 254, which will be described later.

[0038] The display control unit 184 may display a UWF upward-looking fundus image and a UWF downward-looking fundus image on the display 32.

[0039] When the user selects the normal imaging mode for capturing a wide area of ​​the fundus in a frontal view, the SLO control unit 180 performs alignment and focus adjustment. Then, the fixation light control unit 1802 controls the fixation target control device 90 to turn on the central fixation light. This fixes the patient's line of sight in the front, and a UWF frontal view fundus image as shown in FIG. 3B is captured. If a UWF frontal fundus image is acquired, in step 310, the processing unit 186 Image data of the F frontal fundus image is also transmitted in the same manner as above. As with the UWF upward-viewing fundus image and the UWF downward-viewing fundus image, UWF forward-viewing fundus image G, UWF forward-viewing fundus image R, and UWF forward-viewing fundus image RG are generated as UWF forward-viewing fundus images.

[0040] In order to analyze the structure of the peripheral part of the fundus (for example, vortex veins and choroidal vessels around the vortex veins), it is necessary to capture images of the area around the equator and the fundus region beyond the equator toward the anterior segment. When capturing images with a wide angle of view from the front, the subject's eyelids, eyelashes, or the housing of the ophthalmic device 110 may be captured in the image, preventing the peripheral part of the fundus from being captured. In this case, the vortex veins and choroidal vessels around the vortex veins cannot be captured, and an image including all of the peripheral part of the fundus or the vortex veins around the equator cannot be obtained.

[0041] Therefore, in this embodiment, a UWF upward-looking fundus image GU is acquired when the patient's gaze is directed upward, and a UWF downward-looking fundus image GD is acquired when the patient's gaze is directed downward. Then, by combining the two images, a montage image can be generated that reliably captures a wider area than a single UWF front-view fundus image. This montage image includes vortex veins and choroidal vessels around the vortex veins, thereby eliminating the influence of the subject's eyelids, eyelashes, the housing of the ophthalmic device 110, and other elements. This montage image is suitable for detecting lesions or abnormal areas around the fundus, or vortex veins and choroidal vessels around the vortex veins.

[0042] The following describes the case where a montage image is generated by the server 140.

[0043] The configuration of the server 140 will be described with reference to FIG. 8. As shown in FIG. 8, the server 140 includes a computer main unit 252. The computer main unit 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268. The input / output (I / O) port 268 is connected to a storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258. The storage device 254 is configured, for example, with a non-volatile memory. The input / output (I / O) port 268 is connected to the network 130 via the communication interface (I / F) 258. Therefore, the server 140 can communicate with the ophthalmic device 110, the axial length measuring device 120, and the viewer 150. The storage device 254 stores a montage image creation processing program, which will be described later. The montage image creation processing program may also be stored in the ROM 264. The montage image creation processing program is an example of the "image processing program" of the technology of the present disclosure.

[0044] When the CPU 262 of the server 140 executes the montage image creation processing program, the CPU 262 functions as an image acquisition unit 1410, an image processing unit 1420 (including an alignment unit 1421, a binarization processing unit 1422, a composite image generation unit 1424, and a vortex vein analysis unit 1425), a display control unit 1430, and an output unit 1440, as shown in FIG. 9. The image acquisition unit 1410 is an example of an "acquisition unit" of the technology of the present disclosure. The image processing unit 1420 is an example of a "generation unit" of the technology of the present disclosure. The output unit 1440 is an example of an "output unit" of the technology of the present disclosure.

[0045] Next, the montage image creation process by the CPU 262 of the server 140 will be described in detail with reference to Fig. 10. The montage image creation process shown in the flowchart of Fig. 10 is realized by the CPU 262 of the server 140 executing a montage image creation processing program. The montage image creation process is an example of the "image processing method" of the technology of the present disclosure.

[0046] A user (e.g., an ophthalmologist) instructs the viewer 150 to display a fundus image (montage image) of the patient's eye 12 for diagnosing the patient's eye 12 by turning on a montage image display button (not shown). At this time, the operator inputs a patient ID into the viewer 150. The viewer 150 outputs montage image creation instruction data together with the patient ID to the server 140. Having received the montage image creation instruction data and the patient ID, the server 140 executes a montage image creation processing program. The montage image creation processing program may be executed when the UWF upward-view fundus image and the UWF downward-view fundus image captured by the ophthalmologic apparatus 110 are transmitted to the server 140 .

[0047] In step 320 of the flowchart in Figure 10, the image acquisition unit 1410 acquires a UWF upper-view fundus image and a UWF lower-view fundus image from the storage device 254. In step 322, the binarization processing unit 1422 performs processing to emphasize blood vessels on the UWF upper-view fundus image and the UWF lower-view fundus image. Then, binarization processing is performed to binarize the images using a predetermined threshold. The binarization processing emphasizes the blood vessels in the fundus in white.

[0048] In step 324, the alignment unit 1421 aligns the UWF upward-looking fundus image and the UWF downward-looking fundus image. The alignment process in step 324 will be described using the flowchart in Fig. 11. Here, an example will be described in which the UWF downward-looking fundus image is converted using the UWF upward-looking fundus image as a reference (i.e., the UWF upward-looking fundus image is not converted, and only the UWF downward-looking fundus image is converted).

[0049] In step 340 of FIG. 11 , the alignment unit 1421 extracts a feature point group 1 from the UWF up-view fundus image GU by image processing. The feature point group 1 is a plurality of feature points in the fundus image, and as shown in FIG. 7A , it is the optic disc ONHU, the macula MU, and the branching points VBU of retinal blood vessels. Branching points of choroidal blood vessels may also be extracted as feature points. The alignment unit 1421 extracts structural information of only the choroid from the UWF up-view fundus image G, which includes structural information of the retina, and the UWF up-view fundus image R, which also includes structural information of the choroid, by removing the structural information of the retina from the UWF up-view fundus image R. The alignment unit 1421 extracts branching points of retinal blood vessels from the UWF up-view fundus image G, and extracts branching points of choroidal blood vessels from the structural information of only the choroid. The feature points are the pixel with the maximum brightness in the optic disc ONHU region, the pixel with the minimum brightness in the macular MU region, and pixels located at the branching points of retinal blood vessels and choroidal blood vessels, and the coordinates of these pixels are extracted as feature point data.In addition to the branching points of retinal blood vessels and choroidal blood vessels, areas containing characteristic vascular patterns may also be extracted, and the center points of the areas containing these patterns may be used as feature points. Note that the end points, bending points, or meandering points of retinal blood vessels or choroidal blood vessels may be extracted as feature points. The feature points are detected by SIFT (Scale Interpolation Fourier Transform). Variant Feature Transform) and SURF (Speed ​​Up This can be done using techniques such as ped Robust Feature.

[0050] Here, to perform alignment with high accuracy, it is preferable that the number of extracted feature points be four or more. In the UWF up-view fundus image GU, there is only one optic disc and one macula in the test eye. Therefore, by extracting two or more branching points VBU of the retinal blood vessels and choroidal blood vessels, four or more feature points can be extracted from the UWF up-view fundus image GU.

[0051] The optic disc, macula, retinal blood vessels, and choroidal blood vessels present in the center of the fundus are captured in both the UWF upward-looking fundus image GU and the UWF downward-looking fundus image GD, and are therefore suitable targets for selecting feature points for alignment. It is advisable to select from the central part of the fundus, which is the common area with the square fundus image GD. Therefore, in step 340, the alignment unit 1421 extracts a group of feature points 1 by image processing, targeting the area below the center of the UWF up-viewing fundus image GU, which is the center of the fundus.

[0052] The vortex veins 12V1 and 12V3 present in the UWF upward-view fundus image GU, which are present in the peripheral part of the fundus, are excluded from selection as feature points. Because the peripheral part of the fundus is not a common area between the UWF upward-view fundus image GU and the UWF downward-view fundus image GD, structures in the peripheral part of the fundus are excluded from selection as feature points.

[0053] In step 342, the alignment unit 1421 extracts feature point group 2 corresponding to feature point group 1 from the UWF downward-viewing fundus image GD. Feature point group 2 is the optic disc ONHD, macula MD, and retinal blood vessel branching point VBD, as shown in FIG. 7B. Since it is the same eye, the optic disc ONHD corresponds to the optic disc ONHU, and the macula MD corresponds to the macula MU. The branching point VBD corresponds to the blood vessel branching point VBU, and branching points having the same branching pattern as the branching pattern of the branching point VBU are extracted by image recognition processing or the like.

[0054] In step 344, the alignment unit 1421 generates a projective transformation matrix for geometrically transforming the UWF downward-viewing fundus image GD using the feature point group 1 and the feature point group 2. This projective transformation matrix is ​​a matrix for associating the UWF downward-viewing fundus image GD with the UWF upward-viewing fundus image GU. At least four feature points define a homography matrix. In step 346, the UWF downward-looking fundus image GD (see FIG. 7B) is transformed using the generated projective transformation matrix to obtain a transformed UWF downward-looking fundus image GDC (see FIG. 12B). After transformation using the projective transformation matrix, feature point group 1 and feature point group 2 are in the same position, which means that alignment processing has been performed. As a result of this transformation, the UWF downward-looking fundus image GDC is larger (its area is increased) than the UWF downward-looking fundus image GD. In the above description, a projective transformation matrix is ​​generated to associate the UWF downward-viewing fundus image GD with the UWF upward-viewing fundus image GU, and the UWF downward-viewing fundus image GD is transformed. Conversely, a projective transformation matrix may be generated to associate the UWF upward-viewing fundus image GU with the UWF downward-viewing fundus image GD, and the UWF upward-viewing fundus image GU may be transformed. This completes the image registration process at step 324 in FIG. 10, and the montage image creation process proceeds to step 326.

[0055] In step 326 of FIG. 10, the composite image generator 1424 composites the UWF upward-looking fundus image GU and the converted UWF downward-looking fundus image GDC to generate a montage image GM. First, as shown in Fig. 12A, a line segment LGU passing through the optic disc ONHU and the macula MU is set on the UWF up-view fundus image GU. Similarly, as shown in Fig. 12B, a line segment LGD passing through the optic disc ONHD and the macula MD is set on the converted UWF down-view fundus image GDC.

[0056] Next, the composite image generation unit 1424 performs weighting processing on the overlapping region of the UWF up-viewing fundus image GU and the UWF down-viewing fundus image GDC. As shown in Fig. 13, the composite image generation unit 1424 sets a weight of "1" for the upper UWF up-viewing fundus image GUx region, which is the region above the line segment LGU of the UWF up-viewing fundus image GU. The composite image generation unit 1424 sets a weight of "0" for the region above the line segment LGU. Then, the composite image generation unit 1424 sets a weight of "1" for the lower UWF down-viewing fundus image GDCx, which is the region below the line segment LGD of the converted UWF down-viewing fundus image, and sets a weight of "0" for the region above the line segment LGD.

[0057] The composite image generating unit 1424 performs such weighting processing on the UWF upward-viewing fundus image GU and the UWF downward-viewing fundus image GDC to generate a montage image GM in which the UWF upward-viewing fundus image GUx and the UWF downward-viewing fundus image GDCx are combined. As shown, the line segment LG is a line segment connecting the optic disc ONHD and the macula M, and the area above the line segment LG is the UWF upward-looking fundus image GUx, and the area below the line segment LG is the UWF downward-looking fundus image GDCx. The montage image is an example of a "synthetic image" of the technology of the present disclosure.

[0058] Note that the weighting of the overlapping portion of the UWF upward-viewing fundus image GU and the UWF downward-viewing fundus image GDC is not limited to the above example, and the mixing ratio of the UWF upward-viewing fundus image GU and the UWF downward-viewing fundus image GDC can be set to various values.

[0059] In this way, the UWF upward-looking fundus image GU and the UWF downward-looking fundus image GDC are aligned and combined, and this combination provides a fundus image in which the blood vessels of the fundus are not discontinuous, making it possible to analyze vortex veins located in the peripheral or equatorial parts of the fundus, and choroidal blood vessels around the vortex veins, or to analyze abnormalities or lesions.

[0060] Next, in step 328, the vortex vein analysis unit 1425 analyzes the positions of the vortex veins and the diameters of the blood vessels near the vortex veins using the montage image GM. The vortex vein information obtained by the analysis includes information on the number of vortex veins, the positions of the vortex veins, the number of blood vessels connected to the vortex veins, the diameters of the blood vessels around the vortex veins, etc.

[0061] In step 330, the display control unit 1430 generates a display screen 400 (described later) that reflects the montage image as well as patient attribute information corresponding to the patient ID (patient name, age, whether each fundus image is from the right eye or left eye, visual acuity, date and time of shooting, etc.). In step 332, the output unit 1440 outputs the montage image GM and the vortex vein analysis information obtained by the vortex vein analysis to the storage device 254 of the server 140. The montage image GM and the vortex vein analysis information obtained by the vortex vein analysis are stored in the storage device 254 of the server 140. Furthermore, in step 332 , the output unit 1440 outputs image data corresponding to the display screen 400 to the viewer 150 . The display control unit 1430 may output the montage image GM to the display 256 so that it is displayed.

[0062] The flowcharts in Figures 10 and 11 explain the case where a UWF downward-looking fundus image is converted using a UWF upward-looking fundus image as a reference, but this is not limited to this, and a UWF upward-looking fundus image may also be converted using a UWF downward-looking fundus image as a reference. Although the montage image was generated using the binarized image, it is also possible to generate a montage image using the pre-binarized color UWF upward-looking fundus image RG and the UWF downward-looking fundus image RG in a similar manner. In this case, the montage image is binarized after the montage synthesis process.

[0063] The graphic user interface (GUI) using montage images will be described below. As described above, in step 332 of FIG. 10, the server 140 outputs image data corresponding to the display screen 400 to the viewer 150. The viewer 150 receives the image data output from the server 140 and displays the display screen 400 on a display (monitor) not shown. As shown in Fig. 14, the display screen 400 has an information display area 402 and an image display area 404 (404A). Fig. 14 shows the image display area 404A in the first display mode of the image display area 404. The information display area 402 has a patient ID display area 412, a patient name display area 414, an age display area 416, a right eye / left eye display area 418, an axial length display area 420, a visual acuity display area 422, and a photography date and time display area 424. The viewer 150 selects the patient ID display area 412, the photography date and time display area 424, and the like based on the received information. Each display area displays different information.

[0064] The information display area 402 is provided with an image selection icon 430 and a display switching icon 440 .

[0065] The image display area 404A has a montage image display area 450 and a related image display area 460. When the image selection icon 430 is turned on, a pull-down menu is displayed. The pull-down menu displayed when the image selection icon 430 is turned on has a menu for selecting a related image to be displayed in the related image display area 460. For example, the pull-down menu may include a menu for selecting a fluorescent image of the fundus of the subject's eye 12 that has already been acquired (for example, an IA image The following are displayed as selection options: a video of an IA image (an image obtained by indocyanine green angiography), a still image of an IA image, a UWF frontal fundus image, etc. Fig. 14 shows that a montage image GM is displayed in the montage image display area 450, and a video of an IA image GA is displayed in the related image display area 460. Furthermore, the line segment LG is a line segment connecting the optic disc ONH and macula M of the montage image GM. The moving image GA of the IA image displayed in the related image display area 460 is displayed aligned so that the optic disc and macula of the moving image GA are on the line segment LG. The user may be able to select whether or not to display the line segment LG.

[0066] Furthermore, marks indicating vortex veins may be displayed in the montage image and related images at positions corresponding to the vortex veins.

[0067] For example, as shown in FIG. 14, the positions of vortex veins 12V1, 12V2, 12V3, and 12V4 may be detected in a montage image GM, and marks indicating the vortex veins may be displayed at the detected positions in the montage image GM. Furthermore, vortex vein analysis information (such as the number of vortex veins, the positions of vortex veins, the number of blood vessels connected to the vortex veins, and the diameters of blood vessels around the vortex veins) may be displayed on the display screen 400.

[0068] When display switching icon 440 is turned on, a pull-down menu is displayed. The pull-down menu has a menu of images to be displayed in image display area 404. Specifically, the pull-down menu has a menu for displaying a montage image and an IA image side by side, a menu for dividing and combining the montage image and the IA image, and a menu for displaying a montage image side by side with a three-dimensional image obtained by projecting the montage image onto a three-dimensional model using a three-dimensional model. Fig. 14 shows the state when the menu for displaying a montage image and an IA image side by side is selected.

[0069] 15 shows a state in which a menu for dividing, combining, and displaying a montage image and an IA image is selected from the pull-down menu of the display switching icon 440. As shown in FIG. 15, in the image display area 404B, with the dividing line LK as the reference, the IA image (moving image or still image) GA is displayed above the dividing line LK, and the montage image GM is displayed below the dividing line LK. By moving the icon I up or down along the dividing line LK, the display area of ​​the IA image GA and the montage image GM changes. For example, when the icon I moves upward, the dividing line LK also moves upward, and only the portion of the IA image GA above the divided line LK that has been moved upward is displayed, and the portion of the montage image GM between the original position of the dividing line LK and the current position of the dividing line LK is also displayed.

[0070] 16 also shows a state in which a menu for displaying a montage image and a three-dimensional image side by side is selected from the pull-down menu of the display switching icon 440. As shown in FIG. 16, the image display area 404C includes a montage image display area 450. and a 3D image display area 470 that displays a 3D image obtained by projecting the montage image onto a 3D model of the eyeball. Before projecting the montage image onto the 3D model of the subject's eye, the 3D model of the eyeball may be corrected based on data on the patient's axial length received from the server 140. It is also possible to add a menu that displays everything to the pull-down menu of the display switching icon 440, and display two or more of the image display areas 404A to 404C of Figures 14 to 16 simultaneously or sequentially on one screen.

[0071] In place of the binarized montage image GM, the three-dimensional model may display a montage image obtained by combining a UWF upward-looking fundus image and a UWF downward-looking fundus image before binarization.

[0072] As described above, in this embodiment, a montage image is generated by combining a UWF upward-view fundus image and a UWF downward-view fundus image. This allows ophthalmologists to more accurately diagnose the patient's eye using a montage image that captures the fundus equator and peripheral areas. In particular, a montage image that does not include reflections of eyelids, eyelashes, or equipment can be generated. This allows ophthalmologists to grasp the condition of the upper and lower vortex veins. Furthermore, it is possible to diagnose whether or not there is a lesion in the fundus equator or peripheral areas. Furthermore, it is possible to estimate not only lesions in the fundus equator or peripheral areas, but also signs of lesions in the center of the fundus from the condition of the upper and lower vortex veins.

[0073] Next, various modifications will be described.

[0074] (First Modification) In the above embodiment, the montage image creation process is executed by the CPU 262 of the server 140, but the technology of the present disclosure is not limited to this. For example, the process may be executed by the CPU 22 of the ophthalmologic apparatus 110, the CPU of the viewer 150, or even the CPU of another computer connected to the network 130.

[0075] (Second Modification) In the above embodiment, the UWF upward-looking fundus image and the UWF downward-looking fundus image are acquired as follows: That is, the upper fixation light 92U is turned on, and the patient's gaze is directed upward, and a UWF upward-looking fundus image GU of the subject's eye 12 is acquired; and the lower fixation light 92D is turned on, and the patient's gaze is directed downward, and a UWF downward-looking fundus image GD of the subject's eye 12 is acquired. The technology of the present disclosure is not limited to this. For example, the SLO unit 40 is configured to be rotatable in the vertical direction around the center of the pupil of the subject's eye 12.

[0076] When acquiring an upward image, the SLO unit 40 is rotated downward around the center of the pupil of the subject's eye 12. In this state, the SLO unit 40 acquires an image of the fundus from the bottom to the top through the pupil of the subject's eye 12. In this case, the upper fixation light 92U may also be turned on to direct the patient's gaze upward.

[0077] When acquiring a lower image, the SLO unit 40 is rotated upward around the center of the pupil of the subject's eye 12. In this state, the SLO unit 40 acquires an image of the fundus from the top to the bottom through the pupil of the subject's eye 12. In this case, the lower fixation light 92D may also be turned on to direct the patient's gaze downward.

[0078] Furthermore, the present invention is not limited to directing the patient's line of sight in the vertical direction or rotating the SLO unit 40 in the vertical direction around the center of the pupil of the subject's eye 12.

[0079] For example, fixation lamps may be provided at the upper right, upper left, lower right, and lower left sides to guide the line of sight of the subject's eye 12, fundus images may be acquired at each line of sight, and these may be combined to generate a montage image.

[0080] Furthermore, the SLO unit 40 is configured to be rotatable around the center of the pupil of the subject's eye 12 so that the fundus can be photographed through the pupil of the subject's eye 12 from the upper right, upper left, lower right, and lower left sides. By photographing the fundus from each direction with the SLO unit 40, UWF fundus images can be obtained in each direction, and these images can be combined to generate a montage image. In this case, fixation lights arranged at the upper right, upper left, lower right, and lower left sides may be turned on to guide the line of sight of the subject's eye 12 in each direction.

[0081] (Third Modification) The montage image generated by the server 140 may be used not only to visualize vortex veins but also to perform structural analysis of the retina in the peripheral part of the fundus (equatorial part of the fundus), blood vessel analysis, or processing to detect abnormal areas (lesions). The generated montage image may also be used to estimate lesions (or the likelihood of developing them) occurring in the central part of the fundus, such as diabetic retinopathy and age-related macular degeneration. By adding image information about the peripheral part of the fundus (equatorial part of the fundus) to the montage image, lesions can be estimated by taking into account information about the peripheral part of the fundus, rather than by image analysis using an image of the central part of the fundus. Furthermore, structural analysis, blood vessel analysis, and lesion estimation may be performed using AI (artificial intelligence). The retinal structural analysis, blood vessel analysis, or processing to detect abnormal areas (lesions) in the peripheral part of the fundus (fundus equator) may be performed by the vortex vein analysis unit 1425, or may be performed by a separate image analysis unit (not shown).

[0082] (Fourth Modification) In the above embodiment, the montage image creation process is executed by the CPU 262 of the server 140, and the alignment process is performed automatically. When the montage image creation process is performed by the viewer 150, the extraction of feature point group 1 of the UWF upward-viewing fundus image and the extraction of feature point group 2 of the UWF downward-viewing fundus image may be performed manually by a user. Specifically, instead of steps 340 and 342 in FIG. 11, the following processing is executed. The server 140 transmits image data of the UWF upward-looking fundus image GU and the UWF downward-looking fundus image GD to the viewer 150.

[0083] The viewer 150, which has received the image data, detects the macula MU and MD and the optic disc ONHU and ONHD in each of the UWF up-view fundus image GU and the UWF down-view fundus image GD. The viewer 150 aligns and displays the UWF up-view fundus image GU and the UWF down-view fundus image GD side by side so that the line segment connecting the macula MU and the optic disc ONHU in the UWF up-view fundus image GU and the line segment connecting the macula MD and the optic disc ONHD in the UWF down-view fundus image GD coincide as the line segment LGM, as shown in Fig. 17. The viewer 150 sets an upper limit LU and a lower limit LD of a feature point extraction region parallel to the line segment LGM at positions above and below the line segment LGM at a predetermined distance.

[0084] The user sets feature point group 1 from the UWF upward-looking fundus image GU between the upper limit LU and the lower limit LD. Then, the user extracts feature point group 2 corresponding to feature point group 1 from the UWF downward-looking fundus image GD between the upper limit LU and the lower limit LD. The viewer 150 transmits each piece of data on the positions of feature point group 1 and feature point group 2 to the server 140. In step 344 (FIG. 11), the alignment unit 1421 of the server 140 receives each piece of data on the positions of feature point group 1 and feature point group 2 and creates the above-mentioned projective transformation matrix using the received feature point group 1 and feature point group 2.

[0085] (Other variations) The montage image generation process and montage image display process described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention. The ophthalmologic apparatus 110 has a function of capturing an image of an area with an internal illumination angle of 200° (167° in terms of an external illumination angle based on the pupil of the eyeball of the eye 12) with the center 170 of the eyeball of the eye 12 as the reference position, but is not limited to this angle of view. The internal illumination angle may be 200° or more (external illumination angle may be 167° or more and 180° or less). Furthermore, the specifications may also be such that the internal illumination angle is less than 200 degrees (external illumination angle is less than 167 degrees). For example, the angle of view may be such that the internal illumination angle is approximately 180 degrees (external illumination angle is approximately 140 degrees), the internal illumination angle is approximately 156 degrees (external illumination angle is approximately 120 degrees), or the internal illumination angle is approximately 144 degrees (external illumination angle is approximately 110 degrees). The values ​​are merely examples, and any angle of view may be used as long as it allows simultaneous imaging of the peripheral part of the fundus where vortex veins and the like are present, and the central part of the fundus.

[0086] In the above-described examples, the processing is realized by a software configuration using a computer, but the technology of the present disclosure is not limited to this. For example, instead of a software configuration using a computer, the processing may be performed only by a hardware configuration such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Some of the processing may be performed by a software configuration, and the remaining processing may be performed by a hardware configuration.

Claims

[Claim 1] Obtaining a first direction fundus image captured with the line of sight of the subject's eye directed in a first direction, and a second direction fundus image captured with the subject's eye directed in a second direction different from the first direction; converting one of the first direction fundus image and the second direction fundus image to generate a converted image; synthesizing the converted image with the other fundus image of the first direction fundus image and the second direction fundus image to generate a synthesized image; outputting the composite image; Including, In generating the transformed image, extracting at least four feature points on the fundus of the subject's eye from the first direction fundus image and the second direction fundus image, respectively; performing projective transformation on the one fundus image to generate the transformed image in which the positions of the four feature points on the other fundus image coincide with the positions of the four feature points on the one fundus image, In generating the composite image, applying a first weight to a first region in the converted image where the converted image and the other fundus image overlap; applying a second weight to a second region in the other fundus image where the converted image and the other fundus image overlap; and combining the transformed image and the other fundus image based on the first weighting and the second weighting. Image processing methods.

Images