Image processing device, image processing method, and image processing program

Abstract

According to the present disclosure, a deviation area detected in one ophthalmic image is accurately aligned and displayed in another ophthalmic image. This image processing device comprises: an image acquisition unit that acquires a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye to be examined as the first ophthalmic image and is captured at a different time and / or by a different device from the first ophthalmic image; a deviation area information acquisition unit that acquires information regarding a deviation area deviating from a predetermined state in the first ophthalmic image; a displacement information acquisition unit that acquires displacement information indicating a change between the first ophthalmic image and the second ophthalmic image, by comparing the first ophthalmic image and the second ophthalmic image; and a deviation area position identification unit that aligns the second ophthalmic image and a deviation area image indicating the position of the deviation area on the basis of the displacement information, and identifies the position of the deviation area in the second ophthalmic image. The displacement information acquisition unit acquires the displacement information by excluding the deviation area in the first ophthalmic image from the object from which the displacement information is to be acquired.

Description

Image processing apparatus, image processing method, and image processing program

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program.

[0002] Patent Document 1 discloses a technique for generating volume data of an eye to be examined using an optical coherence tomography device. Conventionally, it has been desired to visualize blood vessels based on the volume data of the eye to be examined.

[0003] U.S. Patent No. 10,238,281

[0004] In a first aspect, an image processing apparatus includes an image acquisition unit that acquires a first ophthalmic image and a second ophthalmic image of the same eye to be examined as the first ophthalmic image, the second ophthalmic image being captured at a different time and / or by a different device from the first ophthalmic image; a deviation area information acquisition unit that acquires information on a deviation area that deviates from a predetermined state in the first ophthalmic image; a displacement information acquisition unit that compares the first ophthalmic image and the second ophthalmic image and acquires displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and a deviation area position specifying unit that aligns the second ophthalmic image and a deviation area image indicating the position of the deviation area based on the displacement information and specifies the position of the deviation area in the second ophthalmic image. The displacement information acquisition unit acquires the displacement information by excluding the deviation area in the first ophthalmic image from a target for acquiring the displacement information.

[0005] The second embodiment is an image processing method comprising the steps of: acquiring a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye as the first ophthalmic image and was taken at a different time and / or with a different device than the first ophthalmic image; acquiring information on a deviation area in the first ophthalmic image that deviates from a predetermined state; comparing the first ophthalmic image and the second ophthalmic image to acquire displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and aligning the second ophthalmic image with a deviation area image indicating the position of the deviation area based on the displacement information to identify the position of the deviation area in the second ophthalmic image. When acquiring the displacement information, the deviation area in the first ophthalmic image is excluded from the target for acquiring the displacement information.

[0006] The third embodiment is an image processing program configured to be executed by a computer, comprising the steps of: acquiring a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye as the first ophthalmic image and was taken at a different time and / or with a different device than the first ophthalmic image; acquiring information on a deviation area in the first ophthalmic image that deviates from a predetermined state; comparing the first ophthalmic image and the second ophthalmic image to acquire displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and aligning the second ophthalmic image with a deviation area image indicating the location of the deviation area to identify the location of the deviation area in the second ophthalmic image. When acquiring the displacement information, the deviation area in the first ophthalmic image is excluded from the target for acquiring the displacement information.

[0007] This is a schematic diagram of the ophthalmic system according to the first embodiment. This is a schematic diagram of the ophthalmic device according to the first embodiment. This is a schematic diagram of the server. This is an explanatory diagram of the functions realized by the image processing program on the server's CPU. This is a flowchart that explains in detail the image processing (overlay display processing of deviation areas) by the management server 140 of the ophthalmic system according to the first embodiment. This is a flowchart that explains in detail the image processing (overlay display processing of deviation areas) by the management server 140 of the ophthalmic system according to the first embodiment. This is a schematic diagram illustrating the outline of image processing in the ophthalmic system of the first embodiment. This is a flowchart that explains in detail the image processing (overlay display processing of deviation areas) by the management server 140 of the ophthalmic system according to the second embodiment. This is a flowchart that explains in detail the image processing (overlay display processing of deviation areas) by the management server 140 of the ophthalmic system according to the second embodiment.

[0008] The embodiments for realizing the disclosed technology will be described in detail below with reference to the drawings. Components and processes that perform the same function or operation will be given the same reference numerals throughout the drawings, and redundant explanations may be omitted as appropriate. Furthermore, explanations of configurations not directly related to the disclosed technology or well-known configurations may also be omitted. Additionally, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios. Moreover, each drawing is only a schematic representation to the extent necessary to fully understand the disclosed technology. Therefore, the disclosed technology is not limited to the illustrated examples.

[0009] The disclosed technology is applicable to any ophthalmic device that acquires images related to the eye, such as fundus images. In this embodiment, to simplify the following explanation, we will describe an example of an ophthalmic device having the function of acquiring images, in which the technology is applied to an ophthalmic device used by an observer, such as a physician, to observe the eye of a patient (hereinafter referred to as the "examined eye"), for example, the fundus and the area around the examined eye, such as the anterior segment, for the purpose of diagnosing the eye and performing surgical procedures on the eye in ophthalmology.

[0010] <First Embodiment> Figure 1 shows a schematic configuration of the ophthalmology system 100. As shown in Figure 1, the ophthalmology system 100 comprises an ophthalmology device 110, a server device (hereinafter referred to as "server") 140, and a display device (hereinafter referred to as "viewer") 150. The ophthalmology device 110 acquires fundus images. The server 140 stores multiple fundus images obtained by capturing the funduses of multiple patients by the ophthalmology device 110, and the axial length measured by an axial length measuring device (not shown), corresponding to the patient ID. The viewer 150 displays the fundus images and analysis results acquired by the server 140.

[0011] Server 140 is an example of the "image processing device" described herein.

[0012] The ophthalmic device 110, server 140, and viewer 150 are interconnected via a network 130. The network 130 can be any network, such as a LAN, WAN, the Internet, or a wide-area Ethernet network. For example, if the ophthalmic system 100 is built in a single hospital, a LAN can be used for the network 130.

[0013] The viewer 150 is a client in a client-server system, and multiple units are connected via the network. The server 140 may also be connected via the network in multiple units to ensure system redundancy. Alternatively, if the ophthalmic device 110 has image processing capabilities and the viewer 150 has image viewing capabilities, the ophthalmic device 110 can acquire, process, and view fundus images in a standalone state. Furthermore, if the server 140 has image viewing capabilities, the configuration of the ophthalmic device 110 and the server 140 enables the acquisition, processing, and viewing of fundus images.

[0014] Furthermore, other ophthalmic devices (such as visual field measurement and intraocular pressure measurement equipment) and diagnostic support devices that perform image analysis using AI (Artificial Intelligence) may be connected to the ophthalmic device 110, server 140, and viewer 150 via the network 130.

[0015] Next, the configuration of the ophthalmic device 110 will be explained with reference to Figure 2.

[0016] For the sake of explanation, a scanning laser ophthalmoscope will be referred to as "SLO," and an optical coherence tomography (OCT) will be referred to as "OCT."

[0017] When the ophthalmic device 110 is installed on a horizontal plane, the horizontal direction is defined as the "X direction," the direction perpendicular to the horizontal plane is defined as the "Y direction," and the direction connecting the center of the pupil of the anterior segment of the eye under examination 12 to the center of the eyeball is defined as the "Z direction." Therefore, the X, Y, and Z directions are perpendicular to each other.

[0018] The ophthalmic apparatus 110 includes an imaging device 14 and a control device 16. The imaging device 14 is equipped with an SLO unit 18 and an OCT unit 20, and acquires fundus images of the eye under examination 12. Hereinafter, the two-dimensional fundus image acquired by the SLO unit 18 will be referred to as an SLO image. Also, cross-sectional images and frontal images (en-face images) of the retina created based on OCT data acquired by the OCT unit 20 will be referred to as OCT images.

[0019] The OCT data includes A-scan data, which is OCT data obtained by scanning a single point in the fundus in the depth (optical axis) direction (hereinafter referred to as A-scan) when the tomographic image is acquired at a single point. It also includes B-scan data, which is OCT data obtained by scanning by performing multiple A-scans while moving along the line (hereinafter referred to as B-scan) when the tomographic image is acquired at a line. Note that B-scan data may be generated by interpolating multiple A-scan data. Furthermore, it includes C-scan data, which is OCT data obtained by scanning by repeatedly performing B-scans while moving along the surface (hereinafter referred to as C-scan) when the tomographic image is acquired at a surface. The C-scan data is 3D OCT data and is generated as OCT volume data, making it possible to generate 2D en-face images etc. based on the 3D OCT data. Note that C-scan data may be generated by interpolating multiple B-scan data.

[0020] The control device 16 includes a computer comprising a CPU (Central Processing Unit) 16A, RAM (Random Access Memory) 16B, ROM (Read-Only Memory) 16C, and input / output (I / O) ports 16D.

[0021] The control device 16 includes an input / display device 16E connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphic user interface that displays an image of the eye under examination 12 and accepts various instructions from the user. A touch panel display is an example of a graphic user interface.

[0022] Furthermore, the control device 16 includes an image processor 17 connected to the I / O port 16D. The image processor 17 generates an image of the eye under examination 12 based on the data obtained by the imaging device 14. The control device 16 is connected to the network 130 via a communication interface (I / F) 16F.

[0023] As described above, in Figure 2, the control device 16 of the ophthalmic device 110 is equipped with an input / display device 16E, but the disclosure is not limited thereto. For example, the control device 16 of the ophthalmic device 110 may not be equipped with an input / display device 16E, but may be equipped with a separate input / display device that is physically independent of the ophthalmic device 110. In this case, the display device includes an image processing processor unit that operates under the control of the display control unit 204 of the CPU 16A of the control device 16. The image processing processor unit may display an SLO image or the like based on an image signal that the display control unit 204 has instructed to output.

[0024] The imaging device 14 operates under the control of the CPU 16A of the control device 16. The imaging device 14 includes an SLO unit 18, an imaging optical system 19, and an OCT unit 20. The imaging optical system 19 includes an optical scanner 22 and a wide-angle optical system 30.

[0025] The optical scanner 22 scans the light emitted from the SLO unit 18 in two dimensions, in the X and Y directions. The optical scanner 22 can be any optical element capable of deflecting the light beam, such as a polygon mirror or a galvanometer mirror. A combination of these may also be used.

[0026] The wide-angle optical system 30 combines light from the SLO unit 18 and light from the OCT unit 20.

[0027] The wide-angle optical system 30 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a reflective-refractive optical system combining a concave mirror and a lens. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph not only the central part of the fundus but also the peripheral part of the fundus.

[0028] When using a system that includes an elliptical mirror, a configuration using an elliptical mirror as described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 is also acceptable. Each of the disclosures in International Publication WO2016 / 103484 and International Publication WO2016 / 103489 is incorporated herein by reference in its entirety.

[0029] The wide-angle optical system 30 enables observation of the fundus of the eye in a wide field of view (FOV) 12A. The FOV 12A indicates the range that can be captured by the imaging device 14. The FOV 12A can be expressed as the field of view angle. In this embodiment, the field of view angle can be defined by the internal illumination angle and the external illumination angle. The external illumination angle is the illumination angle of the light beam irradiated from the ophthalmic device 110 onto the eye under examination 12, defined with respect to the pupil 27. The internal illumination angle is the illumination angle of the light beam irradiated onto the fundus of the eye, defined with respect to the center O of the eyeball. The external illumination angle and the internal illumination angle are in a corresponding relationship. For example, if the external illumination angle is 120 degrees, the internal illumination angle corresponds to approximately 160 degrees. In this embodiment, the internal illumination angle is set to 200 degrees.

[0030] Here, SLO fundus images obtained by imaging with an internal illumination angle of 160 degrees or more are referred to as UWF-SLO fundus images. UWF stands for UltraWide Field. The wide-angle optical system 30, which sets the field of view (FOV) of the fundus to an ultra-wide angle, can image the region of the fundus of the eye under examination 12 from the posterior pole to beyond the equator, and can image structures present in the peripheral part of the fundus, such as vortex veins.

[0031] The ophthalmic device 110 can image a region 12A with an internal illumination angle of 200°, using the center O of the eyeball of the eye being examined 12 as the reference position. Note that an internal illumination angle of 200° corresponds to an external illumination angle of 110°, with the pupil of the eyeball of the eye being examined 12 as the reference point. In other words, the wide-angle optical system 30 emits laser light from the pupil with an external illumination angle of 110° and images the fundus region with an internal illumination angle of 200°.

[0032] The SLO system is implemented by the control device 16, SLO unit 18, and imaging optical system 19 shown in Figure 2. The SLO system is equipped with a wide-angle optical system 30, enabling fundus imaging with a wide FOV 12A.

[0033] The SLO unit 18 includes a light source 40 for B light (blue light), a light source 42 for G light (green light), a light source 44 for R light (red light), and a light source 46 for IR light (infrared light (e.g., near-infrared light)), and optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light from the light sources 40, 42, 44, and 46 and guide it into a single optical path. Optical systems 48 and 56 are mirrors, and optical systems 50, 52, and 54 are beam splitters. The B light is reflected by optical system 48, transmitted through optical system 50, and reflected by optical system 54; the G light is reflected by optical systems 50 and 54; the R light is transmitted through optical systems 52 and 54; and the IR light is reflected by optical systems 52 and 56, and each is guided into a single optical path.

[0034] The SLO unit 18 is configured to allow switching between combinations of light sources that emit or emit laser light of different wavelengths, such as a mode that emits R-light and G-light, and a mode that emits infrared light. In the example shown in Figure 2, there are four light sources: a B-light light source 40, a G-light light source 42, an R-light light source 44, and an IR-light light source 46, but the disclosure is not limited thereto. For example, the SLO unit 18 may further include a white light light source and emit light in various modes, such as a mode that emits G-light, R-light, and B-light, or a mode that emits only white light.

[0035] Light incident from the SLO unit 18 into the imaging optical system 19 is scanned in the X and Y directions by the optical scanner 22. The scanned light passes through the wide-angle optical system 30 and the pupil 27 and illuminates the fundus of the eye. The reflected light reflected by the fundus of the eye passes through the wide-angle optical system 30 and the optical scanner 22 and is incident on the SLO unit 18.

[0036] The SLO unit 18 includes a beam splitter 64 that reflects B light and transmits other light from the posterior segment (fundus) of the eye under examination 12, and a beam splitter 58 that reflects G light and transmits other light from the light transmitted through the beam splitter 64. The SLO unit 18 also includes a beam splitter 60 that reflects R light and transmits other light from the light transmitted through the beam splitter 58. The SLO unit 18 also includes a beam splitter 62 that reflects IR light from the light transmitted through the beam splitter 60. The SLO unit 18 includes a B light detection element 70 that detects B light reflected by the beam splitter 64, a G light detection element 72 that detects G light reflected by the beam splitter 58, an R light detection element 74 that detects R light reflected by the beam splitter 60, and an IR light detection element 76 that detects IR light reflected by the beam splitter 62.

[0037] Light incident on the SLO unit 18 via the wide-angle optical system 30 and optical scanner 22 (reflected light reflected by the fundus of the eye) is reflected by the beam splitter 64 and received by the B-light detection element 70 in the case of B-light, and reflected by the beam splitter 58 and received by the G-light detection element 72 in the case of G-light. In the case of R-light, the incident light passes through the beam splitter 58, is reflected by the beam splitter 60 and received by the R-light detection element 74. In the case of IR-light, the incident light passes through the beam splitters 58 and 60, is reflected by the beam splitter 62 and received by the IR-light detection element 76. The image processor 17, operating under the control of the CPU 16A, generates a UWF-SLO image using the signals detected by the B-light detection element 70, the G-light detection element 72, the R-light detection element 74, and the IR-light detection element 76.

[0038] A UWF-SLO image generated using the signal detected by the B-light detection element 70 is called a B-UWF-SLO image (B-color fundus image). A UWF-SLO image generated using the signal detected by the G-light detection element 72 is called a G-UWF-SLO image (G-color fundus image). A UWF-SLO image generated using the signal detected by the R-light detection element 74 is called a R-UWF-SLO image (R-color fundus image). A UWF-SLO image generated using the signal detected by the IR-light detection element 76 is called an IR-UWF-SLO image (IR fundus image). UWF-SLO images include these R-color fundus images, G-color fundus images, B-color fundus images, and IR fundus images. Fluorescence UWF-SLO images, obtained by capturing fluorescence, are also included.

[0039] Furthermore, the control device 16 controls the light sources 40, 42, and 44 to emit light simultaneously. By simultaneously photographing the fundus of the eye under examination 12 with B light, G light, and R light, G-color fundus images, R-color fundus images, and B-color fundus images are obtained where each position corresponds to the others. An RGB color fundus image is obtained from the G-color fundus images, R-color fundus images, and B-color fundus images. The control device 16 controls the light sources 42 and 44 to emit light simultaneously, and by simultaneously photographing the fundus of the eye under examination 12 with G light and R light, G-color fundus images and R-color fundus images are obtained where each position corresponds to the others. An RG color fundus image is obtained from the G-color fundus images and R-color fundus images. Alternatively, a full-color fundus image may be generated using the G-color fundus image, R-color fundus image, and B-color fundus image.

[0040] The wide-angle optical system 30 provides an ultra-wide field of view (FOV) of the fundus, allowing imaging of the region from the posterior pole to beyond the equator of the fundus of the eye under examination 12.

[0041] The OCT system is realized by the control device 16, OCT unit 20, and imaging optical system 19 shown in Figure 2. Because the OCT system is equipped with a wide-angle optical system 30, it enables OCT imaging of the peripheral part of the fundus, similar to the acquisition of SLO fundus images described above. In other words, the wide-angle optical system 30, which sets the field of view (FOV) of the fundus to an ultra-wide angle, enables OCT imaging of the area from the posterior pole to beyond the equator 178 of the fundus of the eye under examination 12. OCT data of structures present in the peripheral part of the fundus, such as vortex veins, can be acquired, and tomographic images of vortex veins and the 3D structure of vortex veins can be obtained by image processing the OCT data.

[0042] The OCT unit 20 includes a light source 20A, a sensor (detection element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0043] The light emitted from the light source 20A is branched by the first optical coupler 20C. One of the branched lights is made into parallel light by the collimating lens 20E as measurement light and then incident on the imaging optical system 19. The measurement light is irradiated onto the fundus via the wide-angle optical system 30 and the pupil 27. The measurement light reflected by the fundus is incident on the OCT unit 20 via the wide-angle optical system 30 and, via the collimating lens 20E and the first optical coupler 20C, is incident on the second optical coupler 20F.

[0044] The other light emitted from the light source 20A and branched by the first optical coupler 20C is incident on the illumination optical system 20D as reference light and, via the reference optical system 20D, is incident on the second optical coupler 20F.

[0045] These lights incident on the second optical coupler 20F, that is, the measurement light reflected by the fundus and the reference light, are interfered by the second optical coupler 20F to generate interference light. The interference light is received by the sensor 20B. The image processor 17 operating under the control of the image processing unit 206 generates OCT data detected by the sensor 20B. It is also possible to generate OCT images such as tomographic images and en-face images by the image processor 17 based on the OCT data.

[0046] Here, the OCT unit 20 can scan a predetermined range (for example, a rectangular range of 6 mm × 6 mm) in one OCT imaging. The predetermined range is not limited to 6 mm × 6 mm, and may be a square range of 12 mm × 12 mm or 23 mm × 23 mm, or a rectangular range such as 14 mm × 9 mm or 6 mm × 3.5 mm, and can be any rectangular range. Also, it may be a range of a circular diameter such as 6 mm, 12 mm, or 23 mm.

[0047] By using the wide-angle optical system 30, the ophthalmic device 110 can scan the region 12A with an internal irradiation angle of 200°. That is, by controlling the optical scanner 22, OCT imaging of a predetermined range including the vortex vein is performed. The ophthalmic device 110 can generate OCT data by the OCT imaging.

[0048] Therefore, the ophthalmic device 110 can generate a tomographic image (B-scan image) of the fundus including the vorticose vein, which is an OCT image, OCT volume data including the vorticose vein, and an en-face image (frontal image generated based on the OCT volume data) that is a cross-section of the OCT volume data. It is needless to say that the OCT image includes an OCT image of the central part of the fundus (the posterior pole of the eyeball where the macula and the optic disc are present).

[0049] The OCT data (or the image data of the OCT image) is sent from the ophthalmic device 110 to the server 140 via the communication interface 16F and stored in the storage device 254.

[0050] In this embodiment, the light source 20A is an example of a wavelength-sweeping type SS-OCT (Swept-Source OCT), but various types of OCT systems such as SD-OCT (Spectral-Domain OCT) and TD-OCT (Time-Domain OCT) may also be used.

[0051] Next, referring to FIG. 3, the electrical configuration of the server 140 will be described. As shown in FIG. 3, the server 140 includes a computer main body 252. The computer main body 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268. A storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258 are connected to the input / output (I / O) port 268. The storage device 254 is composed of, for example, a non-volatile memory. The input / output (I / O) port 268 is connected to the network 130 via the communication interface (I / F) 258. Therefore, the server 140 can communicate with the ophthalmic device 110 and the viewer 150.

[0052] An image processing program is stored in the ROM 264 or the storage device 254.

[0053] The ROM 264 or the storage device 254 is an example of the "memory" of the present disclosure. The CPU 262 is an example of the "processor" of the present disclosure. The image processing program is an example of the "program" of the present disclosure.

[0054] The server 140 stores each piece of data received from the ophthalmic device 110 in the storage device 254.

[0055] Referring to Figure 4, various functions realized by the CPU 262 of the server 140 executing an image processing program will be described. The image processing program executed by the CPU 262 includes parts that realize display control functions, image processing functions, and processing functions. Specifically, by executing the image processing program, the CPU 262 operates as a display control unit 204 that realizes the display control functions shown in Figure 4, an image processing unit 206 that realizes the image processing functions, and a processing unit 208 that realizes the processing functions. The CPU 262 also functions as an image acquisition unit that acquires a first ophthalmic image and another ophthalmic image of the same eye as the first ophthalmic image, but taken at a different time and / or with a different device than the first ophthalmic image. The image processing unit 206 and / or processing unit 208 further include a deviation area information acquisition unit 211, a feature extraction unit 212, a displacement field vector calculation unit 213 (displacement information acquisition unit), and a deviation area position identification unit 214.

[0056] The deviation area information acquisition unit 211 acquires information about the deviation area DA, described later, from the ophthalmic image. Here, the deviation area information acquisition unit 211 acquires information about the deviation area DA by determining the presence and location of the deviation area DA in the ophthalmic image captured by the ophthalmic device 110, as an example. Alternatively, the deviation area information acquisition unit 211 may acquire information about the deviation area DA determined externally, rather than within the system 100.

[0057] The feature extraction unit 212 has the function of extracting parts of an ophthalmic image that have predetermined features. The displacement field vector calculation unit 213 calculates (acquires) a displacement field vector as an example of displacement information showing the change between two ophthalmic images for the purpose of superimposing the deviation area DA. The deviation area position identification unit 214 identifies the position of the deviation area DA identified in one image in the other image according to the displacement field vector. In addition to the position identification process, the deviation area position identification unit 214 may also be configured to perform the deviation area superimposition process at the identified position in the other image. The displacement field vector may be obtained for each pixel in the image, for example, or it may be obtained only for specific feature points. Furthermore, the displacement field vector can be calculated by performing interpolation on the obtained displacement field vector.

[0058] Next, with reference to Figures 5 to 7, the image processing performed by the management server 140 (overlay display processing of deviation areas) will be explained in detail. The CPU 262 of the management server 140 executes an image processing program, thereby realizing the image processing shown in the flowcharts of Figures 5 and 6. Figure 7 illustrates the outline of this image processing.

[0059] The image processing shown in Figure 5 is an example of an image processing method of the present disclosure. Furthermore, the display processing for displaying the image obtained through the image processing shown in Figure 5 is an example of an image display method of the present disclosure.

[0060] In the image processing of this embodiment, when a new ophthalmic image is obtained, areas (deviation areas) are detected in the new image that have features that deviate from past ophthalmic images of the same patient or from standard images, and where it is judged that there is a possibility that a lesion exists in the eye being examined. Deviation areas can be detected and determined, for example, by comparing them with past photographs of the same patient's eyes being examined or by following the results of machine learning using a large number of images as training data. Then, the area in another image of the eye being examined where the deviation area is located can be identified, and the deviation area can be superimposed on the identified area. By performing such superimposition of deviation areas, it becomes easier for physicians to determine whether a lesion is present in the detected deviation area.

[0061] Here, as an example, we will show a case where a full-color UWF-SLO image (hereinafter referred to as the SLO image) obtained using G-color fundus images, R-color fundus images, and B-color fundus images, and a UWF-SLO image (hereinafter referred to as the FA image) obtained by fluorescein angiography (FA) are obtained. Then, we will explain the process of superimposing the deviation area in the latter image when a deviation area is detected in the former. The FA image is suitable for detecting non-perfusion areas because it captures the retinal capillaries with high resolution. These image data are sent from the ophthalmic device 110 to the management server 140 via a communication interface (not shown) and stored in the storage device 254.

[0062] In the SLO image P1 (upper left of Figure 7), a deviation area DA is detected in the SLO image P1 by comparing it with another image P1p (not shown in Figure 7) taken at a different time (e.g., a past point in time) and / or with a different device, and a deviation area extraction image P2 is generated by extracting the deviation area DA (step S11 in Figure 5, Figure 7(a)). Subsequently, it is determined whether or not to perform superimposition display on another image (e.g., FA image) with respect to the detected deviation area DA (step S12). This determination may be made by the user of this system using a predetermined input device, or it may be made automatically by the image processing program under predetermined conditions. If it is determined in step S12 that superimposition display is not necessary (No), this image processing is terminated.

[0063] On the other hand, if it is determined in step S12 that superimposition is necessary (Yes), the SLO image P1 and the corresponding image of the eye under examination (e.g., FA image P3) are collected (step S13). The corresponding image may be read from the storage device 254, or it may be newly generated based on the SLO image P1. Here, as an example, the feature extraction unit 212 generates a vascular-enhanced / extracted image P1' by enhancing and extracting the image of blood vessels contained in the SLO image P1, and the FA image P3 of the same eye under examination, which was captured separately, is read from the storage device 254 (Figure 7). It is also possible to extract image features other than blood vessels (macula, optic nerve head, vascular bifurcation, etc.).

[0064] Next, for example, predetermined image processing is performed on corresponding images P1' and P3, and blood vessels are extracted in each image (step S14), and corresponding blood vessels are determined from among the blood vessels detected in both images (step S15). At this time, in image P1', which is an SLO image, the image may be separated into wavelength components, and blood vessels may be extracted based on the short wavelength components. When blood vessels are extracted based on the short wavelength components of the SLO image, more information about blood vessels on the surface of the retina can be obtained. In corresponding images P1' and P3, blood vessels included in the region other than the deviation area DA are determined (step S16), and further image analysis of the determined blood vessels is performed so that a displacement field vector →v indicating the change in blood vessels (outside the deviation area DA) between corresponding images P1' and P3 is calculated by the displacement vector calculation unit 213 (step S17). As mentioned above, the deviation area DA is an area that has features that deviate from those of a normal image, so it is preferable to exclude the deviation area DA when calculating the displacement field vector →v. The displacement field vector calculation unit 213 can identify a first feature portion in the region of the ophthalmic image P1' excluding the deviation area DA, identify a second feature portion corresponding to the first feature portion in the FA image P3, and calculate a displacement field vector from the change in position between the first feature portion and the second feature portion.

[0065] In the procedure described above, blood vessels are extracted in the corresponding ophthalmic image, and the superposition of the deviation area DA is performed based on the displacement field vector →v, which is a transformation function obtained from the differences in the corresponding blood vessels. This allows for accurate superposition of the deviation area DA on other ophthalmic images. Furthermore, by excluding deviation area DA where lesions may be present and calculating the displacement field vector →v, it becomes possible to perform the superposition of the deviation area DA with even greater accuracy.

[0066] Once the displacement field vector →v is calculated in this manner, the displacement field vector →v is combined with the image data P2' of the deviation area DA included in the deviation area extraction image P2, which contains the deviation area DA, based on the positional relationship between the corresponding images P1' and P3, thereby generating the deformed deviation area extraction image P2'' (step S18). Furthermore, the deviation area superimposition unit 214 generates a superimposed image P3' in which the image of the deviation area DA is superimposed on the FA image P3 (step S19). Alternatively, the displacement field vector →v may be combined with the entire deviation area extraction image P2 to generate the deformed deviation area extraction image P2''.

[0067] Referring to the flowchart in Figure 6, a more detailed procedure for calculating the displacement field vector →v (step S17) will be explained. As mentioned above, the displacement field vector →v is calculated in the region excluding the deviation area DA. At this time, it is possible to choose whether to calculate the displacement field vector →v in the region other than the deviation area DA (step S23: first mode), or to calculate the displacement field vector →v in the region other than the deviation area DA and its surrounding area (an area expanded by +α% beyond the deviation area DA) (step S24: second mode). For example, depending on whether the patient is receiving a prescribed treatment and the treatment effect is being confirmed, or whether the patient is simply being observed, either step S23 or S25 can be selected (steps S21, S22, S24). If treatment is in progress, it is often predicted that the lesion (deviation area DA) will remain in place or shrink due to the improvement of the disease, so step S23 can be executed, excluding only the deviation area DA. On the other hand, when simply observing the patient's progress, it is predicted that the deviation area DA is likely to expand, so the area of ​​deviation area DA + α is excluded (step S25). In this way, by appropriately changing the size of the area excluded from the calculation of the displacement field vector, it becomes possible to perform the superposition processing of the deviation area DA more accurately. In diseases in which the deviation area progresses (non-specific examples include diabetic retinopathy, retinal detachment, hole, pigmentary degeneration, etc.), it is preferable to set the size of +α to change according to the elapsed time. Diabetic retinopathy, retinal detachment, etc. are diseases in which bleeding from surrounding capillaries and avascular areas tend to expand, so the deviation area tends to expand as the disease progresses. The size of +α may also be changed depending on the patient's age. Furthermore, since the disease progresses faster in younger patients, it is possible to set a larger +α, and it is also possible to change the size of +α depending on the type of disease.

[0068] As described above, the ophthalmic system 100 of this embodiment makes it possible to accurately align and display the deviation area detected in one ophthalmic image in other ophthalmic images. By confirming the deviation area in multiple ophthalmic images, a diagnosis of the deviation area can be performed more accurately and with greater certainty. Furthermore, the display of the deviation area in multiple ophthalmic images makes it easier to confirm the progression of the disease and the effectiveness of treatment.

[0069] [Second Embodiment] Next, the ophthalmic system 100 of the second embodiment will be described with reference to Figures 8 to 9. The hardware configuration of the ophthalmic system 100 of this second embodiment is substantially the same as that of the first embodiment (Figures 1 to 4). However, in this second embodiment, the content and procedure of calculating the displacement field vector and the superposition processing of the deviation area DA differ from those of the first embodiment. The flowcharts in Figures 8 to 9 explain this content and procedure.

[0070] Steps S11 to S17 shown in Figure 8 are the same as in the first embodiment. The displacement field vector →v is calculated in the region excluding the deviation area DA (step S17). Subsequently, as shown in Figure 9, the displacement field vector →V within the deviation area DA is calculated (step S31). The displacement field vector →V within the deviation area DA is expected to include a component indicating the degree of disease progression. On the other hand, it is also estimated that the displacement field vector →V includes a component of changes over time between the two images, and a component based on alignment deviations.

[0071] Therefore, in this embodiment, the difference →V - →v between the displacement field vector →V in the deviation area DA and the displacement field vector →v in the area excluding the deviation area DA is calculated (step S32). The difference →V - v can be considered to represent the change purely due to the disease, after subtracting the component of age-related change and the component of alignment deviation. The difference →V - v can be calculated by interpolating the displacement field vector →V. At this time, the interpolation may be performed individually for the X direction and the Y direction. Furthermore, the interpolation may be performed based on the absolute value of the magnitude of the vector or based on the angle of the vector.

[0072] In steps S18 to S19, similar to the first embodiment, a displacement field vector →v is synthesized on the deviation area extraction image P2, which includes the deviation area DA, based on the positional relationship between the corresponding images P1' and P3 (step S18), and a superimposed image P3' is generated in which the image of the deviation area DA is superimposed on the FA image P3 (step S19).

[0073] Alternatively, the displacement field vector →v outside the deviation area DA can be interpolated to obtain the displacement field vector →V' within the deviation area DA, and the difference →V - →V' between the displacement field vectors →V and →V' can be calculated. This difference can be considered to contain only the component due to the progression of the disease in the deviation area DA, and can be used to diagnose the disease.

[0074] As described above, the ophthalmic system 100 of the second embodiment provides the same effects as the first embodiment, and by calculating the difference between the displacement field vector →V within the deviation area DA and the displacement field vector →v outside the deviation area DA, it becomes possible to detect changes in the eye under examination due to disease.

[0075] [Other] Although various embodiments have been described above, this disclosure is not limited to the embodiments described above and includes various modifications. For example, the embodiments described above are described in detail to make this disclosure easier to understand and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0076] For example, in the above embodiment, we described the superposition processing of deviation areas when there is an SLO image as the first modality and an FA image as the second modality. However, the types of modalities are just examples, and various combinations are possible.

[0077] For example, the first modality may be an SLO image as described above, while the second modality may be an image (ICG image) obtained by fluorescein angiography using indocyanine green (ICG) as the contrast agent, instead of an FA image. In this case, as an example, the displacement field vector →v can be determined between the long-wavelength component image obtained by separating the SLO image of the first modality into wavelength components and the ICG image of the second modality, and the deviation area DA can be aligned.

[0078] Furthermore, while the first modality may be an SLO image as described above, the second modality may be an OCT-A image. In this case, for example, a displacement field vector →v can be obtained between the short-wavelength component of the SLO image of the first modality and the top layer image of the OCT-A image of the second modality to align the deviation area DA.

[0079] Alternatively, the first modality can be a standard fundus image (low field of view), and the second modality can be a SLO image. In this case, after trimming the SLO image of the second modality to match the field of view of the fundus image, the displacement field vector →v can be calculated between the short-wavelength component image of the SLO image and the fundus image, and the deviation area DA can be aligned. When extracting blood vessels based on the short-wavelength component of the SLO image, a lot of information about blood vessels on the retinal surface can be obtained. In this case, it becomes easier to correlate with the blood vessel information shown in the fundus image.

[0080] In the above embodiment, image processing (Figure 5) is performed by the server 140, but the disclosure is not limited thereto, and it may also be performed by the ophthalmic device 110, the viewer 150, or an additional image processing device further provided in the network 130.

[0081] In this disclosure, each component (device, etc.) may exist as one or more, as long as it does not create a contradiction.

[0082] The examples described above illustrate cases where image processing is implemented using a software configuration with a computer. However, this disclosure is not limited to these cases, and at least some of the processing may be implemented using a hardware configuration. Furthermore, although a CPU was used as an example of a general-purpose processor above, the term "processor" refers to a broader term and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, Programmable Logical Device, etc.). Therefore, image processing may be performed solely by a hardware configuration, or some of the image processing may be performed by a software configuration and the remaining processing by a hardware configuration.

[0083] Furthermore, the processor operation described above may not be performed by a single processor, but may also be performed by multiple processors working together, or by multiple processors located in physically separate locations.

[0084] Furthermore, in order to have a computer execute the above-mentioned processes, a program in which the above-mentioned processes are written in code that can be processed by a computer may be stored on a storage medium such as an optical disc and distributed.

[0085] Thus, this disclosure includes cases where image processing is implemented using computer-based software configurations and cases where it is not, and therefore includes the following technologies.

[0086] Based on the above disclosure, the following technology is proposed: an image processing program configured to cause a computer to perform the following steps: determining a deviation area that deviates from a predetermined state in a first ophthalmic image; comparing the first ophthalmic image and the second ophthalmic image and outputting displacement information indicating the change between the first ophthalmic image and the second ophthalmic image; and aligning the second ophthalmic image and the deviation area according to a deviation area image indicating the position of the deviation area, the displacement information, and the second ophthalmic image.

[0087] The above examples illustrate cases where image processing is implemented using a computer-based software configuration, but the technology of this disclosure is not limited to these. For example, instead of a computer-based software configuration, image processing may be performed solely by a hardware configuration such as an FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Alternatively, some of the image processing may be performed by a software configuration, and the remaining processing may be performed by a hardware configuration.

[0088] Furthermore, the image processing described above is merely an example, and the technical scope of this disclosure is not limited to the scope described in the embodiments above. Accordingly, various modifications or improvements can be made to the embodiments above, such as deleting unnecessary processing, adding new processing, or changing the processing order, without departing from the spirit of the invention, and such modified or improved forms are also included in the technical scope of this disclosure.

[0089] All documents, patent applications, and technical standards described herein are incorporated by reference in the same manner as when each individual document, patent application, and technical standard is specifically and individually incorporated by reference.

[0090] 140 Server 262 CPU 264 ROM 254 Storage Device 206 Image Processing Unit

Claims

1. An image processing device comprising: an image acquisition unit that acquires a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye as the first ophthalmic image and was captured at a different time and / or with a different device than the first ophthalmic image; a deviation area information acquisition unit that acquires information on a deviation area that deviates from a predetermined state in the first ophthalmic image; a displacement information acquisition unit that compares the first ophthalmic image and the second ophthalmic image and acquires displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and a deviation area position identification unit that aligns the second ophthalmic image and a deviation area image indicating the position of the deviation area based on the displacement information and identifies the position of the deviation area in the second ophthalmic image, wherein the displacement information acquisition unit acquires the displacement information by excluding the deviation area in the first ophthalmic image from the target for acquiring the displacement information.

2. The image processing apparatus according to claim 1, wherein the deviation area position identification unit deforms the deviation area image by synthesizing the displacement information with the deviation area image, and aligns the deformed deviation area image with the second ophthalmic image to identify the position of the deviation area in the second ophthalmic image.

3. The image processing apparatus according to claim 1 or 2, wherein the displacement information acquisition unit has a first mode for acquiring the displacement information using information of the region excluding the deviation area, and a second mode for acquiring the displacement information using information of the region excluding the deviation area and its surrounding region.

4. The image processing apparatus according to claim 3, wherein the displacement information acquisition unit acquires the displacement information using the first mode or the second mode when the first ophthalmic image and the second ophthalmic image, which was captured at a different time than the first ophthalmic image, are acquired.

5. The image processing apparatus according to claim 4, wherein the displacement information acquisition unit acquires the displacement information using the second mode when it is predicted that the area of ​​the deviation area is expanding.

6. The image processing apparatus according to claim 5, wherein the displacement information acquisition unit acquires the displacement information using the second mode when it is predicted that the area of ​​the deviation area in the first ophthalmic image will have expanded when the second ophthalmic image is taken after a period of time has elapsed.

7. The image processing apparatus according to claim 4, wherein the displacement information acquisition unit acquires the displacement information using the first mode when it is predicted that the area of ​​the deviation area is decreasing or has not changed.

8. The image processing apparatus according to claim 4, wherein the displacement information acquisition unit acquires the displacement information using the first mode when it is predicted that the area of ​​the deviation area in the first ophthalmic image will have decreased or not changed when the second ophthalmic image is taken after a period of time has elapsed.

9. The image processing apparatus according to any one of claims 1 to 8, wherein the displacement information acquisition unit identifies a first feature portion in at least the region of the first ophthalmic image excluding the deviation area, identifies a second feature portion corresponding to the first feature portion in the second ophthalmic image, and acquires the displacement information from the change in position between the first feature portion and the second feature portion.

10. The image processing apparatus according to claim 9, wherein the displacement information acquisition unit identifies blood vessels as the first feature portion and the second feature portion.

11. The image processing apparatus according to claim 9, wherein the displacement information acquisition unit identifies at least one of the macula, the optic nerve head, and a vascular bifurcation as the first feature portion and the second feature portion.

12. The image processing apparatus according to any one of claims 1 to 11, wherein the displacement information is a displacement field vector indicating a change between the first ophthalmic image and the second ophthalmic image, and the deviation area position identification unit deforms the deviation area image by combining the displacement field vector with the deviation area image, and identifies the position of the deviation area in the second ophthalmic image by aligning the deformed deviation area image with the second ophthalmic image.

13. The image processing apparatus according to any one of claims 1 to 12, wherein the deviation area location identification unit generates an image in which the deviation area image is superimposed on the second ophthalmic image.

14. The image processing apparatus according to any one of claims 1 to 13, wherein the deviation area information acquisition unit determines the deviation area in the first ophthalmic image.

15. The image processing apparatus according to claim 1, wherein the displacement information acquisition unit obtains first displacement information obtained by excluding the deviation area from the target for acquiring the displacement information, and second displacement information obtained by targeting the deviation area, and further comprises a determination unit that determines a change in the eye under examination based on the difference between the first displacement information and the second displacement information.

16. An image processing method comprising the steps of: acquiring a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye as the first ophthalmic image and was captured at a different time and / or with a different device than the first ophthalmic image; acquiring information on a deviation area in the first ophthalmic image that deviates from a predetermined state; comparing the first ophthalmic image and the second ophthalmic image to acquire displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and aligning the second ophthalmic image with a deviation area image indicating the location of the deviation area to identify the location of the deviation area in the second ophthalmic image, wherein, in acquiring the displacement information, the deviation area in the first ophthalmic image is excluded from the target for acquiring the displacement information.

17. An image processing program configured to allow a computer to perform the following steps: acquiring a first ophthalmic image and a second ophthalmic image which is an ophthalmic image of the same eye as the first ophthalmic image, but which was taken at a different time and / or with a different device than the first ophthalmic image; acquiring information on a deviation area in the first ophthalmic image that deviates from a predetermined state; comparing the first ophthalmic image and the second ophthalmic image to acquire displacement information indicating a change between the first ophthalmic image and the second ophthalmic image; and aligning the second ophthalmic image with a deviation area image indicating the position of the deviation area based on the displacement information to identify the position of the deviation area in the second ophthalmic image, wherein, in acquiring the displacement information, the deviation area in the first ophthalmic image is excluded from the target for acquiring the displacement information.

18. The image processing program according to claim 17, wherein, in determining the location of the deviation area, the deviation area image is deformed by synthesizing the displacement information with the deviation area image, and the location of the deviation area in the second ophthalmic image is determined by aligning the deformed deviation area image with the second ophthalmic image.

19. The image processing program according to claim 17, wherein, in acquiring the displacement information, a first mode is selected in which the displacement information is acquired using information of the region excluding the deviation area, and a second mode is selected in which the displacement information is acquired using information of the region excluding the deviation area and its surrounding region.

20. The image processing program according to claim 19, wherein, in acquiring the displacement information, the displacement information is acquired using the first mode or the second mode when the first ophthalmic image and the second ophthalmic image, which was captured at a different time than the first ophthalmic image, are acquired.

21. The image processing program according to claim 20, wherein, in acquiring the displacement information, the displacement information is acquired using the second mode when it is predicted that the area of ​​the deviation area is expanding.

22. The image processing program according to claim 20, wherein, in acquiring the displacement information, the displacement information is acquired using the first mode when it is predicted that the area of ​​the deviation area is decreasing or has not changed.

23. An image processing program according to any one of claims 17 to 22, wherein, in acquiring the displacement information, a first feature portion is identified in at least the region of the first ophthalmic image excluding the deviation area, a second feature portion corresponding to the first feature portion is identified in the second ophthalmic image, and the displacement information is acquired from the change in position between the first feature portion and the second feature portion.

24. The image processing program according to claim 23, wherein, in acquiring the displacement information, blood vessels are identified as the first feature portion and the second feature portion.

25. The image processing program according to claim 23, wherein, in acquiring the displacement information, at least one of the macula, the optic nerve head, and a vascular bifurcation is identified as the first feature portion and the second feature portion.

26. The image processing program according to claim 17, wherein the displacement information is a displacement field vector indicating a change between the first ophthalmic image and the second ophthalmic image, and in determining the position of the deviation area, the deviation area image is deformed by combining the displacement field vector with the deviation area image, and the position of the deviation area in the second ophthalmic image is determined by aligning the deformed deviation area image with the second ophthalmic image.

27. The image processing program according to any one of claims 17 to 26, further comprising the step of generating an image in which the deviation area image is superimposed on the second ophthalmic image.

28. The image processing program according to any one of claims 17 to 27, further comprising the step of determining the deviation area in the first ophthalmic image.

29. The image processing program according to claim 17, further comprising the step of obtaining first displacement information obtained by excluding the deviation area from the target for obtaining the displacement information, and second displacement information obtained by targeting the deviation area, and determining the change in the eye under examination based on the difference between the first displacement information and the second displacement information.

Images