Image processing device, image processing method, and program

Abstract

This image processing device executes processing that includes: acquiring a tomographic image in the depth direction including choroidal blood vessels and sclera; executing emphasis processing that emphasizes a physical quantity indicating a brightness change in the depth direction in the tomographic image with respect to a partial region of the tomographic image; deriving an image feature amount pertaining to the brightness change in the depth direction in the emphasized region; and determining, on the basis of the image feature amount, a site on the tomographic image in which the image feature amount exceeds a predetermined threshold as a boundary between the choroidal blood vessels and the sclera.

Description

Image processing device, image processing method, and program

[0001] The present disclosure relates to an image processing device, an image processing method, and a program.

[0002] U.S. Patent No. 10,238,281 discloses a technique for generating volume data of a subject's eye using an optical coherence tomography. It has been desired to visualize blood vessels based on the volume data of the subject's eye.

[0003] A first aspect is an image processing device including: an acquisition unit that acquires a depth-wise tomographic image including choroidal blood vessels and the sclera; a derivation unit that performs enhancement processing on a partial region of the tomographic image to enhance a physical quantity indicating a brightness change in the depth direction in the tomographic image, and derives an image feature amount related to the brightness change in the depth direction in the enhanced region; and a determination unit that determines, based on the image feature amount, a portion on the tomographic image where the image feature amount exceeds a predetermined threshold value as a boundary between the choroidal blood vessels and the sclera.

[0004] A second aspect is an image processing method in which a processor performs processing including: acquiring a depth-wise tomographic image including choroidal blood vessels and the sclera; performing enhancement processing on a partial region of the tomographic image to enhance a physical quantity indicating a change in brightness in the depth direction in the tomographic image; deriving an image feature amount related to the change in brightness in the depth direction in the enhanced region; and determining, based on the image feature amount, a portion on the tomographic image where the image feature amount exceeds a predetermined threshold value as a boundary between the choroidal blood vessels and the sclera.

[0005] A third aspect is a program that causes a processor to perform image processing including: acquiring a depth-wise tomographic image including choroidal blood vessels and the sclera; performing enhancement processing on a partial region of the tomographic image to enhance a physical quantity indicating a change in brightness in the depth direction in the tomographic image; deriving an image feature amount related to the change in brightness in the depth direction in the enhanced region; and determining, based on the image feature amount, a portion on the tomographic image where the image feature amount exceeds a predetermined threshold value as a boundary between the choroidal blood vessels and the sclera.

[0006] 1 is a schematic diagram of an ophthalmologic system according to an embodiment; FIG. 2 is a schematic diagram of an ophthalmologic apparatus according to an embodiment; FIG. 3 is a schematic diagram of a server; FIG. 4 is an explanatory diagram of functions realized by an image processing program in a CPU of the server; FIG. 5 is a flowchart showing an example of the flow of image processing by the server; FIG. 6 is a schematic diagram showing the relationship between the eyeball and the position of a vortex vein; FIG. 7 is a diagram showing the relationship between OCT volume data and an en-face image; FIG. 8 is a diagram showing an example of a fundus image of choroidal vessels; FIG. 9 is a flowchart showing an example of the flow of choroid-scleral boundary detection processing; FIG. 10 is a flowchart showing an example of the flow of two-dimensional processing; FIG. 11 is a diagram showing an example of an image in two-dimensional processing; FIG. 12 is a diagram showing an example of a two-dimensional filter; FIG. 13 is a diagram showing an example of the relationship between the position in the depth direction on an image and brightness; FIG. 14 is an explanatory diagram relating to an example of determining the boundary surface between the choroid and the sclera; FIG. 15 is a flowchart showing an example of the flow of three-dimensional processing; FIG. 16 is a diagram showing an example of an image in three-dimensional processing; FIG. 17 is a diagram showing an example of a three-dimensional filter; FIG. 18 is a diagram showing an example of a stereoscopic image of choroidal vessels; FIG. 19 is a diagram showing an example of a display screen using the stereoscopic image of choroidal vessels; FIG. 19 is a flowchart showing an example of the flow of image processing according to an embodiment; FIG. 19 is a diagram showing an example of an image in blood vessel image formation processing; FIG. 10 is a diagram showing an example of a display screen using an image including highlighted blood vessels.

[0007] Hereinafter, embodiments of the disclosed technology will be described in detail with reference to the drawings. Note that components and processes that perform the same actions and functions are given the same reference numerals throughout the drawings, and redundant explanations may be omitted as appropriate. Also, explanations of configurations that are not directly related to the disclosed technology or well-known configurations may be omitted. Also, the dimensional ratios in the drawings are exaggerated for the sake of explanation and may differ from the actual ratios. Furthermore, each drawing is merely a schematic illustration to allow a sufficient understanding of the disclosed technology. Therefore, the disclosed technology is not limited to the illustrated examples.

[0008] The disclosed technology can be applied to an ophthalmic apparatus according to the disclosed technology as long as it is an apparatus that acquires images related to the eye, such as a fundus image, etc. In this embodiment, for the sake of simplicity, the following description will be given of an example of an ophthalmic apparatus having an image acquisition function, in which an observer such as a doctor observes the eye of a patient (hereinafter referred to as the subject eye), for example, the fundus and the periphery of the subject eye, for example, the anterior segment, for the purpose of ophthalmic diagnosis and surgical treatment on the eye.

[0009] <First Embodiment> Fig. 1 shows a schematic configuration of an ophthalmologic system 100. As shown in Fig. 1, the ophthalmologic system 100 includes an ophthalmologic apparatus 110, a server apparatus (hereinafter referred to as "server") 140, and a display device (hereinafter referred to as "viewer") 150. The ophthalmologic apparatus 110 acquires fundus images. The server 140 stores, in association with patient IDs, a plurality of fundus images obtained by photographing the funduses of a plurality of patients using the ophthalmologic apparatus 110 and axial lengths measured by an axial length measuring device (not shown). The viewer 150 displays the fundus images and analysis results acquired by the server 140.

[0010] The server 140 is an example of the "image processing device" of the present disclosure.

[0011] The ophthalmic apparatus 110, the server 140, and the viewer 150 are connected to each other via a network 130. The network 130 may be any network such as a LAN, a WAN, the Internet, or a wide area Ethernet network. For example, if the ophthalmic system 100 is established in a single hospital, a LAN may be used as the network 130.

[0012] The viewer 150 is a client in a client-server system, and multiple viewers 150 are connected via a network. Multiple servers 140 may also be connected via a network to ensure system redundancy. Alternatively, if the ophthalmic apparatus 110 has an image processing function and the image viewing function of the viewer 150, the ophthalmic apparatus 110 can acquire, process, and view fundus images in a standalone state. If the server 140 has the image viewing function of the viewer 150, the configuration of the ophthalmic apparatus 110 and the server 140 can acquire, process, and view fundus images.

[0013] In addition, other ophthalmic devices (examination equipment for visual field measurement, intraocular pressure measurement, etc.) and diagnostic support devices that perform image analysis using AI (Artificial Intelligence) may be connected to the ophthalmic device 110, the server 140, and the viewer 150 via the network 130.

[0014] Next, the configuration of the ophthalmologic apparatus 110 will be described with reference to FIG.

[0015] For ease of explanation, a scanning laser ophthalmoscope will be referred to as an "SLO," and an optical coherence tomography will be referred to as an "OCT."

[0016] When the ophthalmic apparatus 110 is placed on a horizontal plane, the horizontal direction is defined as the "X direction," the vertical direction relative to the horizontal plane is defined as the "Y direction," and the direction connecting the center of the pupil of the anterior segment of the subject's eye 12 and the center of the eyeball is defined as the "Z direction." Therefore, the X direction, Y direction, and Z direction are perpendicular to each other.

[0017] The ophthalmologic apparatus 110 includes an imaging device 14 and a control device 16. The imaging device 14 is equipped with an SLO unit 18 and an OCT unit 20, and acquires a fundus image of the subject's eye 12. Hereinafter, a two-dimensional fundus image acquired by the SLO unit 18 will be referred to as an SLO image. Furthermore, a tomographic image or an en-face image of the retina created based on OCT data acquired by the OCT unit 20 will be referred to as an OCT image.

[0018] The OCT data includes A-scan data, which is OCT data obtained by scanning a single point on the fundus in the depth (optical axis) direction (hereinafter referred to as A-scan) when a single point is used to acquire a tomographic image. It also includes B-scan data, which is OCT data obtained by performing multiple A-scans while moving the A-scan along the line (hereinafter referred to as B-scan) when a line is used to acquire a tomographic image. B-scan data may be generated by interpolating multiple A-scan data. Furthermore, it includes C-scan data, which is OCT data obtained by performing multiple B-scans while moving the B-scan along the surface (hereinafter referred to as C-scan) when a surface is used to acquire a tomographic image. C-scan data is three-dimensional OCT data, generated as OCT volume data, and it is possible to generate two-dimensional en-face images, etc., based on the three-dimensional OCT data. It is also possible to generate C-scan data by interpolating multiple B-scan data.

[0019] The control device 16 comprises a computer having a CPU (Central Processing Unit) 16A, a RAM (Random Access Memory) 16B, a ROM (Read-Only Memory) 16C, and an input / output (I / O) port 16D.

[0020] The control device 16 includes an input / display device 16E connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphic user interface that displays an image of the subject's eye 12 and receives various instructions from the user. An example of the graphic user interface is a touch panel display.

[0021] The control device 16 also includes an image processor 17 connected to an I / O port 16D. The image processor 17 generates an image of the subject's eye 12 based on data obtained by the photographing device 14. The control device 16 is connected to a network 130 via a communication interface (I / F) 16F.

[0022] 2 , the control device 16 of the ophthalmic apparatus 110 includes the input / display device 16E, but the present disclosure is not limited to this. For example, the control device 16 of the ophthalmic apparatus 110 may not include the input / display device 16E, but may instead include a separate input / display device that is physically independent from the ophthalmic apparatus 110. In this case, the display device includes an image processing unit that operates under the control of the display control unit 204 of the CPU 16A of the control device 16. The image processing unit may display an SLO image or the like based on an image signal instructed to be output by the display control unit 204.

[0023] The imaging device 14 operates under the control of the CPU 16A of the control device 16. The imaging device 14 includes an SLO unit 18, an imaging optical system 19, and an OCT unit 20. The imaging optical system 19 includes an optical scanner 22 and a wide-angle optical system 30.

[0024] The optical scanner 22 performs two-dimensional scanning in the X and Y directions with the light emitted from the SLO unit 18. The optical scanner 22 may be any optical element that can deflect a light beam, such as a polygon mirror or a galvanometer mirror, or a combination thereof.

[0025] The wide-angle optical system 30 combines the light from the SLO unit 18 and the light from the OCT unit 20 .

[0026] The wide-angle optical system 30 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric optical system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph the retina in the peripheral part of the fundus as well as the center of the fundus.

[0027] When a system including an elliptical mirror is used, the system using the elliptical mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may be used, the disclosures of which are each incorporated herein by reference in their entirety.

[0028] The wide-angle optical system 30 enables observation of the fundus over a wide field of view (FOV) 12A. The FOV 12A indicates the range that can be photographed by the imaging device 14. The FOV 12A can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal illumination angle and an external illumination angle. The external illumination angle is the illumination angle of the light beam irradiated from the ophthalmic device 110 to the subject's eye 12, determined with the pupil 27 as the reference point. The internal illumination angle is the illumination angle of the light beam irradiated to the fundus, determined with the center O of the eyeball as the reference point. The external illumination angle and the internal illumination angle correspond to each other. For example, if the external illumination angle is 120 degrees, the internal illumination angle corresponds to approximately 160 degrees. In this embodiment, the internal illumination angle is 200 degrees.

[0029] Here, an SLO fundus image captured at an internal illumination angle and a field of view of 160 degrees or more is referred to as a UWF-SLO fundus image. UWF stands for Ultra Wide Field. The wide-angle optical system 30, which provides an ultra-wide field of view (FOV) of the fundus, can capture an image of the area extending from the posterior pole of the fundus of the subject's eye 12 beyond the equator, enabling the image of structures present in the peripheral area of ​​the fundus, such as vortex veins.

[0030] The ophthalmologic apparatus 110 can capture an image of an area 12A with an internal illumination angle of 200°, with the center O of the eyeball of the subject's eye 12 as the reference position. Note that the internal illumination angle of 200° corresponds to an external illumination angle of 110° with the pupil of the eyeball of the subject's eye 12 as the reference. In other words, the wide-angle optical system 30 irradiates the laser light from the pupil at an angle of view with an external illumination angle of 110°, and captures an image of a fundus area of ​​200° with an internal illumination angle.

[0031] 2, the SLO system is realized by a control device 16, an SLO unit 18, and an imaging optical system 19. The SLO system includes a wide-angle optical system 30, and therefore enables fundus imaging with a wide FOV 12A.

[0032] The SLO unit 18 includes a light source 40 for B light (blue light), a light source 42 for G light (green light), a light source 44 for R light (red light), and a light source 46 for IR light (infrared light (e.g., near-infrared light)), as well as optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light from the light sources 40, 42, 44, and 46 and guide them into a single optical path. The optical systems 48 and 56 are mirrors, and the optical systems 50, 52, and 54 are beam splitters. The B light is reflected by the optical system 48, passes through the optical system 50, and is reflected by the optical system 54; the G light is reflected by the optical systems 50 and 54; the R light is transmitted through the optical systems 52 and 54; and the IR light is reflected by the optical systems 52 and 56 and is each guided into a single optical path.

[0033] The SLO unit 18 is configured to be switchable between a light source that emits laser light of different wavelengths or a combination of light sources that emit light, such as a mode that emits R light and G light and a mode that emits infrared light. In the example shown in Fig. 2, the SLO unit 18 includes four light sources: a B light source 40, a G light source 42, an R light source 44, and an IR light source 46; however, the present disclosure is not limited to this. For example, the SLO unit 18 may further include a white light source and emit light in various modes, such as a mode that emits G light, R light, and B light, or a mode that emits only white light.

[0034] Light incident on the imaging optical system 19 from the SLO unit 18 is scanned in the X and Y directions by the optical scanner 22. The scanned light passes through the wide-angle optical system 30 and the pupil 27 and is then irradiated onto the fundus. The light reflected by the fundus passes through the wide-angle optical system 30 and the optical scanner 22 and is incident on the SLO unit 18.

[0035] The SLO unit 18 includes a beam splitter 64 that reflects B light and transmits all light except B light from the posterior segment (fundus) of the eye 12, and a beam splitter 58 that reflects G light and transmits all light except G light from the light that has passed through the beam splitter 64. The SLO unit 18 also includes a beam splitter 60 that reflects R light and transmits all light except R light from the light that has passed through the beam splitter 58. The SLO unit 18 also includes a beam splitter 62 that reflects IR light from the light that has passed through the beam splitter 60. The SLO unit 18 also includes a B light detection element 70 that detects B light reflected by the beam splitter 64, a G light detection element 72 that detects G light reflected by the beam splitter 58, an R light detection element 74 that detects R light reflected by the beam splitter 60, and an IR light detection element 76 that detects IR light reflected by the beam splitter 62.

[0036] Light (light reflected by the fundus) incident on the SLO unit 18 via the wide-angle optical system 30 and the optical scanner 22 is reflected by the beam splitter 64 and received by the B light detection element 70 in the case of B light, and is reflected by the beam splitter 58 and received by the G light detection element 72 in the case of G light. The incident light is transmitted through the beam splitter 58 in the case of R light, reflected by the beam splitter 60, and received by the R light detection element 74. The incident light is transmitted through the beam splitters 58 and 60 in the case of IR light, reflected by the beam splitter 62, and received by the IR light detection element 76. The image processor 17, which operates under the control of the CPU 16A, generates a UWF-SLO image using signals detected by the B light detection element 70, the G light detection element 72, the R light detection element 74, and the IR light detection element 76.

[0037] A UWF-SLO image generated using a signal detected by the B light detecting element 70 is referred to as a B-UWF-SLO image (B-color fundus image). A UWF-SLO image generated using a signal detected by the G light detecting element 72 is referred to as a G-UWF-SLO image (G-color fundus image). A UWF-SLO image generated using a signal detected by the R light detecting element 74 is referred to as an R-UWF-SLO image (R-color fundus image). A UWF-SLO image generated using a signal detected by the IR light detecting element 76 is referred to as an IR-UWF-SLO image (IR fundus image). UWF-SLO images include R-color fundus images, G-color fundus images, B-color fundus images, and even IR fundus images. Fluorescent UWF-SLO images obtained by capturing fluorescence are also included.

[0038] The control device 16 also controls the light sources 40, 42, and 44 to emit light simultaneously. By simultaneously capturing images of the fundus of the subject's eye 12 using B, G, and R light, a G-color fundus image, a R-color fundus image, and a B-color fundus image, each of which corresponds to a different position, are obtained. An RGB color fundus image is obtained from the G-color fundus image, the R-color fundus image, and the B-color fundus image. By simultaneously controlling the light sources 42 and 44 to emit light simultaneously and simultaneously capturing images of the fundus of the subject's eye 12 using G and R light, a G-color fundus image and a R-color fundus image, each of which corresponds to a different position, are obtained. An RG color fundus image is obtained from the G-color fundus image and the R-color fundus image. A full-color fundus image may also be generated using the G-color fundus image, the R-color fundus image, and the B-color fundus image.

[0039] The wide-angle optical system 30 makes the field of view (FOV) of the fundus an ultra-wide angle, and can capture an image of the area from the posterior pole of the fundus of the subject's eye 12 beyond the equator.

[0040] The OCT system is realized by the control device 16, OCT unit 20, and imaging optical system 19 shown in FIG. 2 . The OCT system includes a wide-angle optical system 30, which enables OCT imaging of the peripheral portion of the fundus, similar to the above-described SLO fundus image capture. That is, the wide-angle optical system 30, which provides an ultra-wide fundus field of view (FOV), enables OCT imaging of the region extending from the posterior pole of the fundus of the subject's eye 12 beyond the equator 178. OCT data of structures present in the peripheral portion of the fundus, such as vortex veins, can be acquired, and tomographic images of the vortex veins and the 3D structure of the vortex veins can be obtained by image processing the OCT data.

[0041] The OCT unit 20 includes a light source 20A, a sensor (detecting element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0042] Light emitted from the light source 20A is branched by the first optical coupler 20C. One of the branched beams is collimated by the collimating lens 20E as measurement light and then enters the imaging optical system 19. The measurement light passes through the wide-angle optical system 30 and the pupil 27 and is irradiated onto the fundus. The measurement light reflected by the fundus passes through the wide-angle optical system 30 and enters the OCT unit 20, and then passes through the collimating lens 20E and the first optical coupler 20C and enters the second optical coupler 20F.

[0043] The other light beam emitted from the light source 20A and branched by the first optical coupler 20C is incident as reference light on the reference optical system 20D, passes through the reference optical system 20D, and enters the second optical coupler 20F.

[0044] The light beams incident on the second optical coupler 20F, i.e., the measurement light beam reflected by the fundus and the reference light beam, interfere with each other at the second optical coupler 20F to generate interference light. The interference light beam is received by the sensor 20B. The image processor 17, which operates under the control of the image processing unit 206, generates OCT data detected by the sensor 20B. The image processor 17 can also generate OCT images, such as tomographic images and en-face images, based on the OCT data.

[0045] Here, the OCT unit 20 can scan a predetermined range (e.g., a rectangular range of 6 mm x 6 mm) in one OCT imaging session. The predetermined range is not limited to 6 mm x 6 mm, but may be a square range of 12 mm x 12 mm or 23 mm x 23 mm, or a rectangular range such as 14 mm x 9 mm or 6 mm x 3.5 mm, or any other rectangular range. It may also be a circular range with a diameter of 6 mm, 12 mm, or 23 mm.

[0046] By using the wide-angle optical system 30, the ophthalmic apparatus 110 can scan the area 12A with an internal illumination angle of 200°. That is, by controlling the optical scanner 22, OCT imaging of a predetermined range including vortex veins is performed. The ophthalmic apparatus 110 can generate OCT data through this OCT imaging.

[0047] Therefore, the ophthalmologic apparatus 110 can generate OCT images, such as tomographic images (B-scan images) of the fundus including vortex veins, OCT volume data including vortex veins, and en-face images (frontal images generated based on the OCT volume data) that are cross sections of the OCT volume data. It goes without saying that the OCT images include OCT images of the center of the fundus (the posterior pole of the eyeball where the macula, optic disc, etc. are present).

[0048] The OCT data (or image data of the OCT image) is sent from the ophthalmologic apparatus 110 to the server 140 via the communication interface 16F and stored in the storage device 254.

[0049] In this embodiment, the light source 20A is exemplified as a wavelength-swept type SS-OCT (Swept-Source OCT), but various types of OCT systems may also be used, such as SD-OCT (Spectral-Domain OCT) and TD-OCT (Time-Domain OCT).

[0050] Next, the configuration of the electrical system of the server 140 will be described with reference to FIG. 3 . As shown in FIG. 3 , the server 140 includes a computer main unit 252. The computer main unit 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268. The input / output (I / O) port 268 is connected to a storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258. The storage device 254 is configured, for example, with a non-volatile memory. The input / output (I / O) port 268 is connected to the network 130 via the communication interface (I / F) 258. Therefore, the server 140 can communicate with the ophthalmologic apparatus 110 and the viewer 150.

[0051] The ROM 264 or the storage device 254 stores an image processing program.

[0052] The ROM 264 or the storage device 254 is an example of a "memory" in the present disclosure. The CPU 262 is an example of a "processor" in the present disclosure. The image processing program is an example of a "program" in the present disclosure.

[0053] The server 140 stores each piece of data received from the ophthalmologic apparatus 110 in the storage device 254 .

[0054] The following describes various functions realized by the CPU 262 of the server 140 executing an image processing program. The image processing program executed by the CPU 262 includes a display control function, an image processing function, and various components that realize the processing functions. Specifically, by executing the image processing program, the CPU 262 operates as a display control unit 204 that realizes the display control function, an image processing unit 206 that realizes the image processing function, and a processing unit 208 that realizes the processing function, all of which are shown in FIG. 4 .

[0055] Next, a main flowchart of image processing by the server 140 will be described with reference to Fig. 5. The CPU 262 of the server 140 executes an image processing program to realize the image processing (image processing method) shown in Fig. 5.

[0056] First, in step S10, the image processing unit 206 acquires a fundus image from the storage device 254. The fundus image includes data related to choroidal blood vessels including vortex veins to be displayed in three dimensions, based on a user's instruction.

[0057] Next, in step S20, the image processing unit 206 acquires, from the storage device 254, OCT volume data that is data related to choroidal blood vessels corresponding to the fundus image.

[0058] After acquiring the OCT volume data, the image processing unit 206 performs choroid-scleral boundary detection processing to detect the boundary between the choroidal blood vessels and the sclera in step S22 as preprocessing for the process of extracting choroidal blood vessels, which will be described later.

[0059] In the next step S30, the image processing unit 206 extracts choroidal blood vessels based on the OCT volume data and performs image formation processing of the choroidal blood vessels to generate a stereoscopic image (3D image) of the choroidal blood vessels including the vortex veins and the detected boundaries. The image formation processing of the choroidal blood vessels will be described later.

[0060] When the stereoscopic image (3D image) of the choroidal vessels including the vortex vein vessels is generated, in step S40, the processing unit 208 outputs the generated stereoscopic image (3D image) of the choroidal vessels including the vortex vein vessels, specifically, stores it in the RAM 266 or the storage device 254, and then ends the image processing.

[0061] Here, based on a user instruction, a display screen (an example of the display screen is shown in FIG. 19 , which will be described later) containing a stereoscopic image of choroidal vessels including vortex veins is generated by the display control unit 204. The generated display screen is output as an image signal by the processing unit 208 to the viewer 150. The display screen is displayed on the display of the viewer 150.

[0062] Here, the positional relationship between the choroid 12M and the vortex veins 12V1 and V2 in the eyeball will be described with reference to FIG. 6 . In FIG. 6 , the mesh-like pattern represents the choroidal blood vessels of the choroid 12M. The choroidal blood vessels circulate blood throughout the entire choroid. Blood flows out of the eyeball through multiple vortex veins (usually four to six) present in the subject's eye 12. FIG. 6 shows the superior vortex vein 12V1 and the inferior vortex vein 12V2 present on one side of the eyeball. Vortex veins are often located near the equator. Therefore, to photograph the vortex veins present in the subject's eye 12 and the choroidal blood vessels around the vortex veins, an ophthalmic apparatus 110 capable of scanning at an internal illumination angle of 200° is used, for example.

[0063] The image processing unit 206 acquires a fundus image and identifies vortex veins (VV) to be displayed in three dimensions. Here, as an example, a UWF-SLO image is acquired from the storage device 254 as a UWF fundus image. Next, the image processing unit 206 creates a choroidal vessel image, which is a binarized image, from the acquired UWF-SLO image. Then, the image processing unit 206 identifies a region designated by the user as a vortex vein to be displayed in three dimensions.

[0064] Fig. 8 is a fundus image of choroidal vessels including vortex veins. The fundus image shown in Fig. 8 is an example of a choroidal vessel image, which is a binarized image created from a UWF-SLO image. As shown in Fig. 8, the choroidal vessel image is a binarized image in which pixels corresponding to choroidal vessels and vortex veins are white and pixels in other areas are black.

[0065] 8 shows an image 302 indicating the presence of choroidal vessels connected to a vortex vein. The image 302 shows a case in which a vortex vein 310V1, which is an image of an upper vortex vein 12V1 included in a user-specified area 310A, is identified as a vortex vein (VV) to be displayed in three dimensions, and a region including the choroidal vessels is identified.

[0066] A choroidal vessel image including vortex veins (VVs) is generated by processing image data of an R-UWF-SLO image (R-color fundus image) captured with red light (laser light with a wavelength of 630 to 660 nm) and a G-UWF-SLO image (G-color fundus image) captured with green light (laser light with a wavelength of 500 to 550 nm). Specifically, the choroidal vessel image is generated by extracting retinal vessels from the G-color fundus image, removing retinal vessels from the R-color fundus image, and performing image processing to enhance the choroidal vessels. The disclosure of International Publication WO 2019 / 181981 regarding a method for generating a choroidal vessel image is incorporated herein by reference in its entirety.

[0067] Although the above describes a case in which a vortex vein to be displayed in three dimensions is identified by a user's instruction, the present disclosure is not limited to this. The position of the vortex vein to be displayed in three dimensions may be detected manually or automatically. For example, in the case of manual detection, the position indicated by the user's visual inspection of the displayed choroidal vessels may be detected. In the case of automatic detection, for example, the choroidal vessels may be extracted from a choroidal vessel image, the movement direction of each choroidal vessel (the direction of blood vessel travel) may be estimated, and the position of the vortex vein may be estimated based on the position where the choroidal vessels converge.

[0068] 7 , the OCT volume data 400 is OCT volume data 400 of a predetermined area, for example, a rectangular area of ​​6 mm × 6 mm, including the vortex vein VV, obtained by OCT imaging of one of the multiple vortex veins VV present in the subject's eye using the ophthalmic apparatus 110. N planes at different depths, from a first plane f401 to an Nth plane f40N, are set for the OCT volume data 400. The OCT volume data 400 may be obtained by OCT imaging of each of the multiple vortex veins VV present in the subject's eye using the ophthalmic apparatus 110.

[0069] In this embodiment, the OCT volume data 400D is described as an example of OCT volume data 400 including vortex veins and choroidal blood vessels around the vortex veins. In this case, the choroidal blood vessels include the vortex veins and choroidal blood vessels around the vortex veins.

[0070] The boundary between a region where choroidal blood vessels exist and a region where choroidal blood vessels do not exist, such as the sclera, is important for an observer, such as a doctor, to understand the condition of the subject's eye. However, variations in the thickness of choroidal blood vessels can result in both detection and non-detection of blood vessels. For example, in image processing to extract blood vessels from a fundus image, image noise may be extracted as blood vessels, resulting in the extraction of non-existent blood vessels. On the other hand, even if an image shows blood vessels, the image processing may not extract existing blood vessels, resulting in the extraction of blood vessels that should have been detected. Therefore, the position of the above-mentioned boundary may vary depending on the presence or absence of choroidal blood vessels. Therefore, there is room for improvement in determining the above-mentioned boundary using image processing.

[0071] In terms of brightness in a fundus image, the change in luminance at the boundary between the area where choroidal blood vessels are present and the area where they are not present, specifically the interface between the choroidal blood vessels and the sclera, is more abrupt than the change in luminance at other areas on the test eye. Therefore, the image processing of the present disclosure, which extracts luminance or luminance changes as features from the area including the choroidal blood vessels of the test eye, makes it possible to detect the interface between the choroidal blood vessels and the sclera.

[0072] The above-described image luminance and luminance change are examples of image features of the present disclosure. The image feature indicates a physical quantity related to the brightness of an image, and physical quantities such as brightness, a brightness change tendency, and brightness-related entropy are also applicable. In this embodiment, for simplicity of explanation, a case will be described in which luminance and luminance change are used as examples of physical quantities related to the brightness of an image as image features.

[0073] The feature indicating the luminance change in the embodiment is an example of a physical quantity indicating the brightness change in the present disclosure, and is also an example of a physical quantity indicating the luminance change in the tomographic image in the present disclosure. By using the luminance change in the embodiment, the brightness change can be easily detected.

[0074] Next, with reference to FIG. 9 , the choroid-scleral boundary detection process using image processing according to the present disclosure will be described. The CPU 262 of the server 140 executes the processing routine shown in FIG. 9 to realize the choroid-scleral boundary detection process. The image processing unit 206 executes feature extraction processing to extract image features of the subject's eye when detecting the boundary surface between the choroidal blood vessels and the sclera. Specifically, the CPU 262 of the server 140 executes the processing routine shown in FIG. 9 as the choroid-scleral boundary detection process of step S22 shown in FIG. 5 to realize the choroid-scleral boundary detection process. In step S23, the image processing unit 206 executes two-dimensional image feature extraction processing (hereinafter referred to as 2D processing) on ​​the fundus image, and in step S24, executes three-dimensional image filtering processing (hereinafter referred to as 3D processing). In step S23, 2D processing is performed using image processing such as 2D image filtering to detect the boundary surface. In step S24, 3D processing is performed using image processing such as 3D image filtering to detect the boundary surface.

[0075] The 2D processing and 3D processing in steps S23 and S24 described above are not limited to being performed in both cases, and boundary surfaces may be detected by performing at least one of them.

[0076] Next, the 2D processing executed in step S23 of Fig. 9 will be described with reference to Fig. 10. The 2D processing shown in the flowchart of Fig. 10 is realized by the CPU 262 of the server 140 executing an image processing program. Fig. 11 shows an example of an image resulting from image processing when each image processing step in the 2D processing is executed.

[0077] In step S230, the image processing unit 206 acquires OCT data for 2D processing (e.g., image 230G in FIG. 11 ). Here, OCT volume data 400, which is OCT data, is applied for choroid-sclera boundary detection processing. The OCT volume data 400 includes multiple tomographic image data obtained by scanning. The data acquired in step S230 includes, for example, depth-direction tomographic images (e.g., multiple A-scan data and B-scan data). Image 230G in FIG. 11 shows an example of OCT data for 2D processing acquired by the image processing unit 206. Image 230G includes a background image 230H.

[0078] The OCT volume data 400 also includes data of en-face images, which are cross sections of the OCT volume data, that is, front images generated based on the OCT volume data. In the data of the multiple en-face images, N planes having different depths are set from a first plane f401 to an Nth plane f40N.

[0079] In step S230, the image processing unit 206 can, for example, set a specific layer on the depth-direction tomographic image. Specifically, the image processing unit 206 can analyze the OCT volume data 400 and set a retinal pigment epithelium cell layer (hereinafter referred to as the RPE layer) in the OCT volume data 400. The RPE layer 400R can be identified by performing a predetermined segmentation process on the OCT volume data 400. Alternatively, the RPE layer may be identified as a layer with a brightness exceeding a predetermined threshold in the OCT volume data 400, for example, the layer with the highest brightness. The RPE layer has a higher brightness than other layers.

[0080] Setting the RPE layer is effective for identifying the area where choroidal blood vessels exist, since the area deeper than the RPE layer (the area farther from the center of the eyeball) corresponds to the choroidal area. Instead of the RPE layer, for example, Bruch's membrane, which exists immediately below the RPE layer, may be set. Bruch's membrane can also be identified by performing a predetermined segmentation process on the OCT volume data 400 that is different from that for the RPE layer. Surfaces may be set at positions other than the RPE layer or Bruch's membrane, for example, at any number of pixels and distance. Furthermore, a spherical surface at a fixed distance from the pupil or the center of the eyeball may be set as the reference surface.

[0081] The process of step S230 executed by the image processing unit 206 is an example of the process by an acquisition unit of the present disclosure. That is, the image processing unit 206 operates as an acquisition unit that acquires OCT volume data representing a tomographic image in the depth direction including the choroidal blood vessels and the sclera.

[0082] In step S231, the image processing unit 206 performs background effect suppression processing on each of the multiple tomographic images in the optical axis direction (i.e., depth direction) for each tomographic image (e.g., image 231G in FIG. 11 ). This background effect suppression processing is processing to remove background regions from the acquired image. Specifically, for example, regions of the 2D image acquired in step S230 having pixel values ​​with brightness values ​​exceeding a predetermined threshold are designated as background regions and replaced with a predetermined pixel value (e.g., “0”) so as not to affect the image processing described below. Note that this background effect suppression processing may also apply image processing such as noise removal. The processed image after the background effect suppression processing is stored in RAM 266 by the processing unit 208. Image 231G in FIG. 11 shows an example of an image after the background effect suppression processing. In image 231G, the background image 230H has been removed. Note that the background effect suppression processing performed in step S231 is not essential and may be omitted.

[0083] In step S232, the image processing unit 206 executes a luminance variation enhancement process (for example, image 232G in FIG. 11 ). Here, a process is executed to enhance luminance variations in the image using a predetermined 2D filter. For the luminance variation enhancement process, an enhancement process such as image convolution using a 2D filter can be applied. The image after the luminance variation enhancement process is stored in RAM 266 by the processing unit 208. Image 232G in FIG. 11 shows an example of an image after the luminance variation enhancement process.

[0084] Here, with reference to FIG. 12 , the 2D filter 23F used in the luminance variation enhancement process will be described. The 2D filter 23F is a filter including a region in which the filtering distribution of the light transmission distribution or the light absorption distribution changes continuously or stepwise from a first value (e.g., a minimum value) to a second value (e.g., a maximum value) in the optical axis direction (e.g., from one side of the filter to the other side). In this embodiment, the 2D filter 23F includes three types of filters, 23F1, 23F2, and 23F3, each having a different filtering distribution direction. The 2D filter 23F1 has a filtering distribution formed in a direction in which the depth direction is 0 degrees. The 2D filter 23F2 has a filtering distribution formed in a direction inclined at a predetermined angle θ (e.g., 30 degrees) from the depth direction. The 2D filter 23F3 has a filtering distribution formed in a direction inclined at a predetermined angle −θ (e.g., −30 degrees) in the opposite direction from the depth direction.

[0085] The 2D filter 23F may be a filter having a predetermined filtering distribution that is smaller than the size of the target image (i.e., the tomographic image) and that enhances brightness variations. In this embodiment, the 2D filter 23F is a filter formed by a filtering distribution 23Fx1 whose characteristics continuously change along a curve defined by a cosine function (cos function). That is, the filtering distribution 23Fx1 has a distribution characteristic that continuously changes from a first value (minimum value) to a second value (maximum value) according to a cosine function. In the filtering distribution 23Fx1, a signal value indicating a physical quantity related to the transmission or absorption of light in the 2D filter 23F is used as the filter density, and a correspondence relationship between the density and the position in the optical axis direction of the filter is shown. The characteristics of the filtering distribution are not limited to characteristics along a curve defined by a cosine function (cos function), and other functions may also be used. For example, a characteristic that changes in multiple steps may be used, as shown by a filtering distribution 23Fx2. Alternatively, a characteristic that changes linearly may be used, as shown by a filtering distribution 23Fx3. Furthermore, although these filtering distributions are shown as examples in which the distributions change continuously, the present invention is not limited to this, and filters that change intermittently may also be used. Note that Fig. 12 shows, as an example of the 2D filter 23F, a characteristic in which the signal value indicating the density has both positive and negative values. The 2D filter 23F may also be a filter in which the signal value indicating the density has either a positive or negative value.

[0086] In step S232, the image processing unit 206 performs brightness variation enhancement processing for each tomographic image using the 2D filters 23F1, 23F2, and 23F3. Specifically, the image processing unit 206 performs processing to enhance brightness variations by filtering the tomographic image sequentially using each of the 2D filters 23F1, 23F2, and 23F3. The processing order may be any order as long as the processing uses the 2D filters 23F1, 23F2, and 23F3. Alternatively, the 2D filters 23F1, 23F2, and 23F3 may be independently processed and then combined.

[0087] In the above description, the 2D filters 23F1, 23F2, and 23F3 are applied as the 2D filter 23F, but it goes without saying that any filter that enhances brightness changes may be used, and the 2D filter is not limited to these filters.

[0088] Furthermore, the image processing unit 206 may perform the processes in steps S231 and S232 in reverse order.

[0089] By performing a brightness change enhancement process such as image convolution processing using the 2D filter 23F, an image in which brightness changes are enhanced on the tomographic image is obtained, as shown in image 232G in Figure 11, for example, image 232G including image 232Gx in which areas where brightness changes sharply corresponding to a predetermined density change are enhanced.

[0090] The brightness change emphasis process according to the embodiment is an example of the emphasis process of the present disclosure. When the degree of brightness change exceeds a predetermined degree, the emphasis process can emphasize the brightness change to a greater degree than before the emphasis process. According to the embodiment, the brightness change can be easily derived.

[0091] The enhancement process according to the embodiment corresponds to performing enhancement processing by filtering to enhance a physical quantity related to brightness in a region of a predetermined size smaller than the size of a tomographic image. In this way, local enhancement makes it possible to enhance brightness changes in the smallest unit.

[0092] Furthermore, the tomographic image is a two-dimensional image, and the filtering process corresponds to performing enhancement processing using a two-dimensional filter in which the brightness that is enhanced gradually changes from one side to the other in the two-dimensional region of the 2D filter 23F.

[0093] Furthermore, it is possible to repeatedly perform enhancement processing on a portion of the tomographic image, i.e., the two-dimensional region of the 2D filter 23F, in relation to brightness changes in the depth direction in the tomographic image, within the tomographic image, thereby enabling enhancement processing to be applied to all regions in the tomographic image.

[0094] In step S233, the image processing unit 206 performs binarization processing on each tomographic image (for example, image 233G in FIG. 11 ). The image processing unit 206 acquires a predetermined threshold value stored in the ROM 264 or the storage device 254 for detecting the choroid-scleral boundary, and performs binarization processing using the threshold value. The image after binarization processing is stored in the RAM 266 by the processing unit 208. Image 233G in FIG. 11 shows an example of an image after binarization processing.

[0095] By performing the binarization process, as shown in image 233G in FIG. 11, brightness changes are emphasized on the tomographic image, and an image 233G including a binarized image 233Gx is obtained.

[0096] Although the above description is given of the case where the image processing unit 206 performs binarization processing in step S233, the processing in step S233 is not limited to binarization processing. For example, in step S233, the image processing unit 206 may perform edge detection processing to detect peak positions in the depth direction. This is because the processing in step S233 only needs to be processing that emphasizes region separation for detecting the boundary between the choroidal blood vessels and the sclera.

[0097] In step S234, the image processing unit 206 performs area filtering on each tomographic image (e.g., image 234G in FIG. 11 ). The image processing unit 206 acquires information indicating a predetermined area, which is stored in the ROM 264 or the storage device 254 for use in choroid-scleral boundary detection, and removes the predetermined area from the binarized image 233G. Specifically, the image processing unit 206 removes pixels exceeding a predetermined brightness as image components and areas where the size of a pixel group formed by consecutive pixels is equal to or smaller than a predetermined area, as areas not subject to choroid-scleral boundary detection, from the binarized image 233G. The processed image after area filtering is stored in the RAM 266 by the processing unit 208. Image 234G in FIG. 11 shows an example of an image after area filtering.

[0098] By performing the area filtering process, an image 234G including an image 234Gx from which minute image regions have been removed is obtained, as shown in image 234G in FIG.

[0099] In step S235, the image processing unit 206 performs a removal process for each tomographic image (e.g., image 235G in FIG. 11 ). The image processing unit 206 acquires information indicating a predetermined region stored in the ROM 264 or the storage device 254 for the removal process, and removes the predetermined region from the image 234G after the area filter process. The predetermined region used in this removal process can be determined empirically or anatomically as a region that is not subject to choroid-sclera boundary detection. For example, the image processing unit 206 removes a region extending from a position deeper than a predetermined position in the depth direction from the center of the region shown in the image 234Gx after the area filter process has been performed. More specifically, as shown in image 235G in FIG. 11 , the image of the region deeper than a removal line 235Gy indicating the end of the region a predetermined distance deeper from image 234Gx (the region indicated by the hatched line in FIG. 11 ) is removed.

[0100] The image processing unit 206 may perform the above-described processes of step S234 and step S235 in reverse order.

[0101] The processes of steps S231 to S235 executed by the image processing unit 206 are an example of processes by a derivation unit of the present disclosure. That is, the image processing unit 206 operates as a derivation unit that executes enhancement processing for enhancing a physical quantity indicating a brightness change in the depth direction in a partial region of a tomographic image based on OCT volume data, and derives an image feature quantity related to the brightness change in the depth direction in the enhanced region.

[0102] In step S236, the image processing unit 206 executes a boundary surface determination process to determine a boundary surface for the entire three-dimensional tomographic image as described below (for example, image 236G in FIG. 11 ), and ends this processing routine. Information indicating the boundary surface determined in the boundary surface determination process is stored in RAM 266 by the processing unit 208. Note that the processed image in which the boundary surface determination process has been executed may also be stored in RAM 266 by the processing unit 208. Image 236G in FIG. 11 shows an example of an image in which the boundary surface determination process has been executed. Furthermore, image 23G in FIG. 11 shows an example of an image in which the determined boundary surface 23L is superimposed on the above-described acquired image 230G.

[0103] In step S236, the image processing unit 206 determines a boundary surface using all two-dimensional images for 2D processing, i.e., the entire tomographic image, which is a three-dimensional image. Specifically, the image processing unit 206 sums the multiple images 235G after the removal process in a direction intersecting the depth direction. For example, the image processing unit 206 sums the B-scan images after the process. This makes it possible to reflect regions of a predetermined brightness and regions of brightness variation in the entire tomographic image (image 236Gx in FIG. 11 ). Next, the image processing unit 206 sets the deepest position of the region of a predetermined brightness and regions of brightness variation as the boundary. This deepest position of the boundary is reflected in the entire tomographic image, which is a three-dimensional image, and therefore becomes the boundary surface (boundary surface 23L in FIG. 11 ).

[0104] The processing of step S236 executed by the image processing unit 206 is an example of processing by a determination unit of the present disclosure. That is, the image processing unit 206 operates as a determination unit that determines, based on the image feature amount, a region on a tomographic image where the image feature amount exceeds a predetermined threshold as a boundary between a region where choroidal blood vessels are present and a region where choroidal blood vessels are not present, and as a boundary between the choroidal blood vessels and the sclera.

[0105] Therefore, the image processing unit 206 can determine the area between the choroidal blood vessels and the sclera as the boundary in the area that does not include the choroidal blood vessels.

[0106] The process of step S236 described above corresponds to the following: when a plurality of regions on a tomographic image are derived in which the image feature value indicating the amount of brightness change or the degree of brightness change exceeds a predetermined threshold, the derived regions are set as a plurality of boundary candidates, and one of the plurality of boundary candidates (for example, the deepest position is preferable) is determined as the boundary on the tomographic image. Therefore, by determining one boundary candidate, the processing load for determining the boundary surface can be reduced.

[0107] In the above, binarization processing is performed in step S233, and the boundary surface is determined using the image after binarization processing. However, without performing steps S233 to S235, the boundary surface determination in step S236 may be performed using the image after the luminance change enhancement using the 2D filter in step S232.

[0108] Next, the determination of the boundary that becomes the above-mentioned boundary surface 23L from the image after the processing of step S232 will be further described with reference to Fig. 13. Fig. 13 is a diagram showing the relationship between the depth position and the signal intensity on the output image after the processing of step S232. The signal intensity is determined based on the input value of the image feature amount such as brightness in the tomographic image, and includes a value obtained by differentiating the input value, which is the original signal.

[0109] As shown in FIG. 13 , in the output image after processing in step S232, the signal intensity gradually increases from a shallow position to a predetermined depth Dpth, and gradually decreases as the position increases from the predetermined depth Dpth. In terms of signal intensity in the depth direction, a region where the signal intensity exceeds a predetermined intensity Ik is likely to correspond to a region of blood vessels or the like, and a position deeper than that region is likely to be the boundary. That is, the boundary between the choroid and the sclera is likely to be located at a position where the signal intensity changes sharply deeper than the region where the signal intensity Ik is exceeded. Therefore, in this embodiment, the position where the signal intensity changes sharply, i.e., the position Dpth where the signal intensity reaches the maximum signal intensity Imax, is determined as the boundary. Furthermore, the boundary is not limited to the position of maximum signal intensity, and a position at a predetermined depth from the position Dpth where the signal intensity Imax is reached may also be determined as the boundary. For example, the boundary may be defined as a position between a position where the signal intensity becomes smaller than the maximum signal intensity Imax and a position Dpd at a depth equal to or less than a predetermined signal intensity Id, which is deeper than the position of the maximum signal intensity Imax. Alternatively, the boundary between the choroid and the sclera may be defined as a position that is a predetermined depth (+ the predetermined depth) deeper than the position determined as the boundary so as not to include choroidal blood vessels.

[0110] Setting a position (predetermined depth) that is a predetermined depth from the position determined as the boundary as described above corresponds to setting a position at a predetermined depth from a portion on a tomographic image where an image feature amount, such as a change in luminance indicated by signal intensity, exceeds a predetermined threshold, as the boundary. This makes it possible to reliably determine the boundary of a portion that does not include choroidal blood vessels.

[0111] The above-mentioned boundary can be determined by applying a gradient wd that indicates the differential value of the characteristic curve of the depth and the signal intensity from the viewpoint of the change in the signal intensity. That is, the boundary may be determined as a position where the signal intensity such as brightness becomes smaller than the signal intensity Imax and has a predetermined gradient wd at a position deeper than the position of the signal intensity Imax.

[0112] In the above description, the boundary is determined based on a position where the signal intensity changes sharply at a position deeper than the region where the signal intensity exceeds Ik, but the disclosed technology is not limited to this. For example, the position where the signal intensity reaches the maximum signal intensity Imax may be set as the boundary candidate position Dpth, and a position at a depth that is a predetermined depth that is sufficient to not include choroidal blood vessels may be set as the boundary position from the boundary candidate position Dpth in the depth direction. For example, a position Dpu at a depth where the signal intensity gradually increases and a predetermined depth that is sufficient to not include choroidal blood vessels may be set as the boundary candidate position Dpu. In this case, a position with a predetermined slope wu may be set as the boundary candidate position Dpu.

[0113] Furthermore, although the image processing unit 206 sets the deepest position 23L as the boundary (boundary surface 23L in FIG. 11 ), the boundary may be set so as to follow a region of a predetermined signal strength and a region having a change in signal strength. For example, as shown in FIG. 14 , a deep position in the depth direction of an image 236Gx showing a region of a predetermined signal strength and a region having a change in signal strength in an image 236Gb obtained by summing the processed B-scan images, or a position a predetermined depth from that position, may be set as boundary surface 23Lb. Compared to setting the deepest position as boundary surface La in image 236Ga, which is image 236G, boundary surface 23Lb allows for a curved surface to be set with a higher degree of freedom, making it possible to determine the boundary surface between the choroid and the sclera with high accuracy.

[0114] Setting the boundary surface to follow the region of a predetermined signal intensity and the region having a signal intensity change corresponds to, when multiple regions on a tomographic image are derived in which image features indicating signal intensity changes or the like exceed a predetermined threshold, setting the derived regions as multiple boundary candidates and determining a region connecting the multiple boundary candidates as the boundary on the tomographic image. In this way, it is possible to determine a boundary surface that follows the region between the choroidal blood vessels and the sclera. When setting the boundary surface to follow the region having a signal intensity change, it may be approximated by a curved surface or polygonal surface obtained by extending a curve or polygonal line that approximates the contour of the region having a signal intensity change.

[0115] In this way, the image processing unit 206 can determine the boundary surface between the choroid and the sclera with high accuracy by 2D processing. That is, by performing the image processing by the 2D processing described above, the image processing unit 206 can determine the boundary surface between the choroidal vessels and the sclera, and by superimposing the boundary surface on the choroidal vessel image, the area corresponding to the boundary between the vessels and the sclera can be visualized (see image 23G in FIG. 11 ).

[0116] Although the above description deals with the case where the boundary surface between the choroid and the sclera is determined by 2D processing in the image processing unit 206, the present disclosure is not limited to this. For example, the position of the determined boundary surface may be set as a boundary surface candidate position, and the set boundary surface candidate position may be changed in response to a user instruction. That is, information indicating an instruction to change the position set as the boundary surface candidate position to a determined position desired by the user may be acquired, and the position corresponding to the acquired instruction may be determined as the final determined position of the boundary surface.

[0117] Next, the 3D processing executed in step S24 of Fig. 9 will be described with reference to Fig. 15. The 3D processing shown in the flowchart of Fig. 15 is realized by the CPU 262 of the server 140 executing an image processing program. Fig. 16 shows an example of an image resulting from image processing when each image processing step in the 3D processing is executed. The 3D processing is a process in which the above-mentioned 2D processing is extended to three dimensions.

[0118] In step S240, the image processing unit 206 acquires OCT data for 3D processing (e.g., image 240G in FIG. 16 ). The data acquired in step S240 may be data processed by the above-described 2D processing, or the OCT volume data 400, which is OCT data, may be applied for 3D processing. When applying data processed by the above-described 2D processing to the data acquired in step S240, the data processed in step S235 in FIG. 10 may be applied.

[0119] In step S241, the image processing unit 206 executes a luminance variation enhancement process similar to step S232 in FIG. 10 (for example, image 241G in FIG. 16 ). Here, a process of enhancing luminance variations in a three-dimensional image is executed using a predetermined 3D filter. Hereinafter, three dimensions may be referred to as 3D. The 3D image after the luminance variation enhancement process is executed is stored in RAM 266 by the processing unit 208. Image 241G in FIG. 16 shows an example of a 3D image after the luminance variation enhancement process is executed in 3D.

[0120] In step S242, the image processing unit 206 performs background effect suppression processing on the 3D image (e.g., image 242G in FIG. 16 ). This background effect suppression processing is an extension of the processing in step S231 in FIG. 10 to 3D, and is processing to remove background regions from the 3D image. Specifically, similar to step S231 in FIG. 10 , a process is performed in which, for example, 3D regions of the 3D image having pixel values ​​with brightness values ​​exceeding a predetermined threshold are designated as background regions and replaced with a predetermined pixel value (e.g., “0”). The image after the background effect suppression processing is stored in the RAM 266 by the processing unit 208. Image 242G in FIG. 16 shows an example of a 3D image after the background effect suppression processing. Image 242G shows an image in which the background image 240H has been removed and brightness changes have been emphasized.

[0121] Here, with reference to FIG. 17 , the 3D filter 24F used in the 3D luminance variation enhancement process will be described. The 2D filter 23F is a filter whose 3D filtering distribution, which is a light transmission distribution or a light absorption distribution, changes continuously or stepwise from a first value (e.g., a maximum value) to a second value (e.g., a minimum value) in the optical axis direction (e.g., from one side of the filter to the other). In this embodiment, the 3D filter 24F includes five types of filters, 24F1, 24F2, 24F3, 24F4, and 24F5, each having a different filtering distribution direction. The 3D filter 24F1 has a 3D filtering distribution formed in a direction in which a predetermined direction (e.g., the depth direction) is set to 0 degrees. The 3D filter 24F2 has a filtering distribution formed in a direction inclined at a predetermined angle θ (e.g., 30 degrees) from the depth direction. The 3D filter 24F3 has a 3D filtering distribution formed in a direction inclined at an angle 2θ (e.g., 60 degrees), which is twice as large as that of the 3D filter 24F2. Similarly, the 3D filter 24F4 has a 3D filtering distribution formed in a direction tilted at a predetermined angle −θ (for example, −30 degrees) in the opposite direction from the depth direction, while the 3D filter 24F5 has a 3D filtering distribution formed in a direction tilted at twice the angle −2θ (for example, −60 degrees).

[0122] As with the 2D filter 23F, the 3D filter 24F is formed with a filtering distribution that is smaller than the size of the target 3D image and that is predetermined to emphasize brightness changes. That is, the 3D filter 24F is formed with a filtering distribution that continuously changes in characteristics along a curve defined by a cosine function (cos function) in each predetermined direction.

[0123] In step S241, the image processing unit 206 executes 3D luminance variation enhancement processing on the 3D image using the 3D filters 24F1 to 24F5.

[0124] In the above, five types of filters, 3D filter 23F1 to 3D filter 24F5, are applied as 3D filter 24F, but it goes without saying that any filter that emphasizes brightness changes may be used and the present invention is not limited to these filters.

[0125] Furthermore, the image processing unit 206 may perform the processes in steps S241 and S242 in reverse order.

[0126] By performing the brightness change enhancement process using the 3D filter 24F, an image in which brightness changes are enhanced on the 3D image is obtained, as shown in image 242G in Figure 16, for example, image 242G including image 242Gx in which a 3D area in which brightness changes sharply corresponding to a predetermined density change is enhanced.

[0127] The processing of step S241 corresponds to performing an enhancement process using a three-dimensional filter in which the tomographic image is a three-dimensional image and the area defined by the 3D filter 24F is treated as a three-dimensional area and the brightness gradually changes from one side to the other to enhance the area.

[0128] In step S243, the image processing unit 206 performs binarization processing on the 3D image (e.g., image 243G in FIG. 16 ), similar to step S233 in FIG. 10 . The image processing unit 206 acquires a predetermined threshold value stored in the ROM 264 or the storage device 254 for 3D processing, and performs 3D binarization processing using the threshold value. The image after binarization processing is stored in the RAM 266 by the processing unit 208. Image 243G in FIG. 16 shows an example of an image after 3D binarization processing.

[0129] By performing the 3D binarization process, as shown in image 243G in FIG. 16, brightness changes are emphasized on the 3D image, and an image 243G including a binarized 3D image 243Gx is obtained.

[0130] In step S244, the image processing unit 206 performs volumetric filtering on the 3D image (e.g., image 244G in FIG. 16 ). The image processing unit 206 acquires information indicating a predetermined region having a predetermined volume stored in the ROM 264 or the storage device 254 for 3D, and removes the predetermined region from the binarized 3D image 243G. Specifically, the image processing unit 206 removes pixels exceeding a predetermined brightness as image components and regions where the size of a pixel group formed by consecutive pixels is equal to or smaller than a predetermined volume from the binarized image 243G as regions not subject to boundary detection. The 3D image after volumetric filtering is stored in the RAM 266 by the processing unit 208. Image 244G in FIG. 16 shows an example of a 3D image after volumetric filtering.

[0131] By performing the volumetric filtering process, a 3D image 244G including an image 244Gx from which minute image regions have been removed is obtained, as shown in image 244G in FIG.

[0132] In step S245, the image processing unit 206 performs a removal process on the 3D image (e.g., image 245G in FIG. 16 ). The image processing unit 206 acquires information indicating a predetermined region stored in the ROM 264 or the storage device 254 for the removal process, and removes the predetermined region from the image 244G after the volumetric filtering process. The predetermined region applied in this removal process can also be determined empirically or anatomically as a region not subject to boundary detection. For example, the image processing unit 206 removes a region extending from a position deeper than a predetermined position in the depth direction from the center of the region shown in the image 244Gx after the volumetric filtering process. More specifically, as shown in image 245G in FIG. 16 , the image of the region deeper than a removal line 245Gy indicating the end of the region a predetermined distance deeper from image 244Gx (the region indicated by the hatched line in FIG. 16 ) is removed.

[0133] The image processing unit 206 may perform the above-described steps S244 and S245 in a reversed order.

[0134] In step S246, the image processing unit 206 executes a boundary surface determination process to determine a boundary surface for the entire 3D image (e.g., image 246G in FIG. 16 ), and then terminates this processing routine. That is, the image processing unit 206 determines the boundary surface 24L by setting the deepest position as the boundary in the 3D image after the removal process. Information indicating the boundary surface determined in the boundary surface determination process is stored in the RAM 266 by the processing unit 208. Note that the processed image after the boundary surface determination process may also be stored in the RAM 266 by the processing unit 208. Image 246G in FIG. 16 illustrates an example of an image after the boundary surface determination process has been executed. Furthermore, image 24G in FIG. 16 illustrates an example of a 3D image in which the determined boundary surface 24L is superimposed on the acquired 3D image 240G described above.

[0135] Since the determination of the 3D boundary surface can be realized by extending the determination process in the 2D process described above to three dimensions, detailed description thereof will be omitted. Note that, as in the 2D process described above, the boundary surface determination in step S246 may be performed using the 3D image after the luminance change enhancement by the 3D filter in step S246, without executing steps S242 to S245.

[0136] In this way, the image processing unit 206 can determine the boundary surface between the choroidal blood vessels and the sclera with high accuracy by 3D processing. That is, by performing the image processing by the 3D processing described above, the image processing unit 206 can determine the 3D boundary surface between the choroidal blood vessels and the sclera, and by superimposing the 3D boundary surface on the choroidal blood vessel image, the 3D region corresponding to the boundary between the blood vessels and the sclera can be visualized (see image 24G in FIG. 16 ).

[0137] After the choroid-sclera boundary detection process described above is completed, the image processing unit 206 executes a process for extracting choroidal blood vessels. Next, the image formation process of choroidal blood vessels in step S30, which is executed by the image processing unit 206 and generates a stereoscopic image of the vortex veins (VV), will be described.

[0138] For choroidal vessel extraction, the image processing unit 206 extracts a region corresponding to the choroid from the OCT volume data 400 ( FIG. 7 ) and acquires OCT volume data of the choroid based on the extracted region. The OCT volume data may be acquired by extracting a portion of the OCT volume data scanned so as to include vortex veins and choroidal vessels surrounding the vortex veins. For example, OCT volume data 400D of a region below the RPE layer may be extracted. Alternatively, OCT volume data 400D of a region determined to contain vascular components in the vascular component presence / absence boundary acquisition process described above may be extracted.

[0139] Next, the image processing unit 206 executes a blood vessel extraction process using the OCT volume data 400D. The blood vessel extraction process includes a process of extracting choroidal blood vessels of various blood vessel diameters. Examples of the various blood vessel diameters include choroidal blood vessels of a predetermined size (hereinafter referred to as the ampulla), choroidal blood vessels that extend from the ampulla and exceed a predetermined diameter (hereinafter referred to as the large blood vessels), and choroidal blood vessels that extend from the ampulla and have a predetermined diameter or less (hereinafter referred to as the small blood vessels). Note that the large blood vessels are primarily located in the Haller layer, and the small blood vessels are primarily located in the Sattler layer.

[0140] As shown in FIG. 18 , the image processing unit 206 generates a three-dimensional image 681L of large blood vessels including the dilation and a three-dimensional image 681S of small blood vessels from the OCT volume data 400D. The image data of the three-dimensional images 681L and 681S are stored in the RAM 266 or the storage device 254 by the processing unit 208. Next, the image processing unit 206 reads the three-dimensional images 681L and 681S from the RAM 266, aligns these three-dimensional images, and performs a logical OR operation on each image to generate a three-dimensional image in which the dilation, large blood vessels, and small blood vessels are combined, i.e., a three-dimensional image 681M of the choroidal blood vessels including vortex veins. The image data of the three-dimensional image 681M is stored in the RAM 266 or the storage device 254 by the processing unit 208.

[0141] Choroidal vessels have a variety of diameters, ranging from the aforementioned large to small. Small vessels have lower image contrast than large vessels, and applying image processing designed for large vessels to the entire image may make it difficult to extract all vessels. Therefore, different image processing methods may be used to extract blood vessels according to their diameters, and the extracted images of each blood vessel diameter may be combined. That is, when generating a choroidal vessel image, an image processing unit may be used that extracts and combines choroidal vessels with different diameters to generate a choroidal vessel image. The image processing unit may be the image processing device described in International Publication WO 2023 / 199848. The disclosure of International Publication WO 2023 / 199848 is incorporated herein by reference in its entirety.

[0142] The image processing unit 206 also reads information indicating the boundary between the choroid and the sclera obtained by the above-described choroid-sclera boundary detection process ( FIG. 9 ) from the RAM 266 and combines it into a composite three-dimensional image. Image data of the three-dimensional image 681M including the information indicating the boundary between the choroid and the sclera is stored in the RAM 266 or the storage device 254 by the processing unit 208.

[0143] The following describes a display screen for displaying the generated stereoscopic image (3D image) of choroidal vessels including vortex veins. The display screen is generated by the display control unit 204 of the server 140 based on a user instruction, and is output as an image signal to the viewer 150 by the processing unit 208. The viewer 150 displays the display screen on the display based on the image signal.

[0144] A display screen 500A is shown in Figure 19. As shown in Figure 19, the display screen 500A has an information area 502 and an image display area 504A. The image display area 504A includes a comment field 506 that displays the patient's medical history.

[0145] The information area 502 has a patient ID display field 512, a patient name display field 514, an age display field 516, a visual acuity display field 518, a right eye / left eye display field 520, and an axial length display field 522. In each display area from the patient ID display field 512 to the axial length display field 522, the viewer 150 displays the respective information based on the information received from the server 140.

[0146] The image display area 504A is an area that mainly displays an image of the subject's eye, etc. The image display area 504A is provided with the following display fields, specifically, a UWF fundus image display field 542 and a choroidal blood vessel stereoscopic image display field 548. Although not shown, the image display area 504A can also display an OCT volume data conceptual diagram display field and a tomographic image display field 546 in a superimposed manner.

[0147] The comment field 506 included in the image display area 504A functions as a remarks field in which the patient's medical history is displayed and the ophthalmologist user can optionally input the results of his or her observations and diagnosis.

[0148] The UWF fundus image display field 542 displays a UWF-SLO fundus image 542B obtained by photographing the fundus of the subject's eye using the ophthalmologic apparatus 110. A range 542A indicating the position where the OCT volume data was acquired is superimposed on the UWF-SLO fundus image 542B. If there are multiple pieces of OCT volume data associated with the UWF-SLO image, the multiple ranges may be superimposed and displayed, allowing the user to select one position from the multiple ranges. Figure 19 shows that the range including the vortex vein in the upper right corner of the UWF-SLO image has been scanned.

[0149] The choroidal vessel stereoscopic image display field 548 displays a stereoscopic image (3D image) 548B of the choroidal vessels obtained by image processing the OCT volume data. The stereoscopic image 548B can be rotated around three axes by a user operation. The choroidal vessel stereoscopic image 548B can also display the dilation 548X, a stereoscopic image 548L of the choroidal vessels, which are thick blood vessels extending from the dilation 548X, and a stereoscopic image 548S of the choroidal vessels, which are thin blood vessels, in different display formats. The stereoscopic image 548L of the thick blood vessels and the stereoscopic image 548S of the thin blood vessels may be displayed in different colors, or the background (fill) of the image may be different.

[0150] Furthermore, the boundary surface between the choroid and the sclera, obtained by the choroid-sclera boundary detection process ( FIG. 9 ), can be superimposed and displayed in the choroidal vessel stereoscopic image display field 548. FIG. 19 shows an example in which the boundary surface is displayed as a mesh-like surface 548P. This boundary surface 548P is the boundary between the choroid and the sclera without dividing the choroid, allowing the identification of areas including the choroidal vessel region and enabling accurate treatment of the patient. It also makes it possible to perform quantitative measurements of blood vessel depth and the like with high precision.

[0151] The image display area 504A of the display screen 500A allows a user to view a stereoscopic image of the boundary between the choroid and the sclera. By scanning an area including a vortex vein, the vortex vein and its surrounding choroidal vessels can be displayed in a stereoscopic image. Furthermore, by superimposing the boundary between the choroidal vessels and the sclera, the user can obtain more information for diagnosis.

[0152] As described above, in this embodiment, the boundary between the choroid and the sclera can be obtained based on OCT volume data including the choroid, making it possible to faithfully visualize choroidal blood vessels in three dimensions.

[0153] In the above embodiment, an example has been described in which the boundary surface between the choroid and the sclera, acquired by the choroid-sclera boundary detection process ( FIG. 9 ), is superimposed on a stereoscopic image of the choroidal blood vessels. The technology of the present disclosure is not limited to superimposing the boundary surface between the choroid and the sclera on a stereoscopic image of the choroidal blood vessels, but can also be applied to the structure of the subject's eye, etc. Next, an application example will be described.

[0154] A first application example is to apply the technology of the present disclosure to displaying eyeball shapes such as myopia and staphyloma. Specifically, the above-described detection of the boundary between the choroid and sclera can be applied to a scannable area of ​​the subject's eye to estimate the eyeball shape and display the estimated eyeball shape. A second application example is to apply the technology to calculating the choroidal vascular layer thickness and the lumen ratio. A third application example is to estimate abnormalities in the eyeball shape by comparing the eyeball shape estimated in the first application example with a predetermined standard eyeball shape. It is also possible to display or re-photograph the abnormalities in the estimated eyeball shape for detailed examination. A fourth application example is to apply the technology to detection processes such as the position of the ampulla and the center of staphyloma from the eyeball shape estimated in the first application example. A fifth application example is to apply the technology to strain analysis from a structural mechanics perspective, such as determining where loads are applied to the eyeball. Each of the above application examples can be realized by detecting the boundary surface between the choroid and the sclera on the eyeball and estimating the eyeball shape.

[0155] In addition, the boundary surface between the choroid and the sclera in the above embodiment may be displayed as a boundary line in an image obtained by B-scan. It may also be displayed as a map showing the thickness of the boundary surface and other layers. The boundary surface may be displayed as an animation, starting from a planar initial state, and then changing from the initial state to a state of the boundary surface (e.g., a curved or polyhedral surface) that matches the region between the choroidal blood vessels and the sclera. Furthermore, it may be displayed three-dimensionally using VR display. It may also be displayed on a polarization OCT image or superimposed on the analysis results of polarization OCT. Furthermore, the numerical values ​​of the position, thickness, etc. described in the above-mentioned application examples may be displayed as quantitative values.

[0156] Second Embodiment Next, a second embodiment of the present disclosure will be described. Since the second embodiment has substantially the same configuration as the first embodiment described above, the same components are denoted by the same reference numerals and detailed descriptions thereof will be omitted. In the second embodiment, image processing is performed to display a blood vessel image in detail.

[0157] In the eyeball, for example, in the fundus, new structures, such as fine blood vessels, may be formed. Such blood vessels are sometimes called neovascularization. Although neovascularization can be observed by photographing the fundus, photographing may be difficult depending on the photographing conditions and the state of the subject's eye. On the other hand, even though neovascularization is, for example, fine blood vessels, it leaves traces on the photographed image. Therefore, in the second embodiment, an image that makes it easier to observe blood vessels, such as neovascularization, is formed by performing image enhancement processing on the photographed image.

[0158] In this embodiment, the technology of the present disclosure is applied to a case where a blood vessel image in a fundus image such as an SLO image is displayed instead of the above-described stereoscopic display of choroidal blood vessels including vortex veins.

[0159] Next, image processing by the server 140 according to this embodiment will be described with reference to Fig. 20. The image processing (image processing method) shown in Fig. 20 is realized by the CPU 262 of the server 140 executing an image processing program. This may also be applied to the image formation processing in the above embodiment (the processing of step S30 in Fig. 5).

[0160] In step S12, the image processing unit 206 acquires a fundus image from the storage device 254. The fundus image includes data of an SLO image including a vascular image related to the choroidal blood vessels and neovascularization to be displayed, based on a user instruction.

[0161] In the next step S32, the image processing unit 206 performs image formation processing of blood vessels in the fundus by emphasizing blood vessels related to choroidal blood vessels and neovascularization based on the data of the SLO image. The image formation processing of the blood vessels will be described later.

[0162] Next, in step S42, the processing unit 208 of the image processing unit 206 executes output processing to output an image (fundus image) including the highlighted blood vessels, and ends this processing routine. Here, the processing unit 208 stores the image after processing in step S32 in the RAM 266 or the storage device 254, and ends the output processing.

[0163] Next, the blood vessel image formation process executed in step S32 of Fig. 20 will be described with reference to Fig. 21. The blood vessel image formation process shown in the flowchart of Fig. 21 is realized by the CPU 262 of the server 140 executing an image processing program. Fig. 22 shows an example of an image resulting from image processing when each image processing step in the blood vessel image formation process is executed.

[0164] In step S320, the image processing unit 206 acquires fundus image data for forming a blood vessel image (e.g., image 320G in FIG. 22 ). Here, image data of a green fundus image from the SLO image is used as the fundus image data. Note that image data of another color, for example, a blue fundus image, may also be used as the fundus image data. The fundus image 320G includes at least a blood vessel image 320Gx (shown by dotted lines in FIG. 22 ) of retinal blood vessels such as neovascularization.

[0165] Next, in step S322, the image processing unit 206 performs image enhancement processing (e.g., image 322G in FIG. 22 ). Here, the image enhancement processing includes at least a blood vessel enhancement process that enhances blood vessels. The blood vessel enhancement process enhances at least a blood vessel image 320Gx that indicates a region (site) of retinal blood vessels, such as neovascularization, in a manner that allows it to be differentiated from images of other regions other than blood vessels. This makes it possible to obtain an image that includes a blood vessel image 320Gx (shown by a solid line in FIG. 22 ) in which at least the blood vessel image, such as neovascularization, is enhanced. The image after the image enhancement processing is stored in the RAM 266 by the processing unit 208. Image 322G in FIG. 22 shows an example of an image after the image enhancement processing, including a blood vessel image 322Gx.

[0166] When enhancing a blood vessel image such as neovascularization, it is preferable to perform image processing to remove or enhance structures formed on the fundus in order to improve the visibility of the blood vessel image. In this embodiment, as an example of preferable image processing, a case where each of first to fourth image enhancement processing is performed will be described.

[0167] FIG. 23 shows an example of the first to fourth image enhancement processes that can be performed in step S322. The first image enhancement process executes step S322-1, which describes image processing from the perspective of image frequency components, with the aim of forming an image that makes it easy to visually recognize the presence or absence of structures. Specifically, step S322-1 includes step S322-1A, which removes low-frequency components, and step S322-1B, which inverts the image. By removing low-frequency components in step S322-1, the image processing unit 206 can obtain an image that improves the visibility of the presence or absence of structures formed on the fundus. By inverting the image in step S322-1B, the image processing unit 206 can form a fundus image that shows a dark background, e.g., a black background, in the fundus region. The images after the first image enhancement process are preferably combined, e.g., added, to form the image after the image enhancement process. In FIG. 23, a "+" sign is written following step S322-1. Information indicating the synthesis of the images after the first image enhancement process, for example, information indicating a "+" sign, may be added to the images or may be determined in advance for the process of step S322-1.

[0168] The second image enhancement process executes step S322-2, which represents image processing for enhancing the vascular image of at least the retinal blood vessels, such as neovascularization, as described above. The image after the second image enhancement process is preferably synthesized, for example, added, as an image after the image enhancement process, and in FIG. 23, a "+" sign is written following step S322-2.

[0169] The third image enhancement process executes step S322-3, which represents image processing for enhancing white spot regions, which are regions of a predetermined brightness and a predetermined size on the image. The image after the third image enhancement process is preferably subjected to, for example, subtraction to obtain the image after the image enhancement process, and in FIG. 23, a "-" sign is written following step S322-3.

[0170] The fourth image enhancement process executes step S322-4, which represents image processing for enhancing a bleeding area, which is an area on the image that has a predetermined brightness corresponding to blood and exceeds a predetermined size. The image after the fourth image enhancement process is preferably added as the image after the image enhancement process, and in FIG. 23, a "+" sign is written following step S322-4.

[0171] The information indicating the "+" and "-" signs described above may be added to the image for each image processing, or may be determined in advance for each of the processing steps S322-1 to S322-4.

[0172] Although the first to fourth image enhancement processes have been described above as an example of image enhancement processing, the image enhancement processing of step S322 is not limited to the first to fourth image enhancement processes. The image enhancement processing of step S322 may include at least the second image enhancement process. Furthermore, it is preferable to further include the first image enhancement process, and the third and fourth image enhancement processes may be omitted.

[0173] Next, after the image enhancement process, in step S324 shown in FIG. 21 , the image processing unit 206 performs a summation process to combine the images resulting from the process of step S322. Here, a process of summing multiple images after the image enhancement process described above is performed. This summation process results in an image in which at least a vascular image showing the region (site) of retinal blood vessels such as neovascularization is enhanced. When performing the summation process, images are added and subtracted using a code assigned to the image for each of the image processes described above or a code determined for the process performed from step S322-1 to step S322-2.

[0174] Next, in step S326, the image processing unit 206 performs image generation processing on an image including a blood vessel image (e.g., image 326G in FIG. 22 ). The image generation processing includes performing image processing to further enhance the image by adjusting the signal intensity of an image including at least retinal blood vessels such as neovascularization. That is, the image generation processing performs image processing to enhance at least a blood vessel image 326Gx showing an area (site) of retinal blood vessels such as neovascularization so as to improve visibility compared to images of other areas other than the blood vessels. Thus, it is possible to obtain an image including a blood vessel image 326Gx (shown by a white line in FIG. 22 ) in which at least the blood vessel image such as neovascularization is enhanced. The processed image after the image generation processing is stored in the RAM 266 by the processing unit 208. Image 326G in FIG. 22 shows an example of an image including a blood vessel image 326Gx after image processing.

[0175] Next, the image generation process will be further described. In relation to image 326G shown in FIG. 22, signal intensity characteristics 326Q are shown, which are schematic diagrams of the relationship between the position and signal intensity in a partial region 326Gp containing blood vessels on the output image after the processing of step S324. Here, the signal intensity is determined based on input values ​​of image features such as luminance in the output image after the processing of step S324, and includes values ​​obtained by differentiating the input values, which are the original signals.

[0176] For example, in the image generation process, as a first image processing step, the signal intensity of pixels corresponding to positions below a predetermined signal intensity Ith in the signal intensity characteristic 326Q, which indicates a feature related to brightness, such as luminance, of an image including a blood vessel image, is deleted or set to a predetermined value. Here, the pixel value of the pixel may be deleted or set to a predetermined value. In FIG. 22 , the signal intensity characteristic at positions exceeding the signal intensity Ith is shown by a solid line as characteristic 326Q1. Also in the figure, the deleted signal intensity characteristic at positions below the signal intensity Ith is shown by a dotted line. Next, as a second image processing step, the contrast of the image based on the signal intensity characteristic 326Q1 after the first image processing is set to a predetermined multiple (e.g., double). This contrast setting corresponds to expanding the range of signal intensity variation based on characteristic 326Q1, i.e., expanding the dynamic range. Image processing that deletes or sets the signal intensity of pixels corresponding to positions below the signal intensity Ith to a predetermined value and multiplies the contrast by a predetermined factor is image processing that targets only areas where the signal intensity of the emphasized blood vessels is high, and is effective when emphasizing neovascularization.

[0177] In another example of the image generation process, image processing that takes into account structures on the fundus is also possible. Specifically, as the third image processing, the signal intensity characteristic 326Q described above is adjusted based on the positive or negative sign of the signal. Specifically, the absolute value of the signal intensity is applied, and for example, the signal intensity is set to a value with the sign removed for pixels corresponding to positions where the signal intensity is 0 or less. In FIG. 22, the signal intensity characteristic with a positive sign and the signal intensity characteristic with the negative sign removed are shown by solid lines as characteristic 326Q2. Also in the figure, the signal intensity characteristic with a negative sign is shown by dotted lines. Next, as in the second image processing, as the fourth image processing, the contrast of the image based on the signal intensity characteristic 326Q2 after the third image processing is set to a predetermined multiple (e.g., 2x). The third and fourth image processing are image processing that emphasize areas that represent structures on the fundus and are effective for emphasizing the presence or absence of structures on the fundus.

[0178] In the present embodiment, a case where a blood vessel image is formed in white on a background image in black has been described as an example of a fundus image, but the technology disclosed herein is not limited to this, and other colors may be used. For example, a blood vessel image may be formed in black on a background image in white.

[0179] Next, a display screen for displaying a fundus image including neovascularization, to which the technology of this embodiment is applied, will be described. As in the above embodiment, the display screen is generated by the display control unit 204 of the server 140 based on a user's instruction, and is output as an image signal to the viewer 150 by the processing unit 208. The viewer 150 displays the display screen on the display based on the image signal.

[0180] A display screen 500B is shown in Figure 24. As shown in Figure 24, the display screen 500B has an information area 502 similar to the above embodiment, and an image display area 504B. The image display area 504B includes a comment field 506 similar to the above embodiment.

[0181] The image display area 504B is an area for displaying an image of the subject's eye, etc. The image display area 504B is provided with the following display fields, specifically, a UWF fundus image display field 542 and an image enhancement condition display field 550. Although not shown, the image display area 504B can also display a superimposed OCT volume data conceptual diagram display field and a tomographic image display field. The image display area 504B can also display a superimposed boundary surface between the choroid and sclera obtained by the choroid-sclera boundary detection process ( FIG. 9 ).

[0182] In the UWF fundus image display field 542, an image 326G including a blood vessel image 326Gx after enhancement processing is displayed for a UWF-SLO fundus image obtained by photographing the fundus of the subject's eye with the ophthalmologic apparatus 110.

[0183] The image enhancement condition display field 550 is provided with display fields that display conditions for executing image enhancement processing on a fundus image, which the user can set. Specifically, the image enhancement condition display field 550 includes a condition display field 552 for a first image enhancement processing and a condition display field 554 for a second image enhancement processing. The image enhancement condition display field 550 also includes a condition display field 556 for a third image enhancement processing and a condition display field 558 for a fourth image enhancement processing. The condition display field 552 for the first image enhancement processing allows the user to set predetermined execution conditions for executing the first image enhancement processing. Similarly, the condition display fields 554-558 also allow the user to set execution conditions corresponding to the processing.

[0184] The image display area 504B of the display screen 500A allows the user to view, for example, a fundus image in which neovascularization is emphasized. Furthermore, the image enhancement condition display field 550 allows the user to view a fundus image in which blood vessels and other features are emphasized, reflecting the desired condition values, by setting the desired condition values. This allows the user to obtain more information for diagnosis.

[0185] As described above, in this embodiment, it is possible to form images that facilitate the observation of blood vessels such as neovascularization, and therefore it is possible to faithfully visualize retinal blood vessels such as neovascularization, which are finer than other blood vessels.

[0186] In the above embodiment, image processing (FIG. 5) is performed by the server 140, but the present disclosure is not limited to this and may be performed by the ophthalmic device 110, the viewer 150, or an additional image processing device further provided on the network 130.

[0187] In the present disclosure, each component (device, etc.) may be present in one or more instances, unless a contradiction arises.

[0188] In the above-described examples, image processing is implemented by a software configuration using a computer. However, the present disclosure is not limited to this, and at least a portion of the processing may be implemented by a hardware configuration. Furthermore, while the above description uses a CPU as an example of a general-purpose processor, the term "processor" refers to a processor in a broad sense and includes general-purpose processors (e.g., a CPU (Central Processing Unit), etc.) and dedicated processors (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a programmable logic device, etc.). Therefore, image processing may be performed solely by a hardware configuration, or some of the image processing may be performed by a software configuration and the remaining processing may be performed by a hardware configuration.

[0189] Furthermore, the operations of the above-mentioned processors may not only be performed by a single processor, but may also be performed by multiple processors working together, or may be performed by multiple processors located in physically separate locations working together.

[0190] Furthermore, in order to cause a computer to execute the above-described processing, a program in which the above-described processing is written in computer-processable code may be stored on a storage medium such as an optical disk and distributed.

[0191] As such, the present disclosure includes both cases in which image processing is realized by a software configuration using a computer and cases in which it is not realized, and therefore includes the following techniques.

[0192] The following technology is proposed based on the above disclosure: A computer program product for determining the boundary between choroidal vessels and the sclera, the computer program product comprising a computer-readable storage medium that is not itself a temporary signal, the computer-readable storage medium storing a program that causes a computer to: acquire OCT volume data representing a depth-wise tomographic image including the choroidal vessels and the sclera, perform enhancement processing based on the OCT volume data to enhance a physical quantity representing a brightness change in the depth direction in a partial region of the tomographic image, derive an image feature amount related to the brightness change in the depth direction in the enhanced region, and determine, based on the image feature amount, a region on the tomographic image where the image feature amount exceeds a predetermined threshold as the boundary between the choroidal vessels and the sclera.

[0193] In the above example, an example was given in which image processing is realized by a software configuration using a computer, but the technology of the present disclosure is not limited to this. For example, instead of a software configuration using a computer, image processing may be performed only by a hardware configuration such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Part of the image processing may be performed by a software configuration, and the remaining part may be performed by a hardware configuration.

[0194] Furthermore, the image processing described above is merely an example, and the technical scope of the present disclosure is not limited to the scope described in the above embodiment. Therefore, various modifications or improvements can be made to the above embodiment, such as deleting unnecessary processing, adding new processing, or changing the processing order, without departing from the spirit of the present disclosure, and such modifications or improvements are also included in the technical scope of the present disclosure.

[0195] The disclosure of Japanese Patent Application No. 2024-116272 is incorporated herein by reference in its entirety, and all documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising: an acquisition unit that acquires a depth-wise tomographic image including choroidal blood vessels and the sclera; a derivation unit that performs enhancement processing on a partial region of the tomographic image to enhance a physical quantity that indicates a brightness change in the depth direction in the tomographic image, and derives an image feature amount related to the brightness change in the depth direction in the enhanced region; and a determination unit that determines, based on the image feature amount, a portion on the tomographic image where the image feature amount exceeds a predetermined threshold as a boundary between the choroidal blood vessels and the sclera.

2. The image processing device according to claim 1, wherein the physical quantity indicating the brightness change is a physical quantity indicating the luminance change of the tomographic image.

3. The image processing device according to claim 2, wherein said enhancement processing enhances the degree of luminance change to a greater degree than before enhancement when the degree of luminance change exceeds a predetermined degree.

4. The image processing device according to any one of claims 1 to 3, wherein the derivation unit performs enhancement processing by filtering to enhance the physical quantity on an area of ​​a predetermined size smaller than the size of the tomographic image.

5. The image processing device according to claim 4, wherein the tomographic image is a two-dimensional image, and the filtering process performs the enhancement process using a two-dimensional filter that treats the region as a two-dimensional region and gradually changes brightness to enhance it from one side to the other.

6. An image processing device according to claim 4 or claim 5, wherein the tomographic image is a three-dimensional image, and the filtering process performs the enhancement process using a three-dimensional filter that enhances the region from one side to the other as a three-dimensional region and whose brightness gradually changes.

7. The image processing device according to any one of claims 1 to 5, wherein the derivation unit repeatedly performs enhancement processing on a brightness change in the depth direction in the tomographic image for a partial region of the tomographic image within the tomographic image.

8. An image processing device according to any one of claims 1 to 7, wherein when multiple regions on a tomographic image are derived in which the image feature amount exceeds a predetermined threshold, the determination unit sets the derived regions as multiple boundary candidates and determines one of the multiple boundary candidates as the boundary on the tomographic image.

9. An image processing device according to any one of claims 1 to 5, wherein when multiple regions on a tomographic image are derived in which the image feature amount exceeds a predetermined threshold, the determination unit sets the derived regions as multiple boundary candidates and determines a region connecting the multiple boundary candidates as the boundary on the tomographic image.

10. An image processing device according to any one of claims 1 to 9, wherein the determination unit determines, as the boundary, a position at a predetermined depth from a part on the tomographic image where the image feature amount exceeds a predetermined threshold.

11. The image processing device according to any one of claims 1 to 10, wherein the determination unit determines the area where the change in the image feature amount is greatest as the boundary.

12. The image processing device according to any one of claims 2 to 11, wherein the determination unit determines the position on the image where the brightness is greatest as the boundary.

13. An image processing method in which a processor performs processing including: acquiring a depth-wise tomographic image including choroidal blood vessels and the sclera; performing enhancement processing on a partial region of the tomographic image to enhance a physical quantity indicating a change in brightness in the depth direction in the tomographic image; deriving image features related to the change in brightness in the depth direction in the enhanced region; and determining, based on the image features, a portion on the tomographic image where the image features exceed a predetermined threshold as the boundary between the choroidal blood vessels and the sclera.

14. A program that causes a processor to perform image processing including: acquiring a depth-wise tomographic image including choroidal blood vessels and the sclera; performing enhancement processing on a partial region of the tomographic image to enhance a physical quantity indicating a change in brightness in the depth direction in the tomographic image; deriving image features related to the change in brightness in the depth direction in the enhanced region; and determining, based on the image features, a portion on the tomographic image where the image features exceed a predetermined threshold as the boundary between the choroidal blood vessels and the sclera.

Images