Image processing method, image processing apparatus, and image processing program

Abstract

To provide an image processing method for analyzing choroidal blood vessels.SOLUTION: The method includes a step of acquiring an OCT image of a subject's eye by optical coherence tomography (OCT), a step of extracting a region other than a structure of the subject's eye in the OCT image, and a step of generating a first display screen in which position information on a scan position for acquiring the OCT image is superimposed on the region.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing method, an image processing device, and an image processing program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2019-154993 discloses a technique for displaying a tomographic image of a subject's eye (an image captured by optical coherence tomography (OCT), referred to as an OCT image). It is required to show the user the position on the subject's eye where the OCT image was acquired. Summary of the Invention

[0003] An image processing method according to a first aspect of the disclosed technology includes the steps of acquiring an OCT image of a test eye, extracting an area in the OCT image other than structures of the test eye, and generating a first display screen in which positional information regarding the scanning position for acquiring the OCT image is superimposed on the area.

[0004] An image processing device according to a second aspect of the disclosed technology includes an image acquisition unit that acquires an OCT image of the subject's eye, a fundus image processing unit that extracts an area other than the structures of the subject's eye from the OCT image, and an image synthesis unit that generates a first display screen in which positional information regarding the scanning position for acquiring the OCT image is superimposed on the area.

[0005] The image processing program of the third aspect of the technology disclosed herein causes a computer to function as a fundus image processing unit that extracts areas other than structures of the test eye in an OCT image of the test eye acquired by an image acquisition unit, and an image synthesis unit that generates a first display screen in which positional information regarding the scanning position for acquiring the OCT image is superimposed on the areas. [Brief explanation of the drawings]

[0006] [Figure 1]FIG. 1 is a block diagram of an ophthalmology system 100. [Figure 2] 1 is a schematic diagram showing the overall configuration of an ophthalmologic apparatus 110. FIG. [Figure 3] FIG. 2 is a block diagram of the electrical configuration of the management server 140. [Figure 4] FIG. 2 is a functional block diagram of a CPU 262 of the management server 140. [Figure 5] FIG. 2 is a functional block diagram of an image processing control unit 206 of a CPU 262 of the management server 140. [Figure 6A] FIG. 6 is a diagram showing the relationship between a UWF-SLO image 600 and character information. [Figure 6B] FIG. 7 is a diagram showing the relationship between an OCT image 700 and an area without fundus structures. [Figure 7A] 10 is a diagram showing a first display screen in which position information 300 in the form of text information is superimposed on an OCT image 700. FIG. [Figure 7B] FIG. 10 is a diagram showing another example of the first display screen in which a silhouette image 310 combined with a mark 302 indicating the OCT imaging position is superimposed on an OCT image 700. [Figure 8] FIG. 7 is a diagram showing a second display screen 700C on which an OCT image 700 and a silhouette image 310 are displayed so as not to overlap each other. [Figure 9A] FIG. 3 is a diagram showing position information 302 indicating the photographing position of an OCT image superimposed on a UWF-SLO image. [Figure 9B] 7 is a schematic diagram showing a process in which an OCT image 700 is displayed instead of a UWF-SLO image 650. FIG. [Figure 9C] FIG. 7 is a schematic diagram showing an OCT image 700 displayed in place of the UWF-SLO image 650. [Figure 10] 4 is a flowchart of image processing in the present embodiment. [Figure 11A] This is a schematic diagram of an area 320 in place. [Figure 11B] 3 is a histogram showing the luminance values ​​of pixels in region 320. [Figure 12A]10 is a schematic diagram showing a case where a structure of the fundus and a part of the region 320 overlap with each other. [Figure 12B] 3 is a histogram showing the luminance values ​​of pixels in region 320. [Figure 13A] FIG. 3 is a schematic diagram showing an area 320 in an inappropriate position. [Figure 13B] 3 is a histogram showing the luminance values ​​of pixels in region 320. [Figure 14] 10 is a flowchart for generating a display screen showing the positional relationship between an enface OCT image and a high-density scan OCT image in this embodiment. [Figure 15] 8 shows a display screen 800A in which a three-dimensional spherical stereoscopically depicted icon 410 generated from an enface OCT image is superimposed on a high density scan OCT image 800. [Figure 16] Display screen 800B shows a three-dimensional spherical icon 410 generated from an enface OCT image superimposed on a high-density scan OCT image 810 obtained when a volume scan is performed using a high-density scan. [Figure 17] FIG. 7 is a schematic diagram illustrating a case where a part of an OCT image 700 and a part of position information 310 are overlapped. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, the present embodiment will be described in detail with reference to the drawings.

[0008] The configuration of an ophthalmologic system 100 will be described with reference to Fig. 1. As shown in Fig. 1, the ophthalmologic system 100 includes an ophthalmologic apparatus 110, a management server apparatus (hereinafter referred to as "management server") 140, and a display device (hereinafter referred to as "viewer") 150. The ophthalmologic apparatus 110 acquires fundus images. The management server 140 stores, in association with the patient IDs, multiple fundus images and axial lengths obtained by photographing the funduses of multiple patients using the ophthalmologic apparatus 110. The viewer 150 displays the fundus images and analysis results acquired by the management server 140.

[0009] The viewer 150 includes a display 156 that displays fundus images and analysis results acquired by the management server 140, a mouse 155M, and a keyboard 155K that are used for operation.

[0010] The ophthalmologic apparatus 110, the management server 140, and the viewer 150 are connected to each other via a network 130. The viewer 150 is a client in a client-server system, and is connected via the network. A plurality of viewers 150 may be connected to the network 130. Furthermore, in order to ensure system redundancy, a plurality of servers 140 may be connected via the network 130. If the ophthalmic apparatus 110 has an image processing function and an image viewing function of the viewer 150, it becomes possible for the ophthalmic apparatus 110 to acquire, process, and view fundus images in a standalone state. Also, if the management server 140 has an image viewing function of the viewer 150, it becomes possible for the configuration of the ophthalmic apparatus 110 and the management server 140 to acquire, process, and view fundus images.

[0011] Other ophthalmic devices (examination devices for visual field measurement, intraocular pressure measurement, etc.) and a diagnosis support device that performs image analysis using AI (Artificial Intelligence) may be connected to the ophthalmic device 110, the management server 140, and the viewer 150 via the network 130. In the present disclosure, each component (device, etc.) may exist alone or in two or more instances, as long as no contradiction occurs.

[0012] Next, the configuration of the ophthalmologic apparatus 110 will be described with reference to Fig. 2. For convenience of description, a scanning laser ophthalmoscope will be referred to as "SLO", and optical coherence tomography will be referred to as "OCT".

[0013] When the ophthalmic apparatus 110 is placed on a horizontal plane, the horizontal direction is defined as the "X direction," the vertical direction relative to the horizontal plane is defined as the "Y direction," and the direction connecting the center of the pupil of the anterior segment of the subject's eye 12 and the center of the eyeball is defined as the "Z direction." Therefore, the X direction, Y direction, and Z direction are perpendicular to each other.

[0014] The ophthalmologic apparatus 110 includes an imaging device 14 and a control device 16. The imaging device 14 is equipped with an SLO unit 18 and an OCT unit 20, and acquires a fundus image of the fundus of the subject's eye 12. Hereinafter, a two-dimensional fundus image acquired by the SLO unit 18 will be referred to as an SLO image. Also, a tomographic image or a front image (en-face image) of the retina created based on OCT data acquired by the OCT unit 20 will be referred to as an OCT image.

[0015] The control device 16 comprises a computer having a CPU (Central Processing Unit) 16A, a RAM (Random Access Memory) 16B, a ROM (Read-Only Memory) 16C, and an input / output (I / O) port 16D.

[0016] The control device 16 includes an input / display device 16E connected to the CPU 16A via an I / O port 16D. The input / display device 16E has a graphic user interface that displays an image of the subject's eye 12 and receives various instructions from the user. An example of the graphic user interface is a touch panel display.

[0017] The control device 16 also includes an image processing device 17 connected to an I / O port 16D. The image processing device 17 generates an image of the subject's eye 12 based on data obtained by the photographing device 14. The control device 16 is connected to a network 130 via a communication interface (I / F) 16F.

[0018] 2, the control device 16 of the ophthalmic apparatus 110 includes the input / display device 16E, but the technology of the present disclosure is not limited to this. For example, the control device 16 of the ophthalmic apparatus 110 may not include the input / display device 16E, but may include a separate input / display device that is physically independent from the ophthalmic apparatus 110. In this case, the display device includes an image processing processor unit that operates under the control of the CPU 16A of the control device 16. The image processing processor unit may display an SLO image or the like based on an image signal instructed to be output by the CPU 16A.

[0019] The imaging device 14 operates under the control of the CPU 16A of the control device 16. The imaging device 14 includes an SLO unit 18, an imaging optical system 19, and an OCT unit 20. The imaging optical system 19 includes a first optical scanner 22, a second optical scanner 24, and a wide-angle optical system 30.

[0020] The first optical scanner 22 performs two-dimensional scanning in the X and Y directions with the light emitted from the SLO unit 18. The second optical scanner 24 performs two-dimensional scanning in the X and Y directions with the light emitted from the OCT unit 20. The first optical scanner 22 and the second optical scanner 24 may be optical elements that can deflect a light beam, such as a polygon mirror or a galvanometer mirror. Alternatively, a combination of these may be used.

[0021] The wide-angle optical system 30 includes an objective optical system (not shown in FIG. 2) having a common optical system 28 , and a combining unit 26 that combines the light from the SLO unit 18 and the light from the OCT unit 20 .

[0022] The objective optical system of the common optical system 28 may be a reflective optical system using a concave mirror such as an elliptical mirror, a refractive optical system using a wide-angle lens, or a catadioptric system combining concave mirrors and lenses. By using a wide-angle optical system using an elliptical mirror or a wide-angle lens, it becomes possible to photograph the retina in the peripheral part of the fundus as well as the center of the fundus.

[0023] When a system including an elliptical mirror is used, the system using the elliptical mirror described in International Publication WO2016 / 103484 or International Publication WO2016 / 103489 may be used. The disclosures of International Publication WO2016 / 103484 and International Publication WO2016 / 103489 are each incorporated herein by reference in their entirety.

[0024] The wide-angle optical system 30 enables observation of the fundus over a wide field of view (FOV) 12A. The FOV 12A indicates the range that can be photographed by the imaging device 14. The FOV 12A can be expressed as a field of view. In this embodiment, the field of view can be defined by an internal illumination angle and an external illumination angle. The external illumination angle is the illumination angle of the light beam irradiated from the ophthalmic device 110 to the subject's eye 12, determined with the pupil 27 as the reference. The internal illumination angle is the illumination angle of the light beam irradiated to the fundus F, determined with the eyeball center O as the reference. The external illumination angle and the internal illumination angle correspond to each other. For example, if the external illumination angle is 120 degrees, the internal illumination angle corresponds to approximately 160 degrees. In this embodiment, the internal illumination angle is 200 degrees.

[0025] Here, an SLO fundus image captured at an internal illumination angle of 160 degrees or more is referred to as a UWF-SLO fundus image. UWF stands for Ultra Wide Field.

[0026] The SLO system is realized by a control device 16, an SLO unit 18, and an imaging optical system 19 shown in Fig. 2. The SLO system includes a wide-angle optical system 30, and therefore enables fundus imaging with a wide FOV 12A.

[0027] The SLO unit 18 includes a B (blue light) light source 40, a G (green light) light source 42, an R (red light) light source 44, and an IR (infrared (e.g., near-infrared) light) light source 46, as well as optical systems 48, 50, 52, 54, and 56 that reflect or transmit the light from the light sources 40, 42, 44, and 46 and guide them into a single optical path. The optical systems 48, 50, and 56 are mirrors, and the optical systems 52 and 54 are beam splitters. The B light is reflected by the optical system 48, passes through the optical system 50, and is reflected by the optical system 54; the G light is reflected by the optical systems 50 and 54; the R light is transmitted through the optical systems 52 and 54; and the IR light is reflected by the optical systems 52 and 56 and is each guided into a single optical path.

[0028] The SLO unit 18 is configured to be switchable between a light source that emits laser light of different wavelengths, or a combination of light sources that emit light, such as a mode that emits G light, R light, and B light, and a mode that emits infrared light. In the example shown in FIG. 2, four light sources are provided: a B light (blue light) light source 40, a G light source 42, an R light source 44, and an IR light source 46; however, the technology of the present disclosure is not limited to this. For example, the SLO unit 18 may further include a white light source and emit light in various modes, such as a mode that emits only white light.

[0029] Light incident on the photographing optical system 19 from the SLO unit 18 is scanned in the X and Y directions by the first optical scanner 22. The scanning light passes through the wide-angle optical system 30 and the pupil 27 and is irradiated onto the posterior segment (fundus) of the subject's eye 12. The light reflected by the fundus passes through the wide-angle optical system 30 and the first optical scanner 22 and is incident on the SLO unit 18.

[0030] The SLO unit 18 includes a beam splitter 64 that reflects B light and transmits all light except B light from the posterior segment (fundus) of the subject's eye 12, and a beam splitter 58 that reflects G light and transmits all light except G light from the light that has passed through the beam splitter 64. The SLO unit 18 includes a beam splitter 60 that reflects R light and transmits all light except R light from the light that has passed through the beam splitter 58. The SLO unit 18 includes a beam splitter 62 that reflects IR light from the light that has passed through the beam splitter 60. The SLO unit 18 includes a B light detecting element 70 that detects B light reflected by the beam splitter 64, a G light detecting element 72 that detects G light reflected by the beam splitter 58, an R light detecting element 74 that detects R light reflected by the beam splitter 60, and an IR light detecting element 76 that detects IR light reflected by the beam splitter 62.

[0031] Light (reflected light reflected by the fundus) incident on the SLO unit 18 via the wide-angle optical system 30 and the scanner 22 is reflected by the beam splitter 64 and received by the B light detection element 70 in the case of B light, and transmitted through the beam splitter 64, reflected by the beam splitter 58, and received by the G light detection element 72 in the case of G light. The incident light is transmitted through the beam splitters 64 and 58 in the case of R light, reflected by the beam splitter 60, and received by the R light detection element 74. The incident light is transmitted through the beam splitters 64, 58, and 60 in the case of IR light, reflected by the beam splitter 62, and received by the IR light detection element 76. The image processing device 17, which operates under the control of the CPU 16A, generates a UWF-SLO image using signals detected by the B light detection element 70, the G light detection element 72, the R light detection element 74, and the IR light detection element 76. The B photodetector element 70, the G photodetector element 72, the R photodetector element 74, and the IR photodetector element 76 may be, for example, a PD (photodiode) or an APD (avalanche photodiode). In the SLO unit 18, light reflected (scattered) and returned from the fundus, which is the object, passes through the first optical scanner 22 and reaches the photodetector elements, so it always returns to the same position, i.e., the position where the B photodetector element 70, the G photodetector element 72, the R photodetector element 74, and the IR photodetector element 76 are located. Therefore, the photodetector elements do not need to be configured as a planar (two-dimensional) element like an area sensor, and a point-like (zero-dimensional) detector such as a PD or APD is optimal in this embodiment. However, it is not limited to a PD or APD, and a line sensor (one-dimensional) or area sensor (two-dimensional) can also be used.

[0032] UWF-SLO images include those obtained by photographing the fundus with G-color light (G-color fundus images) and those obtained by photographing the fundus with R-color light (R-color fundus images). UWF-SLO images include those obtained by photographing the fundus with B-color light (B-color fundus images) and those obtained by photographing the fundus with IR light (IR fundus images).

[0033] The control device 16 also controls the light sources 40, 42, and 44 to emit light simultaneously. By simultaneously photographing the fundus of the subject's eye 12 with B light, G light, and R light, a G-color fundus image, an R-color fundus image, and a B-color fundus image, each of which corresponds to each other, are obtained. An RGB color fundus image is obtained from the G-color fundus image, the R-color fundus image, and the B-color fundus image. The control device 16 controls the light sources 42 and 44 to emit light simultaneously, and by simultaneously photographing the fundus of the subject's eye 12 with G light and R light, a G-color fundus image and an R-color fundus image, each of which corresponds to each other, are obtained. By mixing the G-color fundus image and the R-color fundus image at a predetermined mixing ratio, an RG color fundus image is obtained.

[0034] UWF-SLO images also include UWF-SLO images (videos) taken with ICG fluorescence. When indocyanine green (ICG) is injected into a blood vessel, it reaches the fundus, first the retina, then the choroid, and passes through the choroid. UWF-SLO images (videos) are moving images from the time indocyanine green (ICG) is injected into a blood vessel and reaches the retina to after it has passed through the choroid.

[0035] The image data of the B color fundus image, the G color fundus image, the R color fundus image, the IR fundus image, the RGB color fundus image, the RG color fundus image, and the UWF-SLO image are sent from the ophthalmologic device 110 to the management server 140 via the communication I / F 16F.

[0036] The OCT system is realized by a control device 16, an OCT unit 20, and an imaging optical system 19 shown in Figure 2. The OCT system is equipped with a wide-angle optical system 30, which enables fundus imaging with a wide FOV 12A in addition to capturing the SLO fundus image described above. The user specifies a position on the UWF-SLO image from which to acquire an OCT image, and the OCT image is acquired by scanning (capturing) the specified position. The OCT unit 20 includes a light source 20A, a sensor (detecting element) 20B, a first optical coupler 20C, a reference optical system 20D, a collimating lens 20E, and a second optical coupler 20F.

[0037] Light emitted from the light source 20A is branched by the first optical coupler 20C. One of the branched beams is collimated by the collimating lens 20E as measurement light and then enters the imaging optical system 19. The measurement light is scanned in the X and Y directions by the second optical scanner 24. The scanning light passes through the wide-angle optical system 30 and the pupil 27 and is irradiated onto the fundus. The measurement light reflected by the fundus passes through the wide-angle optical system 30 and the second optical scanner 24 and enters the OCT unit 20, and then passes through the collimating lens 20E and the first optical coupler 20C and enters the second optical coupler 20F.

[0038] The other light beam emitted from the light source 20A and branched by the first optical coupler 20C is incident as reference light on the reference optical system 20D, passes through the reference optical system 20D, and then enters the second optical coupler 20F.

[0039] The light beams incident on the second optical coupler 20F, i.e., the measurement light beam reflected from the fundus and the reference light beam, interfere with each other to generate interference light. The interference light beam is received by the sensor 20B. The image processing device 17, which operates under the control of the image processing controller 206, generates OCT images such as tomographic images and en-face images based on the OCT data detected by the sensor 20B.

[0040] Image data of the OCT image and position information regarding the OCT acquisition position (information indicating the area and location scanned during OCT imaging, such as pixel positions and coordinate data on the UWF-SLO image, or scanner drive signals) are sent from the ophthalmic device 110 to the management server 140 via the communication I / F 16F and stored in the memory device 254.

[0041] In this embodiment, the light source 20A is exemplified as a wavelength-swept type SS-OCT (Swept-Source OCT), but various types of OCT systems may also be used, such as SD-OCT (Spectral-Domain OCT) and TD-OCT (Time-Domain OCT).

[0042] Next, the configuration of the electrical system of the management server 140 will be described with reference to FIG. 3. As shown in FIG. 3, the management server 140 includes a computer main unit 252. The computer main unit 252 has a CPU 262, a RAM 266, a ROM 264, and an input / output (I / O) port 268. The I / O port 268 is connected to a storage device 254, a display 256, a mouse 255M, a keyboard 255K, and a communication interface (I / F) 258. The storage device 254 is configured with a non-volatile memory such as an HDD (hard disk drive) or an SSD (solid state drive). The input / output (I / O) port 268 is connected to the network 130 via the communication I / F 258. Therefore, the management server 140 can communicate with the ophthalmologic apparatus 110 and the viewer 150. The storage device 254 stores an image processing program, which will be described later. The image processing program may also be stored in the ROM 264. The CPU 262 corresponds to the "processor" of the technology disclosed herein. The ROM 264 and the RAM 266 correspond to the "memory" of the technology of the present disclosure.

[0043] The management server 140 stores the data received from the ophthalmologic apparatus 110 in the storage device 254 and executes various image processing and data processing using the CPU 262 .

[0044] Next, various functions realized by the CPU 262 of the management server 140 executing the image processing program will be described with reference to Fig. 4. The image processing program has a display control function, an image processing control function, and a processing function. When the CPU 262 executes the image processing program having these functions, the CPU 262 functions as a display control unit 204, an image processing control unit 206, and a processing unit 208, as shown in Fig. 4.

[0045] Next, various functions of the image processing control unit 206 will be described with reference to Fig. 5. The image processing control unit 206 functions as a fundus image processing unit 2060 that performs image processing such as generating an image in which choroidal blood vessels and the like are sharpened from a fundus image, performing image processing to identify areas of fundus structures from a fundus image, and performing image processing to identify areas in which the structures are not captured, and an image synthesis unit 2062 that synthesizes multiple images as necessary. The fundus image processing unit 2060 corresponds to the "image acquisition unit" and "fundus image processing unit" of the technology disclosed herein. The image synthesis unit 2062 corresponds to the "image synthesis unit" of the technology disclosed herein.

[0046] First, the position information regarding the photographing position of the OCT image (the scanning position of the fundus for acquiring the OCT image) will be described. 6A is a diagram in which a UWF-SLO image 600 of the fundus of the subject's eye 12 is divided into a plurality of areas, and character strings with a regularity are assigned to the areas. This is an example in which the symbols are used as position information indicating the position where the OCT image was taken. In Figure 6A, coordinate values ​​A, B, and C are set vertically on the page, and 1, 2, and 3 are set horizontally on the page, and the UWF-SLO image 600 is divided into nine 3x3 areas. The position information is a character string (A1, A2, A3, B1, B2, B3, C1, C2, C3) that combines the first coordinate value, which is a letter in the vertical direction, and the second coordinate value, which is a number in the horizontal direction. The character string may not only consist of letters and numbers, but may also use symbols such as hyphens and spaces in addition to letters and numbers, such as "A-2" (three letters: A, a hyphen, and a 2), to improve user visibility. The nine areas of the UWF-SLO image can be identified by identifying them with regular character strings. By knowing the rules for these character strings in advance, the user can roughly grasp the location of the UWF-SLO image 600 just by looking at the location information, which is a character string. Position information is generated that associates the position where the OCT image was taken with the symbol, and an OCT image is generated with the position information superimposed.Even if the OCT image is displayed on the front of a display, the user can understand the position on the fundus of the OCT image displayed on the front of the display by looking at the superimposed position information. The position information may be a string of characters that combines letters representing vertical positions and numbers representing horizontal positions as coordinate values ​​on the UWF-SLO image, or may be a string of characters that indicates the subject's ear (E) side, nose (N) side, upper (U) side, or lower (D) side, and by adding characters to the string and increasing the number of characters, the position information may be used to subdivide the area. For example, when subdividing "A-2" among the nine areas, a character string "A-2 N" is generated by adding "N" representing the nasal side to "A-2." This character string "A-2 N" can represent the position of the nasal side within the A-2 area in Figure 6A.

[0047] The position information on the UWF-SLO image 600 may be expressed by a character string as described above, or may be expressed by a position based on structural features on the retina. For example, it may be a coordinate value using an orthogonal coordinate system in which x is set horizontally and y is set vertically, with the macula or ONH (optic nerve head) at the center.

[0048] Next, the position on the OCT image where the positional information, which is a character string, is superimposed will be explained using Fig. 6B. Fig. 6B is a diagram showing an OCT image 700 taken at a photographing position specified by the user. The OCT image 700 is an image generated solely from the OCT data detected by sensor 20B. On the OCT image 700, there are a region 720 of the retina and choroid, which are fundus structures, a region 740 of the vitreous body, and a region 760 outside the choroid (the region surrounded by a dotted line in FIG. 6B). Generally, pixels with low brightness values ​​are concentrated in regions 740 and 760. Region 740 of the vitreous body is photographed darkly because there is no structure that reflects the measurement light and therefore no reflected light is obtained, while region 760 is photographed darkly because the measurement light is absorbed by the tissue of the fundus and does not return as reflected light. In Figures 6B, 7A, 7B, 8, 9C, 11A, 12A, and 13A, areas made up of pixels with low brightness values ​​are displayed in white to improve visibility, but these areas appear dark in the actual image. FIG. 6B shows an example in which the OCT image 700 is displayed over the entire area of ​​the display 156 of the viewer 150.

[0049] Hereinafter, examples of display forms for displaying position information of OCT imaging positions according to this embodiment will be described with reference to FIGS. 7A to 9C. 7A is a diagram showing a display screen 700A in which position information 300 relating to the position where OCT data was acquired is displayed as text information superimposed on an OCT image 700 in an area 740 in which fundus structures are not displayed. The text of position information 300 is "Photographing position: A-2 N," written using coordinate values ​​indicating the nine areas in which UWF-SLO image 600 is divided, as described in FIG. 6A. Position information 300 may also be displayed in an area 760 instead of area 740. If the OCT image 700 is displayed over the entire area of ​​the display 156 of the viewer 150, the user cannot grasp the position on the fundus of the OCT image 700. By displaying a character string indicating the position in an area where no major structures exist, the user can recognize the character string and grasp the position where the OCT data was acquired. FIG. 7B shows a display screen 700B in which a silhouette image 310 of the UWF-SLO image is superimposed on the OCT image 700 instead of the character string in FIG. 7A as positional information relating to the position where the OCT data was acquired. A silhouette image is a monochrome image with the outline filled in. In this embodiment, it is an image of the outline of a UWF-SLO image, with the outline filled in with a monochrome color such as gray. The contour may be a contour extracted from the actual UWF-SLO image 600, a contour simulating the contour of the UWF-SLO image, or a contour of a shape obtained by deforming the UWF-SLO image. The silhouette image 310 can also be considered a guide image (navigation image) that shows the user the scan position where the OCT image was taken. Position information 302, indicated by an arrow that roughly indicates the position where the OCT data was acquired, is synthesized into the silhouette image 310. The arrow-shaped position information 302 is synthesized at a position on the silhouette image 310 that corresponds to the imaging position on the fundus where the B-scan was performed. 7B, position information indicating the photographing position is displayed as a silhouette image 310 in an area 740 (see FIG. 6B) where no fundus structures exist on the OCT image 700. The silhouette image 310 may be displayed in an area 760 instead of the area 740. Additionally, the position information 302 indicating the imaging position may be displayed in a bright color to make it stand out among the solid colors of the silhouette image. Furthermore, the position information 302 may be displayed as an arrow in the case of a B-scan. In the case of a C-scan or volume scan, it may be displayed as a shape (circle, rectangle, etc.) corresponding to the scanned area. The silhouette image 310 may be a symbolic figure such as an icon. Furthermore, fundus structures such as ONH, macula, or choroidal blood vessels may be superimposed on the silhouette image 310 of the UWF-SLO image using a technique such as CG. Furthermore, instead of the silhouette image 310 of the icon-shaped UWF-SLO image 600, a reduced version of the actual UWF-SLO image 600 may be displayed. Furthermore, if another OCT image is captured at a different position from the OCT image in question, the position information 302 indicating the capture position of the OCT image 700 displayed on the display is displayed in a conspicuous color such as red. On the other hand, position information indicating the capture positions of other OCT images (OCT images that have been captured but are not displayed on the display) may be displayed in a relatively subdued color such as blue superimposed on the silhouette image. The display form of the capture positions may also be changed, such as by using solid and dotted lines instead of colors.

[0050] 8 is a schematic diagram showing a case where an OCT image 700C and a silhouette image 310 are displayed without overlapping on the same screen of the same display 156. In this embodiment, the object to be observed in detail is the OCT image 700C, so the OCT image 700C is displayed as large as possible, and the silhouette image 310, onto which the position information 302 marking the imaging position is combined, may be displayed small, for example, at about the size of a thumbnail.

[0051] 9A, 9B, and 9C are explanatory diagrams showing one aspect of changes over time on the screen of the display 156, etc., when an imaging position 302 is specified on the UWF-SLO image 650 and an OCT image 700 is acquired. The imaging position of the OCT image is specified on the UWF-SLO image 650. In FIG. 9A, position information 302, which is a mark indicating the imaging position, is superimposed on the UWF-SLO image 650. Then, the ophthalmic apparatus 110 controls the OCT unit 20 and the imaging optical system 19 for the specified imaging position to acquire the OCT image 700.

[0052] 9B shows the process in which an OCT image 700 acquired at a specified imaging position 302 is displayed in place of the UWF-SLO image 650 with the imaging position 302 superimposed thereon. In FIG. 9B, over the course of a few seconds immediately after the OCT image is acquired, the UWF-SLO image 650 gradually fades out, while the OCT image 700 gradually fades in. That is, the photographing position 302 changes the transmittance of the superimposed UWF-SLO image 650 from 100% to 0%, and simultaneously changes the transmittance of the OCT image 700 from 0% to 100%. Figure 9B shows an example of this process, showing a display screen on which a UWF-SLO image 6550 with a transmittance of 50% and an OCT image 700 with a transmittance of 50% are superimposed.

[0053] 9C shows an OCT image 700 displayed in place of the UWF-SLO image 650 with the imaging position 302 superimposed thereon. As shown in FIGS. 9A, 9B, and 9C, when the imaging position 302 is designated and the OCT image 700 is acquired, the screen of the display 156 or the like gradually switches from the UWF-SLO image 650 displaying the imaging position 302 to the OCT image, thereby allowing the user to confirm the OCT image 700 after impressing the imaging position 302 on the user. Conversely, the UWF-SLO image 650 with the imaging position 302 superimposed thereon may be displayed instead of the OCT image 700.

[0054] 10 is a flowchart of image processing in this embodiment. The CPU 262 of the management server 140 executes this image processing program, thereby realizing the functions shown in FIG. In step 100 , the processing unit 208 acquires the subject ID entered by the user in the viewer 150 . In step 102, the fundus image processing unit 2060 acquires the subject information (such as name and age) of the subject ID stored in the storage device 254, the fundus image datasets (UWF-SLO image, OCT image), and information about the location where the OCT image was acquired. If there are multiple image datasets, all image datasets may be acquired, or a screen prompting the viewer 150 to select may be displayed to acquire a specific image dataset, or the image dataset with the most recent date and time may be acquired. Furthermore, the image dataset may include not only the UWF-SLO image, OCT image, and information about the location where the OCT image was acquired, but also the name of the disease, diagnostic history, visual field test, and other test data.

[0055] In step 104, the fundus image processor 2060 generates position information 300, 302, 310 based on information about the positions where the OCT images were acquired. The position information 300 is text information indicating an area on the UWF-SLO image described in Fig. 7A. From the information regarding the position where the OCT image was acquired, the fundus image processing unit 2060 identifies the area where the OCT image was acquired and identifies text information consisting of a character string corresponding to the imaging position. Position information 302 is imaging position information obtained based on the OCT imaging position, and position information 310 is the silhouette image described above. In order to superimpose and display the position information 302 and the position information 310, a silhouette image 310 onto which the OCT imaging position described in Fig. 7B is superimposed, or a UWF-SLO image (Figs. 9A and 9B) onto which the OCT position information 302 is superimposed is generated.

[0056] In step 106, the fundus image processor 2060 determines the position at which to superimpose the position information 300, 302, or 310 on the OCT image 700. Specifically, the fundus image processor 2060 identifies an area in the OCT image 700 that is free of fundus structures and has an area large enough to display the position information 300, 302, or 310. In other words, it searches for (extracts) area 740 or area 760 shown in FIG. 6B. If multiple areas that satisfy the conditions are found, the largest area may be determined.

[0057] Specifically, the display position is determined so that the position information 300, 302, or 310 does not overlap with the region to be observed by the user in the OCT image 700 (such as the retina or choroid 720 in FIG. 6B, or the vitreous region where membranes are floating, etc.) For example, if the OCT image 700 is an OCT image obtained by capturing an image of the posterior segment of the subject's eye, the position information must be displayed so as not to overlap with the retina. Whether the position information 300, 302, or 310 is placed on the vitreous side, where no fundus structures are imaged, or on the choroid side depends on the physician's area of ​​interest. When it is desired to view the detailed structure of the dark area on the vitreous side, such as vitreous traction, the position information 300, 302, or 310 may be superimposed on the choroid side, which is the lower layer of the retina. Furthermore, in cases of pachychoroid diseases (such as CSC and PCV) characterized by the choroid, and age-related macular degeneration, the position information 300, 302, or 310 is positioned on the vitreous side so as not to overlap with the area showing the fundus structures. The display position of the position information 300, 302, or 310 may be determined by user input. Alternatively, the position information 300, 302, or 310 may be displayed so as not to overlap with the area showing the fundus structure, and the display position of the position information 300, 302, or 310 may be moved in response to a user operation.

[0058] 11A to 13B, a method for searching (extracting) areas in an OCT image where fundus structures are not displayed using image processing will be described. Areas in which fundus structures are not displayed are generally made up of a concentration of pixels with low brightness values. Therefore, the fundus image processing unit 2060 determines whether fundus structures are captured based on whether a concentration of pixels with low brightness values ​​is present. Specifically, the fundus image processing unit 2060 generates a histogram counting the number of pixels for each brightness value. Then, the fundus image processing unit 2060 calculates the ratio of the number of pixels above a certain brightness value or the ratio of the number of pixels below a certain brightness value from the histogram. Then, from the calculated pixel ratio, the fundus image processing unit 2060 may determine an area in which fundus structures are not displayed. For example, if the ratio of the number of pixels below a certain brightness value is equal to or greater than a predetermined percentage, it is possible to determine that the area does not include an object. Also, if the ratio of the number of pixels above a certain brightness value is equal to or less than a predetermined percentage, it is possible to determine that the area does not include an object.

[0059] In this case, the brightness threshold may be set arbitrarily by the user. For example, the user may set the threshold to 10%, 5%, 3%, or 1% of the maximum brightness value in the OCT image, which is a tomographic image obtained by B-scan. Alternatively, the user may set the threshold to 20%, 15%, 10%, or 5% of the median brightness value in the OCT image.

[0060] FIG. 11A is a schematic diagram of an area 320 where position information 300, 302, or 310 is superimposed and displayed, which is located in a position that does not overlap with the area of ​​the fundus structure in the OCT image 700, and FIG. 11B is a histogram showing the brightness values ​​of the pixels in the area 320. 11B, the ratio of pixels with a brightness value of 50 or less is 1, the ratio of pixels with a brightness value of 100 or less is 1, and the ratio of pixels with a brightness value of 150 or more is 0, indicating that region 320 is a dark region that does not contain fundus structures. In other words, when the histogram has a shape like that of FIG. 11B, region 320 can be said to be an appropriate region for superimposing and displaying position information 300, 302, or 310.

[0061] FIG. 12A is a schematic diagram of an area 320 where position information 300, 302, or 310 is superimposed and displayed, where the area 320 overlaps with part of the area of ​​the fundus structure in the OCT image 700, and FIG. 12B is a histogram showing the brightness values ​​of the pixels in the area 320. 12B, the ratio of pixels with a brightness value of 50 or less is approximately 1, the ratio of pixels with a brightness value of 100 or less is approximately 1, and the ratio of pixels with a brightness value of 150 or more is close to 0, indicating that region 320 is a region that does not include many fundus structures. In other words, when the histogram has a shape like that of Figure 12B, region 320 is not suitable for superimposing and displaying position information 300, 302, or 310, but it can be inferred that the superimposed display will not affect the user's observation of the OCT image.

[0062] FIG. 13A is a schematic diagram of an area 320 where position information 300, 302, or 310 is superimposed and displayed, where the area 320 overlaps with an area of ​​a fundus structure in an OCT image 700, and FIG. 13B is a histogram showing the brightness values ​​of the pixels in area 320. 13B, the ratio of pixels with a brightness value of 50 or less is approximately 0.5, the ratio of pixels with a brightness value of 100 or less is approximately 0.7, and the ratio of pixels with a brightness value of 150 or more is approximately 0.13, indicating that region 320 is a region that includes fundus structures. In other words, when the histogram has a shape like that of Figure 13B, region 320 can be said to be an unsuitable region for superimposing and displaying position information 300, 302, or 310.

[0063] The size of the area where the position information 300, 302, or 310 is displayed may be arbitrarily set by the user depending on the imaging site, imaging magnification, etc. The size of the area can be set to, for example, 5%, 10%, 15%, 20%, or 25% of the OCT image. If it is not possible to identify an area of ​​a predetermined size that does not contain fundus structures in the OCT image, the size of the area may be set smaller and it may be determined whether or not the fundus structures are included in that area.

[0064] Next, in step 108, the image synthesis unit 2062 generates an OCT image in which the position information 300, 302, or 310 is superimposed on the OCT image 700 as a display screen (700A, 700B).

[0065] In step 110, the processing unit 208 outputs the generated display screens (700A, 700B) to the storage device 254 for storage, outputs the image data of the display screens to the viewer 150, and terminates the program. The viewer 150 receives the image data of the display screen and displays the display screen (700A, 700B) on the display 156. Furthermore, if an area of ​​a predetermined size that does not contain fundus structures cannot be identified in the OCT image, the display control unit 204 may switch to a second display mode that displays a second display screen (700C) in which the OCT image and the position information 300, 302, or 310 do not overlap, as shown in FIG. 8, rather than a first display mode that displays a first display screen (700A, 700B) in which the position information 300, 302, or 310 is superimposed on the OCT image as shown in FIG. 7A or 7B. The second display screen may also include a display screen 700D shown in FIG. 17 in which part of the OCT image overlaps with the position information 300, 302, or 310 (i.e., a display screen in which the position information 300, 302, or 310 does not fit within the OCT image, but rather extends beyond the OCT image).

[0066] The above-described embodiment describes image processing when the ophthalmic apparatus 110 is a hybrid model having both the SLO unit 18 and the OCT unit 20. The technology of the present disclosure is applicable not only to the hybrid model but also to a dedicated OCT model (not shown). Since dedicated OCT machines do not have an SLO unit, they can use enface OCT images generated from OCT volume data obtained by scanning a wide field of view instead of UWF-SLO images. Enface OCT images are images cut out from a plane of OCT volume data. Specifically, with dedicated OCT machines, a rough scan is performed to obtain an enface OCT image, and then a higher-resolution OCT image (a high-density scan OCT image with a large number of A-scans per unit area) is obtained at a specified imaging position on the enface OCT image. Due to the principles of OCT, there is a limit to the number of A-scans per unit time, and there is a trade-off between the scanning range and scanning density. Rough scanning means performing low-density scanning over a wide area, while high-density scanning means performing high-density scanning over a narrow area. OCT images obtained with rough scanning have low resolution, while OCT images obtained with high-density scanning have high resolution. A flowchart for generating a display screen showing the positional relationship between an enface OCT image and a high-density scan OCT image will be described with reference to Fig. 14. The CPU 262 of the management server 140 executes this image processing program to realize the functions shown in Fig. 4. In step 120 , the processing unit 208 acquires the subject ID entered by the user in the viewer 150 . In step 122, the fundus image processing unit 2060 acquires the subject information (such as name and age) of the subject ID stored in the storage device 254, the fundus image dataset, which is an enface OCT image (in this embodiment, a frontal image of the fundus, which is an image equivalent to the UWF-SLO image in the previous embodiment), the high-density scan OCT image (in this embodiment, a tomographic image obtained by B-scan), and information regarding the location where the high-density scan OCT image was acquired.

[0067] In step 124, the fundus image processor 2060 generates position information based on information about the position where the high density scan OCT image was acquired.

[0068] In step 126, the fundus image processor 2060 generates a reduced enface OCT image with the acquisition position of the high-density scan OCT image added. The reduced enface OCT image may be a reduced version of the enface OCT image itself, or a silhouette image that shows only an outline of the enface OCT image on an icon. Fundus structures such as ONH, macula, or choroidal blood vessels may be superimposed on the silhouette image using techniques such as computer graphics. Alternatively, the enface image may be processed to deform to fit the shape of the eyeball, resulting in a three-dimensional spherical icon as shown in FIG. 15.

[0069] In step 128, the fundus image processing unit 2060 determines the position at which to superimpose the information on the acquisition position on the high-density scan OCT image (for example, extracting an area where fundus structures are not displayed). In this embodiment, the information on the acquisition position is displayed so as not to overlap with the area showing the fundus structures. In a high-density scan OCT image, areas where fundus structures are not displayed are generally filled with pixels having low brightness values. In this embodiment, the information on the acquisition position is displayed in the area where the pixels with the lowest brightness values ​​are concentrated.

[0070] In step 130, the fundus image processing unit 2060 generates a display screen in the image synthesis unit 2062 in which a reduced enface OCT image is superimposed on the high density scan OCT image.

[0071] In step 132, the processing unit 208 outputs the generated display screen to the storage device 254 for storage, and also outputs the image data of the display screen to the viewer 150, and then ends the program. The viewer 150 receives the image data of the display screen and displays the display screen on the display 156 .

[0072] 15 shows a display screen 800A in which a three-dimensional spherical icon 410 generated from an enface OCT image is superimposed on a high-density scan OCT image 800. The imaging position 402 where the high-density scan was performed is superimposed on the icon 410. In FIG. 15, as in FIG. 7A and other figures, areas composed of pixels with low brightness values ​​are displayed in white for visibility, but these areas appear dark in the actual image. The icon 410 with the imaging position 402 superimposed is displayed in a position that does not overlap with fundus structures in the high-density scan OCT image 800.

[0073] 16 shows a display screen 800B in which a three-dimensional spherical stereoscopically depicted icon 410 generated from an enface OCT image is superimposed on a high-density scan OCT image 810 obtained when a volume scan is performed as the high-density scan. The high-density scan OCT image in FIG. 15 is a tomographic image obtained by a B scan, but FIG. 16 differs from FIG. 15 in that it is a stereoscopic OCT image (3D OCT image) obtained by a volume scan.

[0074] As described above, in this embodiment, the acquisition position of an OCT image is changed to an image showing a wide area of ​​the fundus, for example, an image obtained by processing a UWF-SLO image, and the image is superimposed on the OCT image and displayed. This allows the user to observe a tomographic image of the fundus of the subject's eye while recognizing the acquisition position of the OCT image. This makes it easier for the user to determine the acquisition position of a tomographic image, particularly when observing a tomographic image of the peripheral part of the fundus, which does not contain characteristic structures such as the ONH or macula. This is because OCT images of the peripheral part of the fundus only contain information such as the outer shape of the retina or choroid, making it difficult to determine the acquisition position of the OCT image. Furthermore, because full-range OCT allows for a larger depth range of the retina to be imaged, the proportion of fundus structures such as the retina in the OCT image is smaller. Conversely, low-brightness areas (black areas) in the OCT image are larger due to factors such as the lack of reflection of fixed light or the absorption of measurement light, preventing the reflected light from returning. Therefore, when an OCT image obtained using full-range OCT is enlarged and displayed on the entire display screen, the user cannot easily determine which part of the fundus the OCT image represents. The technology disclosed herein can display OCT imaging position information in areas where no fundus structures are present (black areas of a predetermined size). This provides the advantage that the user can recognize which part of the fundus the OCT image represents, even when an OCT image obtained using full-range OCT is enlarged and displayed.

[0075] The image processing in each of the above-described embodiments is merely an example. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0076] In the above-described embodiments, image processing is assumed to be performed by a software configuration using a computer, but the technology of the present disclosure is not limited to this. For example, instead of a software configuration using a computer, image processing may be performed only by a hardware configuration such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Part of the image processing may be performed by a software configuration, and the remaining part may be performed by a hardware configuration.

[0077] As such, the technology of the present disclosure includes both cases in which image processing is realized by a software configuration using a computer and cases in which it is not realized, and therefore includes the following technologies. (First Technology) Image processing performed by a processor, acquiring an OCT image of the subject's eye; extracting an area other than the structure of the subject's eye from the OCT image; generating a first display screen on which position information regarding a scan position for acquiring the OCT image is superimposed on the area; An image processing method comprising: (Second Technology) a memory and a processor coupled to the memory; The processor: acquiring an OCT image of the subject's eye; extracting an area other than the structure of the subject's eye from the OCT image; generating a first display screen on which position information regarding a scan position for acquiring the OCT image is superimposed on the area; An image processing device that performs the above. (Third Technology) 1. A computer program product for processing images, comprising: the computer program product comprises a computer-readable storage medium that is not itself a transitory signal; The computer-readable storage medium stores a program, The program is configured to: acquiring an OCT image of the subject's eye; extracting an area other than the structure of the subject's eye from the OCT image; generating a first display screen on which position information regarding a scan position for acquiring the OCT image is superimposed on the area; A computer program product that causes the execution of

[0078] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0079] In addition, the disclosure of Japanese Patent Application No. 2021-029129, filed on February 25, 2021, is incorporated herein by reference in its entirety.

Claims

1. an OCT image acquisition unit that acquires an OCT image of the subject's eye; a fundus image processing unit that performs image processing to extract an area other than the structure of the subject's eye based on a ratio of pixels that are present in a predetermined area in the OCT image and have a brightness value lower than a predetermined value; an image synthesis unit that generates a first display screen in which predetermined information related to the OCT image is superimposed on an area other than the structure of the subject's eye; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the predetermined information is position information relating to a scan position for acquiring the OCT image.

3. 3. The image processing apparatus according to claim 2, wherein the position information is character information corresponding to the scan position.

4. The location information is 3. The image processing apparatus according to claim 2, wherein the mark indicating the scan position is an image superimposed on a silhouette image that imitates the shape of a fundus image.

5. an SLO image acquisition unit that acquires an SLO image of the subject's eye, The image processing apparatus according to claim 2 , wherein the position information is an image obtained by adjusting a guide image showing the scanning position on the SLO image to a size suitable for the region.

6. the image synthesis unit generates a second display screen in which the position information and the OCT image are laid out so as not to overlap each other; The image processing device according to claim 2 , further comprising a display control unit that switches between outputting the first display screen and outputting the second display screen.

7. The image processing device according to claim 1 , wherein the fundus image processing unit performs the image processing on an area smaller than the specified area when the image processing fails to extract the specified area as an area other than a structure of the test eye.

8. Computer, a fundus image processing unit that performs image processing to extract an area other than the structure of the subject's eye based on a ratio of pixels that are below a predetermined brightness value and exist in a predetermined area in the OCT image of the subject's eye acquired by the OCT image acquisition unit, and an image synthesis unit that generates a first display screen in which predetermined information related to the OCT image is superimposed on the area other than the structure of the subject's eye; An image processing program that functions as a

Images