Ophthalmic image processing device and ophthalmic image processing program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NIDEK CO LTD
- Filing Date
- 2022-03-31
- Publication Date
- 2026-08-04
Smart Images
Figure 0007899565000001 
Figure 0007899565000002 
Figure 0007899565000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an ophthalmic image processing apparatus that processes data of an ophthalmic image, which is a tomographic image of a tissue of an eye to be examined, and an ophthalmic image processing program executed in the ophthalmic image processing apparatus.
Background Art
[0002] Techniques for capturing tomographic images of tissues of an eye to be examined are known. For example, the OCT apparatus of Patent Document 1 captures a two-dimensional tomographic image of a cross section passing through a scanning line by scanning measurement light on the scanning line. Further, a three-dimensional tomographic image is obtained by arranging a plurality of two-dimensional tomographic images captured for a plurality of scanning lines.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] Generally, data of the captured tomographic image is stored in a storage device as it is. As a result, problems such as compression of the storage capacity of the storage device and a decrease in the processing speed when processing the stored data are likely to occur. In particular, in recent years, the high definition, high resolution, high depth, and high speed of tomographic image capture have advanced, and the frequency of capturing tomographic images for diagnosis and the like has also increased. Therefore, reducing the amount of data of the tomographic image to be stored in the storage device has become an important issue.
[0005] A typical object of the present disclosure is to provide an ophthalmic image processing apparatus and an ophthalmic image processing program capable of appropriately processing data of a tomographic image of a tissue of an eye to be examined.
Means for Solving the Problems
[0006] An ophthalmic image processing apparatus provided in a typical embodiment of this disclosure is an ophthalmic image processing apparatus that processes data of an ophthalmic image, which is a tomographic image of the tissue of an eye to be examined, wherein the control unit of the ophthalmic image processing apparatus includes: an image acquisition step of acquiring an ophthalmic image taken by an ophthalmic image acquisition device; an extraction step of extracting image data within a target range for data storage from each of a plurality of partial images arranged in a predetermined direction within the ophthalmic image; and a position information acquisition step of acquiring position information which is information indicating the position of each of the plurality of extracted images extracted from each of the plurality of partial images in the extraction step, and is information for arranging the data of the plurality of extracted images to reconstruct the region extracted in the extraction step from the ophthalmic image. After excluding the data of images other than those from which the extracted images were extracted from the aforementioned multiple partial images from the data to be saved, A storage step involves storing the data of the plurality of extracted images and the position information of each of the plurality of extracted images in a storage means. An image reconstruction step in which the region extracted in the extraction step is reconstructed from the ophthalmic image by arranging the data of the plurality of extracted images stored in the storage means based on the respective position information, and a background interpolation step in which pixel information of the background region where information is missing is added to the image reconstructed in the image reconstruction step. Execute this.
[0007] An ophthalmic image processing program provided in a typical embodiment of this disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes data of an ophthalmic image, which is a tomographic image of the tissue of an eye to be examined, wherein the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device, and comprises: an image acquisition step of acquiring an ophthalmic image taken by an ophthalmic image acquisition device; an extraction step of extracting image data within a target range for data storage from each of a plurality of partial images arranged in a predetermined direction within the ophthalmic image; and a position information acquisition step of acquiring position information which is information indicating the position of each of the plurality of extracted images extracted from each of the plurality of partial images in the extraction step, and is information for arranging the data of the plurality of extracted images to reconstruct the region extracted in the extraction step from the ophthalmic image. After excluding the data of images other than those from which the extracted images were extracted from the aforementioned multiple partial images from the data to be saved, A storage step involves storing the data of the plurality of extracted images and the position information of each of the plurality of extracted images in a storage means. An image reconstruction step in which the region extracted in the extraction step is reconstructed from the ophthalmic image by arranging the data of the plurality of extracted images stored in the storage means based on the respective position information, and a background interpolation step in which pixel information of the background region where information is missing is added to the image reconstructed in the image reconstruction step. The ophthalmic image processing device is made to execute the above.
[0008] According to the ophthalmic image processing device and ophthalmic image processing program relating to this disclosure, tomographic image data of the tissue of the eye being examined is processed appropriately. [Brief explanation of the drawing]
[0009] [Figure 1] This is a block diagram showing the schematic configuration of the ophthalmic image processing system 100. [Figure 2] This is an explanatory diagram illustrating how the ophthalmic imaging device 1 captures ophthalmic images of the tissue 50 of the eye being examined. [Figure 3] This figure shows an example of an ophthalmic image 61 taken by the ophthalmic imaging device 1. [Figure 4] This flowchart shows an example of ophthalmic image processing performed by the ophthalmic image processing device 40. [Figure 5] This figure shows the captured ophthalmic image 61 and the ophthalmic image 62 from which the image region has been extracted from the ophthalmic image 61. [Figure 6] This image is extracted from partial image 70 within ophthalmic image 62. It is an explanatory diagram illustrating one example of how to extract image 80. [Figure 7] This figure shows a reference image 80A obtained by arranging multiple extracted images 80, each extracted from a multiple partial image 70, in the X direction without considering their respective positions. [Figure 8] This figure shows an example of reconstructed image data 90 and background-filled image data 91. [Modes for carrying out the invention]
[0010] <Overview> The ophthalmic image processing device illustrated in this disclosure processes ophthalmic image data, which are tomographic images of tissue of an eye under examination. The control unit of the ophthalmic image processing device performs an image acquisition step, an extraction step, a position information acquisition step, and a storage step. In the image acquisition step, the control unit acquires an ophthalmic image captured by an ophthalmic image capture device. In the extraction step, the control unit extracts image data within the target range for data storage from each of a plurality of partial images arranged in a predetermined direction within the ophthalmic image. In the position information acquisition step, the control unit acquires position information indicating the position of each of the plurality of extracted images extracted from each of the plurality of partial images in the extraction step. In the storage step, the control unit causes the data of the plurality of extracted images and the position information of each of the plurality of extracted images to be stored in a storage means.
[0011] According to the technology disclosed herein, extracted images are obtained from each of multiple partial images constituting an ophthalmic image, specifically images within the target range for data storage. Furthermore, the positional information of each extracted image is stored along with the data of the multiple extracted images. In other words, data from an appropriate range of images within the ophthalmic image is extracted and stored. In addition, if each extracted image is positioned based on its positional information, the extracted portion of the image is appropriately reconstructed. Therefore, compared to the case where the entire data of the ophthalmic image is stored, the strain on the storage capacity of the memory device is appropriately suppressed.
[0012] Various devices can be used as ophthalmic imaging devices for acquiring (generating) tomographic images. For example, an OCT device that acquires tomographic images of the tissue of the eye under examination using the principle of optical coherence tomography can be used. In this case, the tomographic image may be a motion contrast image (e.g., an OCT angiography image) obtained by acquiring multiple OCT signals at different times from the same position in the retinal layer of the fundus. MRI (magnetic resonance imaging) devices or CT (computed tomography) devices may also be used as ophthalmic imaging devices. The tomographic image acquired in the image acquisition step may be a two-dimensional tomographic image or a three-dimensional tomographic image. Furthermore, the tomographic image in this disclosure is a tomographic image of the fundus of the eye under examination. However, the tomographic image may also be a tomographic image of tissue other than the fundus in the eye under examination (e.g., the anterior segment).
[0013] Various devices can function as ophthalmic image processing devices. For example, an ophthalmic imaging device that takes tomographic images may itself function as an ophthalmic image processing device in this disclosure. In this case, the ophthalmic imaging device can take tomographic images of the eye under examination and appropriately store the captured tomographic images in a storage means. Alternatively, a device capable of exchanging data with an ophthalmic imaging device (e.g., a personal computer (PC) or a mobile terminal) may function as an ophthalmic image processing device. Multiple control units (e.g., the control unit of an ophthalmic imaging device and the control unit of a PC) may cooperate to perform processing.
[0014] The control unit may further perform a reference site identification step in which it identifies at least one of several parts within the tissue visible in the ophthalmic image as a reference site. In the extraction step, the control unit may extract image data within the target range based on the reference site from each of several partial images arranged in a predetermined direction within the ophthalmic image as extracted images. In this case, image data within an appropriate range based on the identified reference site is extracted and stored. Therefore, image data from unnecessary ranges is appropriately excluded, and image data from necessary ranges is stored.
[0015] A method for specifying a reference site within a tissue shown in an ophthalmic image can be appropriately selected. For example, the control unit may specify the reference site within the ophthalmic image by known image processing (such as edge detection or the like). Also, a mathematical model trained by a machine learning algorithm may be used to specify the reference site within the input ophthalmic image. In this case, the control unit may input the ophthalmic image into the mathematical model and obtain the specification result of the reference site output by the mathematical model to specify the reference site.
[0016] The ophthalmic image may be a tomographic image that captures a layer structure (such as a fundus layer structure or the like) in which a plurality of layers are stacked among the tissues of the subject eye. In the reference site specification step, the control unit may specify at least any one of the plurality of layers included in the layer structure and the boundaries of the layers (hereinafter sometimes referred to as "layer·boundaries") as the reference site. The layer·boundaries included in the layer structure tend to have a shape along the overall shape of the layer structure. Therefore, even when the layer structure is not flat (for example, when the layer structure is curved or when a deformation due to a disease or the like has occurred), by extracting an extraction image with the layer·boundary as the reference site, an appropriate range of images corresponding to the shape of the layer structure is likely to be saved.
[0017] In the reference site specification step, the control unit may specify at least the boundary or layer on the outermost layer side among the layer structures shown in the ophthalmic image as the reference site. The outermost layer side of the layer structure is the boundary between the region where the layer structure exists and the region where the layer structure does not exist. Therefore, by using the layer·boundary on the outermost layer side of the layer structure as the reference site, an extraction image including the layer structure is more likely to be appropriately extracted. When the tissue shown in the ophthalmic image is the retinal tissue of the fundus, the layer·boundary on the outermost layer side is the retinal surface or the boundary of the retinal surface (for example, the boundary on the surface side).
[0018] In the extraction step, the control unit may include, in the target range for extracting the extraction image from each of the plurality of partial images, a predetermined range from the reference site on the outermost surface side in the layer structure toward the deep layer side of the layer structure. In this case, as large a range as possible in the layer structure is likely to be included in the extraction image. Also, the problem that an image of the tissue on the surface side of the layer structure is not extracted is less likely to occur. Therefore, an image of the layer structure of the eye to be examined is more appropriately extracted and saved.
[0019] Note that the size of the image extraction range from the reference site on the outermost surface side in the layer structure toward the deep layer side (that is, the size of the "predetermined range toward the deep layer side" described above) can be set as appropriate. For example, the control unit may set the size of the image extraction range from the reference site toward the deep layer side (hereinafter sometimes referred to as the "offset amount toward the deep layer side") according to an instruction input by the user. In this case, the user can efficiently and appropriately save the data of the image of the desired range in the layer structure of the eye to be examined. Also, the control unit may adjust the offset amount toward the deep layer side according to at least any one of the imaging conditions of the ophthalmic image (for example, imaging magnification, imaging angle of view, imaging depth, etc.). In this case, the data of an image in an appropriate range according to the imaging conditions is extracted. Also, the offset amount toward the deep layer side may be a predetermined fixed value.
[0020] In the extraction step, the control unit may include, in the target range for extracting the extraction image from each of the plurality of partial images, a predetermined range from the reference site on the outermost surface side in the layer structure toward the side opposite to the deep layer side of the layer structure (hereinafter, for convenience, referred to as the "front side"). In this case, for example, even if the identification accuracy of the reference site on the outermost surface side decreases, an image of the tissue on the surface side of the layer structure is likely to be appropriately extracted. Also, even if some object is reflected on the surface side of the outermost layer / boundary due to various influences (for example, the influence of a disease, etc.), the object reflected on the surface side is also likely to be extracted. Therefore, an image is more likely to be appropriately extracted.
[0021] In addition, similar to the offset amount toward the deeper layers, the size of the image extraction range toward the foreground from the reference area on the surface side (hereinafter sometimes referred to as the "foreground offset amount") can also be set as appropriate. For example, the control unit may set the foreground offset amount according to instructions entered by the user. The control unit may also adjust the foreground offset amount according to the ophthalmic image acquisition conditions (for example, at least one of the following: magnification, field of view, depth of view, etc.). Furthermore, the foreground offset amount may be a predetermined fixed value.
[0022] Furthermore, among the layered structures visible in ophthalmic images, it is also possible to use layers or boundaries located deeper than the outermost layer or boundary as reference sites. For example, if the tissue visible in an ophthalmic image is retinal tissue of the fundus, at least one of the layers or boundaries located deeper than the outermost retinal surface (e.g., ganglion cell layer, inner plexiform layer, inner granular layer, outer plexiform layer, outer limiting membrane, intracellular and external segmental junction of photoreceptor cells, retinal pigment epithelium, Bruch's membrane, choroid, etc., and their boundaries) may be used as a reference site. For example, by using a layer or structure that is more clearly visible among the layered structures visible in ophthalmic images as a reference site, the accuracy of identifying the reference site may be improved, and images may be extracted and saved more appropriately.
[0023] Furthermore, the layers and boundaries identified as reference sites within the layered structure visible in ophthalmic images may be changed according to various conditions. For example, there are layers and boundaries whose visibility in ophthalmic images changes depending on the location of the tissue. Therefore, the layers and boundaries identified as reference sites may be changed depending on the location of the tissue visible in ophthalmic images.
[0024] Furthermore, the control unit may identify multiple layers and boundaries visible in the ophthalmic image as reference areas. In this case, during the extraction step, the control unit may select a range from the identified reference areas (e.g., the outermost layer or boundary) and other reference areas (e.g., layers or boundaries located on the deeper side of the layered structure) as the target range and extract images from each of the multiple partial images. As a result, image data within a specific range of the layered structure visible in the ophthalmic image is appropriately extracted and stored.
[0025] Each of the multiple partial images may be a long image extending in a direction that intersects the layers within the layered structure visible in the ophthalmic image. The multiple partial images may be arranged continuously within the ophthalmic image in a direction that intersects the direction in which each partial image extends. In this case, the layer and boundary designated as the reference area within each partial image becomes easier to identify. As a result, more appropriately extracted images are extracted and saved.
[0026] For example, if the ophthalmic image is an OCT image, the multiple partial images may be A-scan images extending in a direction along the optical axis of the OCT measurement light. The OCT measurement light is irradiated from the front to the tissue of the eye being examined (e.g., fundus tissue). Therefore, the A-scan images extending in a direction along the optical axis of the OCT measurement light become images (pixel rows) that extend in a direction intersecting the layers within the tissue. Thus, by using multiple A-scan images that constitute the ophthalmic image as partial images, it becomes easier to extract images that include appropriate regions of the layered structure. Note that each A-scan image may be used as a partial image as is, or multiple A-scan images may be used as the constituent units of each partial image.
[0027] However, it is also possible to modify the partial images. For example, if the ophthalmic image is an OCT image, multiple pixel sequences that intersect perpendicularly with the A-scan image (so-called C-scan images) may be used as partial images. Alternatively, the direction in which at least one layer or boundary (for example, a layer or boundary designated as a reference area) extends may be detected, and multiple partial images intersecting in the detected direction may be appropriately set. Furthermore, if the layer structure is curved, a collection of multiple partial images may be arranged in a sector shape or similar configuration so that each of the multiple partial images intersects with the curved shape of the layer structure.
[0028] Furthermore, it is possible to use a reference area other than layers and boundaries among the multiple areas visible in the ophthalmic image. For example, a diseased area visible in the ophthalmic image may be identified as a reference area, and extracted images may be extracted based on this identified reference area. In this case, the extracted images will be appropriately extracted and saved according to the diseased area. Alternatively, vascular areas, etc., visible in the ophthalmic image may be identified as reference areas.
[0029] If, as a result of the reference area identification step, there are areas where the reference area identification result is missing, a further interpolation step may be performed to fill in the missing reference area identification result based on the reference area identification result in other areas. In this case, even if some of the reference area is not identified due to various influences (e.g., image quality or disease), the extracted image will be extracted from multiple partial images based on the reference area whose identification result has been interpolated by the interpolation step. Therefore, the image may be extracted and saved more appropriately.
[0030] The specific method of the interpolation step can also be selected as appropriate. For example, if there are missing locations where the identification result is missing, the control unit may interpolate the identification result of the missing location based on the identification result of a reference region in a location adjacent to the missing location (for example, a partial image adjacent to the partial image containing the missing location). For example, the control unit may interpolate the reference region position by using the position between the identification position of a partial image adjacent to one side of the partial image containing the missing location and the identification position of a partial image adjacent to the other side as the reference region position.
[0031] In addition, it is possible to extract images using methods other than the method of extracting images from ophthalmic images based on a reference area. For example, the control unit may perform image processing on the acquired ophthalmic image to detect the image region containing the image, and extract the image data within the target range including the detected image region as the extracted image. Alternatively, the control unit may extract the image data of an area in the ophthalmic image that is highly likely to contain an image (for example, an area in the ophthalmic image that is predetermined based on the expectation that an image will be found) as the extracted image.
[0032] In the extraction step, the multiple extracted images are arranged in the direction of the multiple partial images without considering positional information to form an image (reference image). This image may be obtained by removing at least a portion of the areas other than the layer structure from the ophthalmic image acquired in the image acquisition step, and by flattening the curvature of the layer structure. In this case, in the reference image, at least a portion of the areas other than the layer structure is removed from the ophthalmic image, appropriately reducing the amount of image data, and the layer curvature information reproduced by positional information is removed, resulting in a flattened layer structure. Therefore, the necessary data is appropriately extracted and saved from the ophthalmic image data.
[0033] The extracted images may be extracted from the ophthalmic image and saved while maintaining the thickness of the layer structure in the ophthalmic image. The extracted image data may be saved either uncompressed or with lossless compression. In this case, if each extracted image is positioned based on its location information, the layer structure in the original ophthalmic image will be appropriately reconstructed while maintaining its thickness.
[0034] The control unit may further perform an image reconstruction step in which it reconstructs the region extracted in the extraction step from the ophthalmic image by arranging the data of multiple extracted images stored in the storage means based on their respective positional information. In this case, the extracted images stored with a smaller amount of data than the original ophthalmic image are appropriately reconstructed.
[0035] Furthermore, the control unit can store the differences between the data of multiple extracted images and the data of other extracted images, rather than storing the data of multiple extracted images as is. For example, the control unit may reduce the amount of data for some extracted images by storing the differences between some extracted images and the data of neighboring (e.g., adjacent) extracted images. When reconstructing the image, the control unit can simply combine the data of some extracted images with the data of neighboring extracted images based on the differences.
[0036] <Embodiment> The following describes one typical embodiment of the present disclosure. In this embodiment, we will illustrate the processing of a tomographic image of the fundus tissue of the eye E under examination, which is taken by an OCT device, a type of ophthalmic imaging device. However, the ophthalmic image to be processed may be an image of tissue other than the fundus tissue. For example, the ophthalmic image to be processed may be an image of tissue other than the fundus of the eye E under examination (e.g., the anterior segment). In addition, images of biological tissue other than the eye E under examination (e.g., skin, digestive organs, or brain, etc.) (i.e., images other than ophthalmic images) may be the subject of processing. Furthermore, as mentioned above, the imaging device is not limited to an OCT device.
[0037] Referring to Figure 1, the schematic configuration of the ophthalmic image processing system 100 of this embodiment will be described. The ophthalmic image processing system 100 of this embodiment comprises an ophthalmic image acquisition device 1 and an ophthalmic image processing device 40. The ophthalmic image acquisition device 1 acquires a tomographic image of a living organism (in this embodiment, the fundus of the eye under examination). In detail, the ophthalmic image acquisition device (OCT device) 1 of this embodiment acquires a two-dimensional image that extends in a first direction (scanning direction) and a second direction intersecting the first direction (depth direction, which is the direction of the optical axis of the measurement light) by scanning measurement light on the tissue of the living organism and continuously receiving light from the tissue over time. The ophthalmic image acquisition device 1 can also acquire a three-dimensional tomographic image of the living organism by scanning measurement light on each of multiple scan lines within a two-dimensional measurement area in the living organism and acquiring multiple two-dimensional tomographic images. The ophthalmic image processing device 40 performs processing of the ophthalmic image data acquired (captured) by the ophthalmic image acquisition device 1.
[0038] The configuration of the ophthalmic imaging device 1 of this embodiment will now be described. The ophthalmic imaging device (OCT device) 1 comprises an OCT unit 10 and a control unit 30. The OCT unit 10 comprises an OCT light source 11, a coupler (optical divider) 12, a measurement optical system 13, a reference optical system 20, and a light-receiving element 22.
[0039] The OCT light source 11 emits light (OCT light) for acquiring ophthalmic image data. The coupler 12 splits the OCT light emitted from the OCT light source 11 into measurement light and reference light. In addition, the coupler 12 in this embodiment combines and interferes the measurement light reflected by tissue (in this embodiment, the fundus of the eye E under examination) with the reference light generated by the reference optical system 20. In other words, the coupler 12 in this embodiment serves as both a branching optical element that splits the OCT light into measurement light and reference light, and a multiplexing optical element that combines the reflected measurement light with the reference light. It is also possible to change the configuration of at least one of the branching optical element and the multiplexing optical element. For example, elements other than the coupler (e.g., a circulator, beam splitter, etc.) may be used.
[0040] The measurement optical system 13 guides the measurement light, split by the coupler 12, to the subject and returns the measurement light reflected by the tissue back to the coupler 12. The measurement optical system 13 comprises a scanning unit (scanner) 14, an illumination optical system 16, and a focus adjustment unit 17. The scanning unit 14 is driven by a drive unit 15, which allows it to scan a spot-shaped measurement light in a two-dimensional direction intersecting the optical axis of the measurement light. In this embodiment, two galvanometer mirrors capable of deflecting the measurement light in different directions are used as the scanning unit 14. However, another device that deflects light (e.g., at least one of a polygon mirror, resonant scanner, or acousto-optic element) may be used as the scanning unit 14. The illumination optical system 16 is located downstream of the scanning unit 14 in the optical path (i.e., on the subject side) and irradiates the tissue with the measurement light. The focus adjustment unit 17 adjusts the focus of the measurement light by moving an optical element (e.g., a lens) provided in the illumination optical system 16 in a direction along the optical axis of the measurement light.
[0041] The reference optical system 20 generates reference light and returns it to the coupler 12. In this embodiment, the reference optical system 20 generates reference light by reflecting the reference light split by the coupler 12 using a reflective optical system (e.g., a reference mirror). However, the configuration of the reference optical system 20 can also be changed. For example, the reference optical system 20 may transmit the light incident from the coupler 12 without reflection and return it to the coupler 12. The reference optical system 20 includes an optical path length difference adjustment unit 21 that changes the optical path length difference between the measurement light and the reference light. In this embodiment, the optical path length difference is changed by moving the reference mirror in the optical axis direction. The configuration for changing the optical path length difference may be provided in the optical path of the measurement optical system 13.
[0042] The photodetector 22 detects the interference signal by receiving the interference light between the measurement light and the reference light generated by the coupler 12. In this embodiment, the principle of Fourier-domain OCT is employed. In Fourier-domain OCT, the spectral intensity of the interference light (spectral interference signal) is detected by the photodetector 22, and a complex OCT signal is obtained by performing a Fourier transform on the spectral intensity data. Examples of Fourier-domain OCT include Spectral-domain-OCT (SD-OCT) and Swept-source-OCT (SS-OCT). It is also possible to employ, for example, Time-domain-OCT (TD-OCT).
[0043] The control unit 30 controls various aspects of the ophthalmic imaging device 1. The control unit 30 includes a CPU 31, RAM 32, ROM 33, and non-volatile memory (NVM) 34. The CPU 31 is a controller that performs various controls. The RAM 32 temporarily stores various information. The ROM 33 stores programs executed by the CPU 31, as well as various initial values. The NVM 34 is a non-transient storage medium that can retain its contents even when the power supply is interrupted.
[0044] A monitor 37 and an operation unit 38 are connected to the control unit 30. The monitor 37 is an example of a display unit that displays various images. The operation unit 38 is operated by the user to input various operation instructions to the ophthalmic imaging device 1. Various devices such as a mouse, keyboard, touch panel, and foot switch can be used for the operation unit 38. Alternatively, various operation instructions may be input to the ophthalmic imaging device 1 by inputting sound into the microphone.
[0045] The general configuration of the ophthalmic image processing device 40 will now be described. In this embodiment, a personal computer (hereinafter referred to as "PC") is used as the ophthalmic image processing device 40. However, a device other than a PC may be used as the ophthalmic image processing device. For example, the ophthalmic image acquisition device 1 itself may function as an ophthalmic image processing device that performs the ophthalmic image processing described later. The ophthalmic image processing device 40 includes a CPU 41, RAM 42, ROM 43, and NVM 44. The CPU 41 is a controller that performs various controls. The RAM 42, ROM 43, and NVM 44 can store various information as described above. The ophthalmic image processing program for performing the ophthalmic image processing described later (see Figure 4) may be stored in the NVM 44. In addition, a monitor 47 and an operation unit 48 are connected to the ophthalmic image processing device 40. The monitor 47 is an example of a display unit that displays various images. The operation unit 48 is operated by the user to input various operation instructions to the ophthalmic image processing device 40. The control unit 48 can use various devices such as a mouse, keyboard, and touch panel, similar to the control unit 38 of the ophthalmic imaging device 1. In addition, various operation instructions may be input to the ophthalmic imaging device 40 by inputting sound into the microphone 46.
[0046] The ophthalmic image processing device 40 can acquire various data (for example, data of ophthalmic images taken by the ophthalmic image acquisition device 1) from the ophthalmic image acquisition device 1. The various data can be acquired by at least one of the following methods: wired communication, wireless communication, and a removable storage device (for example, a USB memory).
[0047] Referring to Figures 2 and 3, an example of a method for capturing ophthalmic images to be processed by the ophthalmic image processing device 40 of this embodiment, and an example of the configuration of the ophthalmic image will be described. In this embodiment, an example is given of processing two-dimensional tomographic image data and saving it to a storage device (e.g., NVM44). However, the techniques illustrated in this disclosure can also be applied to processing three-dimensional tomographic image data and saving it to a storage device.
[0048] As shown in Figure 2, the ophthalmic imaging device 1 of this embodiment scans a spot-shaped light (measurement light) on living tissue 50 (in the example shown in Figure 2, fundus tissue). More specifically, the ophthalmic imaging device 1 of this embodiment scans the light on a scan line 52 extending in a predetermined direction on the tissue 50, thereby capturing an ophthalmic image (two-dimensional tomographic image) 61 (see Figure 3) that extends in the Z direction along the optical axis of the light and in the X direction intersecting the Z direction (perpendicularly in this embodiment).
[0049] As shown in Figure 3, in this embodiment, the direction in which the spot-shaped measurement light is scanned over the tissue (referred to as the "B scan direction") is defined as the X direction. The depth direction of the tissue 50 along the optical axis of the measurement light (i.e., the direction perpendicular to the X direction, referred to as the "A scan direction") is defined as the Z direction. The ophthalmic image 61 is a tomographic image capturing the layered structure of multiple layers of biological tissue. Specifically, the ophthalmic image 61 in this embodiment captures the layered structure of the fundus tissue of the eye being examined. As described above, the ophthalmic image acquisition device 1 scans the measurement light irradiated from the front side of the fundus tissue in the X direction, thereby capturing a two-dimensional ophthalmic image 61 that extends in the X direction (B scan direction) and the Z direction (A scan direction). Therefore, the entirety of the layers within the layered structure captured in the ophthalmic image 61 intersects in the Z direction (A scan direction).
[0050] The ophthalmic image 61 of this embodiment is composed of multiple elongated A-scan images (i.e., pixel rows extending in the Z direction along the optical axis of the measurement light) arranged in a predetermined X direction (a direction intersecting the Z direction) extending in the Z direction (A-scan direction). The multiple A-scan images extend in a direction intersecting the layers of the layered structure captured in the ophthalmic image 61. As will be described in detail later, in this embodiment, each of the multiple A-scan images is treated as a partial image, which is the unit from which the image to be saved is extracted.
[0051] The ophthalmic image processing in this embodiment will be described with reference to Figures 4 to 8. In this embodiment, the ophthalmic image processing device 40, which is a PC, acquires data of ophthalmic images 61 (hereinafter sometimes simply referred to as "ophthalmic images 61") from the ophthalmic image acquisition device 1, processes the acquired data of ophthalmic images 61, and stores it in a storage device. However, as mentioned above, other devices may also function as ophthalmic image processing devices. For example, the ophthalmic image acquisition device 1 itself may perform ophthalmic image processing. Alternatively, multiple control units (for example, the CPU 31 of the ophthalmic image acquisition device 1 and the CPU 41 of the ophthalmic image processing device 40) may cooperate to perform ophthalmic image processing. The CPU 41 of the ophthalmic image processing device 40 performs the ophthalmic image processing shown in Figure 4 according to the ophthalmic image processing program stored in the NVM 44.
[0052] First, the CPU 41 acquires an ophthalmic image 61 of the tissue of the eye being examined (in this embodiment, the retinal tissue of the fundus) (S1). As illustrated in Figures 3 and 5, the ophthalmic image 61 in this embodiment shows the layered structure of the retinal tissue of the fundus.
[0053] The CPU 41 pre-extracts a portion of the ophthalmic image 61 acquired in S1 that contains images of tissue (image region) (S2). As shown in Figure 5, the ophthalmic image acquisition device 1 of this embodiment can capture tissue at a high depth, even deeper than the deepest edge of the retinal tissue (the lower edge of the image in the ophthalmic image 61 in Figure 5). However, when using the ophthalmic image 61 for diagnosis, it is often sufficient to observe the retinal tissue. Therefore, in this embodiment, the CPU 41 identifies the image region in the ophthalmic image 61 acquired in S1 that contains images of tissue (the region shown within the dotted frame in the ophthalmic image 61 in Figure 5), and pre-extracts the ophthalmic image 62 containing the detected image region from the ophthalmic image 61 acquired in S1. As a result, the processing load of various processes executed thereafter (for example, the process of identifying a reference area in S3) and the amount of data in the image data to be processed are reduced.
[0054] The method for identifying the image region within the ophthalmic image 61 can be selected as appropriate. For example, the CPU 41 may identify the image region by performing known image processing on the ophthalmic image 61. Alternatively, a mathematical model pre-trained by a machine learning algorithm to identify the image region within the ophthalmic image 61 may be used. In this case, the CPU 41 may input the ophthalmic image 61 into the mathematical model and obtain the image region identification result output by the mathematical model. It is also possible to omit the processing in S2.
[0055] Next, the CPU 41 identifies at least one of several tissue regions visible in the ophthalmic image 61 acquired in S1 as a reference region (S3). As will be described in detail later, the reference region serves as a criterion for extracting images within the range to be saved from the ophthalmic image 61 acquired in S1 (in this embodiment, the ophthalmic image 62 from which the image region has been extracted from the ophthalmic image 61).
[0056] As shown in Figure 6, the ophthalmic image 62 shows a layered structure of multiple layers within the tissue of the eye being examined. In S3 of this embodiment, at least one of the multiple layers and layer boundaries included in the layered structure shown in the ophthalmic image 62 is identified as the reference area RS. Each layer and boundary included in the layered structure tends to have a shape that conforms to the overall shape of the layered structure. Therefore, even if the layered structure is not flat (for example, if the layered structure is curved), by extracting images using the layers and boundaries as reference area RS in S6 and S7 described later, it becomes easier to save images of an appropriate range corresponding to the shape of the layered structure.
[0057] In detail, in S3 of this embodiment, the outermost boundary or layer (the upper end in Figure 6) of the layered structure visible in the ophthalmic image 62 (the retinal surface in this embodiment) is identified as the reference area RS. The outermost layer of the layered structure is the boundary between the region where the layered structure exists and the region where the layered structure does not exist. Therefore, by using the outermost layer boundary of the layered structure as the reference area RS, images containing the layered structure can be extracted more appropriately.
[0058] In step S3 of this embodiment, the CPU 41 identifies the reference region RS within the ophthalmic image 62 by performing known image processing (e.g., edge detection) on the ophthalmic image 62. However, it is also possible to change the method for identifying the reference region RS. For example, the CPU 41 may input the ophthalmic image 62 into a mathematical model trained by a machine learning algorithm and obtain the result of identifying the reference region RS output by the mathematical model.
[0059] Next, if there are areas in the ophthalmic image 62 where the identification result of the reference area RS is missing, the CPU 41 supplements the identification result of the reference area RS in the missing area based on the identification result of the reference area RS in other areas (areas where the identification result was obtained normally) (S4). As a result, even if a part of the reference area RS is not identified in S3, in S6 and S7 described later, the image within the storage range is appropriately extracted based on the reference area RS. As an example, in S4 of this embodiment, the identification result of the missing area is supplemented based on the identification result of the reference area RS in an area adjacent to the missing area (for example, an A-scan image adjacent to the A-scan image containing the missing area). Specifically, the CPU 41 supplements the position of the reference area RS in the missing area by using the position between the identified position of the A-scan image adjacent to one side of the A-scan image containing the missing area (which may be a series of consecutive A-scan images) and the identified position of the A-scan image adjacent to the other side as the position of the reference area RS in the missing area. However, it is also possible to change the interpolation method.
[0060] Next, the CPU 41 identifies the extraction range of the Nth (initial value of N is "1") partial image 70 from among the multiple partial images 70 arranged in a predetermined direction within the ophthalmic image 62, based on the reference region RS (S6). As shown in Figure 6, the multiple partial images 70 are arranged continuously in a predetermined direction within the ophthalmic image 62. In this embodiment, each of the multiple partial images 70 is a long image extending in a direction intersecting the layers within the layered structure captured in the ophthalmic image 62 (in this embodiment, the Z direction), and is arranged continuously in a direction intersecting the direction in which the partial images 70 extend (in this embodiment, the X direction). Therefore, the layer / boundary designated as the reference region (in this embodiment, the outermost layer / boundary) is easily identified within each partial image 70. As mentioned above, in this embodiment, the multiple partial images 70 are A-scan images extending in a direction along the optical axis of the OCT measurement light.
[0061] An example of a method for identifying the extraction range of each partial image 70 based on a reference region RS will be described in detail. In this embodiment, the CPU 41 includes a predetermined range in the extraction range from the outermost reference region RS in the layer structure of the ophthalmic image 62 toward the deeper layers of the layer structure (the lower side of the drawing, which is the +Z direction in the example shown in Figure 6). In other words, the CPU 41 identifies the position of the partial image 70 extending in the Z direction, where an offset amount (+offset1) toward the deeper layers (+Z direction) is added to the Z coordinate of the reference region RS, as the deep-side end DE of the extraction range in the partial image 70. As a result, the range from the outermost reference region RS toward the deeper layers of the layer structure is extracted afterward, making it easier to extract as wide a range of the layer structure as possible.
[0062] Furthermore, the CPU 41 includes a predetermined range in the extraction target range from the most superficial reference site RS in the layered structure of the ophthalmic image 62 toward the front side (the upper side of the drawing, which is the -Z direction in the example shown in Figure 6), opposite to the deeper side of the layered structure. In other words, the CPU 41 identifies the position obtained by adding an offset amount (-offset2) toward the front side (+Z direction) to the Z coordinate of the reference site RS in the partial image 70 extending in the Z direction as the front end SE of the extraction target range in the partial image 70. As a result, for example, even if the accuracy of identifying the most superficial reference site RS decreases, images of the superficial tissues in the layered structure become easier to extract appropriately. Also, even if some object is captured on the surface side beyond the most superficial layer boundary due to various influences (for example, the influence of disease), the object captured on the surface side also becomes easier to extract.
[0063] In this embodiment, the CPU 41 sets the offset amount to the deeper layer (+ofset1) and the offset amount to the front layer (-ofset2) according to the instructions entered by the user. Therefore, the user can efficiently and appropriately save image data of a desired range of the layer structure of the eye under examination. The CPU 41 may also adjust at least one of the two offset amounts according to the shooting conditions of the ophthalmic image 61 (for example, at least one of the following: magnification, field of view, depth of view, etc.). In this case, image data of an appropriate range according to the shooting conditions will be extracted. At least one of the two offset amounts may also be a predetermined fixed value.
[0064] Next, the CPU 41 extracts data from the Nth partial image 70 within the range to be extracted (extracted image 80) (S7). As shown in Figure 6, the extracted image 80 extracted from each partial image 70 tends to include an appropriate range based on the reference region RS, while unnecessary ranges are easily excluded.
[0065] Figure 7 shows a reference image 80A obtained by arranging multiple extracted images 80, each extracted from a plurality of partial images 70, in the X direction (i.e., the arrangement direction of the plurality of partial images 70) without considering their respective positions. As shown in Figure 7, the reference image 80A, which is a collection of multiple extracted images 80, includes an appropriate range of tissue (layer structure in this embodiment) based on the identified reference region RS. Furthermore, it can be seen that the amount of image data is smaller compared to the ophthalmic images 61 and 62. In other words, in reference image 80A, at least a portion of the areas other than the layer structure has been removed from the ophthalmic images 61 and 62, and the curvature of the layer structure has been flattened. By removing at least a portion of the areas other than the layer structure, the amount of image data is appropriately reduced, and the layer curvature information reproduced by positional information is removed, resulting in a flattened layer structure. In the reference image 80A shown in Figure 7, the background portion (white portion in Figure 8) in the reconstructed image data 90 (details will be described later) shown in Figure 8 has been removed from the ophthalmic image 62 shown in Figure 5, and the curvature has been flattened.
[0066] Furthermore, the extracted images 80 in this embodiment are extracted from the ophthalmic image 62 while maintaining the thickness of the layer structure. Therefore, as will be described in detail later, each extracted image 80 is reconstructed based on positional information, thereby appropriately reconstructing the layer structure in the original ophthalmic image 62 while maintaining its thickness.
[0067] The CPU 41 acquires position information indicating the position of the Nth extracted image 80 (i.e., the extracted image 80 extracted from the Nth partial image 70) (S8). Various types of information can be used for the position information. For example, coordinate information of at least one of the following may be acquired as the position information of the Nth extracted image 80: the position of the reference region RS in the Nth partial image 70, the position of the deep end DE of the extraction target range, and the position of the near end SE.
[0068] The CPU 41 stores the data of the extracted image 80 extracted from the Nth partial image 70, and the position information indicating the location of the Nth extracted image 80, in a storage means (e.g., NVM 44) (S9). The CPU 41 stores the data of the extracted image 80 in the storage means either without compression, or after lossless compression that allows the image, including the layered structure, to be reproducible after decoding. Next, the CPU 41 determines whether processing of all partial images 70 in the ophthalmic image 62 has been completed (S10). If processing of some of the partial images 70 has not yet been completed (S10: NO), "1" is added to the counter N indicating the order of the partial images 70 (S11), and the process returns to S6, and the processes of S6-10 are repeated. Once processing of all partial images 70 is complete, the process of saving the extracted image 80 and position information is terminated.
[0069] Referring to Figure 8, the image reconstruction process, which reconstructs the image based on the saved extracted image 80 and location information, will be described. The CPU 41 of the ophthalmic image processing device 40 executes the image reconstruction process according to the ophthalmic image processing program stored in the NVM 44.
[0070] When the user specifies the data to be reconstructed, the CPU 41 retrieves the data specified by the user (data for multiple extracted images 80 and the position information of each extracted image 80) from among the multiple data stored in the storage means. The CPU 41 generates reconstructed image data 90 in which the images of the extracted regions are reconstructed by arranging the data of the multiple extracted images 80 based on their respective position information. As shown in Figure 8, in the reconstructed image data 90, the regions extracted from the ophthalmic images 61 and 62 are appropriately reconstructed.
[0071] Furthermore, in the reconstructed image data 90, information from areas that were not extracted as the extracted image 80 is missing. The CPU 41 can also generate background-filled image data 91 by supplementing the pixel information of the background areas where information is missing in the reconstructed image data 90. As shown in Figure 8, the background-filled image data 91 is an image that is even closer to the actually captured ophthalmic images 61 and 62 because black pixel information is supplemented as the background.
[0072] The technology disclosed in the above embodiments is merely an example. Therefore, it is possible to modify the technology exemplified in the above embodiments. First, it is possible to omit some of the processes exemplified in the above embodiments. For example, the process of pre-extracting the image region (S2) may be omitted. Also, the process of supplementing missing parts of the result of identifying the reference region RS (SS4) can be omitted.
[0073] In the above embodiment, the outermost layer or boundary among the layered structures visible in the ophthalmic image 62 is designated as the reference site RS. However, it is possible to change the reference site. For example, if the tissue visible in the ophthalmic image 62 is retinal tissue of the fundus, at least one of the layers or boundaries located deeper than the outermost retinal surface (e.g., ganglion cell layer, inner plexiform layer, inner granular layer, outer plexiform layer, outer limiting membrane, intracellular and external segmental junction of photoreceptor cells, retinal pigment epithelium, Bruch's membrane, choroid, etc., and their boundaries) may be designated as the reference site. For example, by designating a layer or structure that is more clearly visible among the layered structures visible in the ophthalmic image 62 as the reference site, the accuracy of identifying the reference site may be improved, and the image may be extracted and saved more appropriately.
[0074] Furthermore, the layers and boundaries identified as reference sites among the layered structures visible in the ophthalmic image 62 may be changed according to various conditions. For example, there are layers and boundaries whose visibility in ophthalmic images changes depending on the location of the tissue. Therefore, the layers and boundaries identified as reference sites may be changed depending on the location of the tissue visible in the ophthalmic image.
[0075] Furthermore, the CPU 41 may identify several of the multiple layers and boundaries visible in the ophthalmic image 62 as reference areas. In this case, the CPU 41 may extract images from each of the multiple partial images, using the area enclosed by one of the identified reference areas (for example, the outermost layer or boundary) and other reference areas (for example, layers or boundaries located on the deeper side of the layered structure) as the extraction target area. As a result, image data within a specific range of the layered structure visible in the ophthalmic image 62 is appropriately extracted and saved.
[0076] Furthermore, it is possible to use a reference area other than layers and boundaries among the multiple areas visible in the ophthalmic image. For example, a diseased area visible in the ophthalmic image may be identified as a reference area, and extracted images may be extracted based on this identified reference area. In this case, the extracted images will be appropriately extracted and saved according to the diseased area. Alternatively, vascular areas, etc., visible in the ophthalmic image may be identified as reference areas.
[0077] Note that the process of acquiring an ophthalmic image in S1 of Figure 4 is an example of the "image acquisition step". The process of identifying the reference region RS in S3 of Figure 4 is an example of the "reference region identification step". The processes of extracting image data within the target range from the partial image 70 in S6 and S7 of Figure 4 are examples of the "extraction step". The process of acquiring the location information of the extracted image 80 in S8 of Figure 4 is an example of the "location information acquisition step". The process of saving the data and location information of the extracted image 80 in S9 of Figure 4 is an example of the "save step". The process of supplementing the identification result of the reference region RS in S4 of Figure 4 is an example of the "supplementation step". The process of reconstructing the image shown in Figure 8 is an example of the "image reconstruction step".
[0078] 1. Ophthalmic imaging device 40. Ophthalmic Image Processing Equipment 41 CPU 44 NVM 61,62 Ophthalmic images 70 Partial Images 80 extracted images 90 Reconstructed image data 91 Background interpolation image data RS reference site
Claims
1. An ophthalmic image processing device that processes ophthalmic image data, which is a tomographic image of the tissue of the eye being examined, The control unit of the ophthalmic image processing device is An image acquisition step in which ophthalmic images taken by an ophthalmic imaging device are acquired, An extraction step of extracting data from images within a target range for data storage from each of a plurality of partial images arranged in a predetermined direction within the ophthalmic image, A position information acquisition step, which acquires position information that indicates the position of each of the multiple extracted images extracted from each of the multiple partial images in the extraction step, and is information for reconstructing the region extracted in the extraction step from the ophthalmic image by arranging the data of the multiple extracted images, A storage step in which, among the multiple partial images, data of images other than those from which the multiple extracted images were extracted is excluded from the data to be saved, and the data of the multiple extracted images and the positional information of each of the multiple extracted images are saved to a storage means, An image reconstruction step in which the region extracted in the extraction step is reconstructed from the ophthalmic image by arranging the data of the plurality of extracted images stored in the storage means based on the respective position information, The image reconstruction step includes a background interpolation step in which pixel information of the background region where information is missing is added to the reconstructed image, An ophthalmic image processing apparatus characterized by performing the following:
2. An ophthalmic image processing apparatus according to claim 1, The control unit, A reference site identification step is further performed to identify at least one of the multiple sites within the tissue visible in the ophthalmic image as a reference site. The ophthalmic image processing apparatus is characterized in that, in the extraction step, it extracts image data within a target range based on the reference region from each of the plurality of partial images arranged in a predetermined direction within the ophthalmic image, as the extracted image.
3. An ophthalmic image processing apparatus according to claim 2, The aforementioned ophthalmic images are tomographic images capturing the layered structure of the tissue of the eye being examined, in which multiple layers are stacked. The ophthalmic image processing apparatus is characterized in that, in the reference area identification step, at least one of the multiple layers included in the layer structure and the boundaries of the layers is identified as the reference area.
4. An ophthalmic image processing apparatus according to claim 2 or 3, An ophthalmic image processing apparatus characterized in that, if, as a result of performing the reference area identification step, there are areas where the identification result of the reference area is missing, a further supplementation step is performed to supplement the identification result of the reference area in the missing areas based on the identification result of the reference area in other areas.
5. A fundus image processing apparatus according to any one of claims 1 to 4, The aforementioned ophthalmic images are tomographic images capturing the layered structure of the tissue of the eye being examined, in which multiple layers are stacked. The fundus image processing apparatus is characterized in that, when the plurality of extracted images extracted in the extraction step are arranged in the direction of the arrangement of the plurality of partial images without considering the positional information, the resulting image is one in which at least a portion of the region other than the layer structure is removed from the ophthalmic image and the curvature of the layer structure is flattened.
6. A fundus image processing apparatus according to any one of claims 1 to 5, The aforementioned ophthalmic images are tomographic images capturing the layered structure of the tissue of the eye being examined, in which multiple layers are stacked. The fundus image processing apparatus is characterized in that the extracted image is extracted from the ophthalmic image and stored while maintaining the thickness of the layer structure in the ophthalmic image.
7. An ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic image data, which is a tomographic image of the tissue of the eye being examined, The ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device, An image acquisition step in which ophthalmic images taken by an ophthalmic imaging device are acquired, An extraction step of extracting data from images within a target range for data storage from each of a plurality of partial images arranged in a predetermined direction within the ophthalmic image, A position information acquisition step, which acquires position information that indicates the position of each of the multiple extracted images extracted from each of the multiple partial images in the extraction step, and is information for reconstructing the region extracted in the extraction step from the ophthalmic image by arranging the data of the multiple extracted images, A storage step in which, among the multiple partial images, data of images other than those from which the multiple extracted images were extracted is excluded from the data to be saved, and the data of the multiple extracted images and the positional information of each of the multiple extracted images are saved to a storage means, An image reconstruction step in which the region extracted in the extraction step is reconstructed from the ophthalmic image by arranging the data of the plurality of extracted images stored in the storage means based on the respective position information, The image reconstruction step includes a background interpolation step in which pixel information of the background region where information is missing is added to the reconstructed image, An ophthalmic image processing program characterized by causing the ophthalmic image processing device to execute the following.