Ophthalmic image processing apparatus and ophthalmic image processing program

Abstract

To cause a user to determine an abnormality in a structure of a tissue in an ophthalmologic image more appropriately.SOLUTION: A control CPU 23 of an ophthalmologic image processing device 21 acquires an ophthalmologic image including tomographic images of a plurality of tomographic surfaces in an eye to be examined, and inputs the ophthalmologic image to a mathematical model trained by a machine learning algorithm so as to acquire a probability distribution for identifying two or more tissues included in a plurality of tissues in the tomographic images, generate structure abnormality degree maps 51A-51F indicating two-dimensional distribution of structure abnormality degrees in the tissues for every two or more tissues on the basis of the probability distribution, and display two or more structure abnormality degree maps 51A-51F generated for every two or more tissues simultaneously on a display device 28 in an aligned manner.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to an ophthalmic image processing apparatus that processes ophthalmic images of an eye to be examined, and an ophthalmic image processing program executed in the ophthalmic image processing apparatus.

Background Art

[0002] Conventionally, various techniques have been proposed for estimating abnormalities such as the structure of an object shown in an image.

[0003] For example, in Patent Document 1 by the inventor of the present application, a method using a learned model that obtains a probability distribution for identifying tissues in an ophthalmic image by inputting the ophthalmic image has been proposed. According to Patent Document 1, a quantitative degree of structural abnormality is obtained based on the probability distribution output from the learned model.

[0004] In addition, Patent Document 1 also describes generating a structural abnormality degree map showing a two-dimensional distribution of the degree of structural abnormality in a tissue. In Patent Document 1, for a tissue including a plurality of layers and boundaries, it is described that a structural abnormality degree map for the entire tissue or for any specific layer or boundary is generated.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, from a single structural abnormality degree map for the entire tissue or for any specific tissue, although the presence or absence of an abnormality can be grasped, it has been difficult to grasp the whole picture of the structural abnormality.

[0007] A typical object of the present disclosure is to provide an ophthalmic image processing apparatus and an ophthalmic image processing program that can more appropriately allow a user to determine an abnormality in the structure of a tissue shown in an ophthalmic image.

Means for Solving the Problem

[0008] An ophthalmic image processing apparatus provided by a typical embodiment in the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image of an eye to be examined. The control unit of the ophthalmic image processing apparatus acquires an ophthalmic image including tomographic images of a plurality of tomographic planes in the fundus of the eye to be examined, and inputs the ophthalmic image into a mathematical model trained by a machine learning algorithm, thereby obtaining a probability distribution for identifying two or more layers or layer boundaries of the fundus included in a plurality of tissues in the tomographic image. A structure abnormality degree map representing a two-dimensional distribution of the structure abnormality degree at the layer or layer boundary is generated for each of two or more layers or layer boundaries according to the correspondence between the gradation value of each pixel in the structure abnormality degree map and the structure abnormality degree being different for each layer or layer boundary, and two or more of the structure abnormality degree maps generated for each of two or more layers or layer boundaries are simultaneously arranged and displayed on a display device. in the world According to the probability distribution, for each of two or more layers or layer boundaries, and two or more of the structure abnormality degree maps generated for each of two or more layers or layer boundaries are simultaneously arranged and displayed on a display device.

[0009] An ophthalmic image processing program provided by a typical embodiment in the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image of an eye to be examined. When the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing apparatus, an image acquisition step of acquiring an ophthalmic image including tomographic images of a plurality of tomographic planes in the fundus of the eye to be examined as an ophthalmic image captured by an ophthalmic image capturing apparatus, an acquisition step of obtaining a probability distribution for identifying two or more layers or layer boundaries of the fundus included in a plurality of tissues in the tomographic image by inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm, and a structure abnormality degree map representing a two-dimensional distribution of the structure abnormality degree at the layer or layer boundary, where the correspondence between the gradation value of each pixel in the structure abnormality degree map and the structure abnormality degree is different for each layer or layer boundary. in the worldAs different ones according to the above, a structural abnormality degree map generation step of generating for each of two or more layers or layer boundaries based on the probability distribution, and a display step of simultaneously arranging and displaying on a display device two or more of the structural abnormality degree maps generated for each of two or more layers or layer boundaries are executed by the ophthalmic image processing device.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to more appropriately allow a user to determine an abnormality in the structure of a tissue shown in an ophthalmic image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0012] <Summary> Hereinafter, exemplary embodiments of the present disclosure will be described. In this embodiment, mainly, a method for generating and displaying a structural abnormality map from an ophthalmic image will be described.

[0013] In this embodiment, an ophthalmic image is processed by an ophthalmic image processing apparatus. The control unit of the ophthalmic image processing apparatus acquires an ophthalmic image including tomographic images of a plurality of tomographic planes in the eye to be examined. The control unit inputs the ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying two or more tissues included in a plurality of tissues in the tomographic image. The control unit generates a structural abnormality map representing a two-dimensional distribution of the degree of structural abnormality in the tissue for each of two or more tissues based on the probability distribution. The control unit simultaneously arranges and displays two or more structural abnormality maps generated for each of two or more tissues on a display device.

[0014] On each structural abnormality map, a site where a structural abnormality has occurred in each tissue is visualized. The user can accurately grasp the degree of structural abnormality at each position within the two-dimensional region by means of the structural abnormality map. Further, from two or more structural abnormality maps generated for each of two or more tissues simultaneously arranged and displayed on the display device, the distribution and spread of the structural abnormality in the depth direction can be visually understood. Therefore, the simultaneous display of two or more structural abnormality maps generated for each of two or more tissues is useful for the user to quickly grasp the overall situation of the structural abnormality.

[0015] Note that the plurality of tissues identified by the mathematical model may be located at different positions with respect to the depth direction (z - direction) of the eye to be examined. As an example, among the plurality of layers and layer boundaries in the fundus, two or more layers or boundaries may be identified by the mathematical model.

[0016] In this embodiment, an ophthalmic image including tomographic images of a plurality of tomographic planes in the eye to be examined is acquired and further processed by a control unit. The positions of the plurality of tomographic planes in the eye to be examined are different from each other. In the following description, unless otherwise specified, the tomographic plane is a plane extending in the XZ direction. However, it is not necessarily limited to this. Each tomographic image may be, for example, an OCT image taken by an OCT device. At this time, the OCT image may be a two - dimensional OCT image or a three - dimensional OCT image. Also, the tomographic image may be a motion contrast image (hereinafter referred to as an MC image). The MC image is created from motion contrast data obtained by processing a plurality of OCT data acquired from the same position at different times. The tomographic image does not necessarily have to be limited to an OCT image, and may be taken by a device other than an OCT device (for example, a shine - proof camera, etc.).

[0017] The mathematical model may be trained using a training data set in which the input side is data of tomographic images of tissues of the eye to be examined taken in the past, and the output side is data indicating the tissues in the tomographic images on the input side. In this case, the trained mathematical model can appropriately output a probability distribution for identifying tissues by inputting a tomographic image.

[0018] Note that the specific form of the mathematical model that outputs the probability distribution can be appropriately selected. For example, the mathematical model may output a probability distribution having as a probability variable the coordinates where at least one of a specific boundary of the tissue and a specific site exists within the region of the input tomographic image. In this case, based on the probability distribution output by the mathematical model, at least one of the boundary and the specific site in the tomographic image is appropriately and directly identified. Note that in this case, the "specific boundary" for which the probability distribution is to be obtained may be one boundary or a plurality of boundaries. When obtaining the probability distributions of a plurality of boundaries, the control unit may obtain the probability distributions separately for each of the plurality of boundaries. Similarly, the number of "specific sites" for which the probability distribution is to be obtained may be one or a plurality. Also, the region in the ophthalmic image that is the unit for obtaining the probability distribution may be any of a one-dimensional region, a two-dimensional region, and a three-dimensional region. The dimension of the coordinates serving as the probability variable may coincide with the dimension of the region that is the unit for obtaining the probability distribution.

[0019] Based on the obtained probability distribution, structure information indicating the degree of abnormality of the structure in the tissue is obtained. As a two-dimensional map of the structure information, a structure abnormality degree map representing the two-dimensional distribution of the degree of abnormality of the structure in the tissue may be generated.

[0020] Here, the structural information may be, for example, the degree of deviation of the acquired probability distribution with respect to the probability distribution when the tissue is accurately identified. The degree of deviation may include the entropy (average amount of information) of the acquired probability distribution. Entropy represents the degree of uncertainty, randomness, and disorder. In the present disclosure, the entropy of the probability distribution output when the tissue is accurately identified is 0. Also, as the degree of abnormality of the structure in the tissue increases and it becomes more difficult to identify the tissue, the entropy increases. Therefore, by using the entropy of the probability distribution as the degree of deviation, the degree of abnormality of the structure in the tissue can be more appropriately quantified. However, values other than entropy may be adopted as the degree of deviation. For example, at least any one of the standard deviation, coefficient of variation, variance, etc., which indicate the degree of dispersion of the acquired probability distribution, may be used as the degree of deviation. KL divergence, which is a measure of the difference between probability distributions, may be used as the degree of deviation. Also, the maximum value of the acquired probability distribution may be used as the degree of deviation.

[0021] Such a degree of deviation may be calculated by the control unit based on the probability distribution output by the mathematical model. Also, the conversion from the probability distribution to the degree of deviation may be performed within the mathematical model, and thereby, the degree of deviation may be obtained as the output from the mathematical model. Furthermore, the generation of the structural abnormality degree map may also be performed within the mathematical model, and the structural abnormality degree map may be obtained as the output from the mathematical model.

[0022] <Modification example of the structural abnormality degree map: Difference map> In the fundus, it is known that the fovea and the optic nerve head, etc., have significantly different structures from other regions in the fundus. Therefore, the degree of structural deviation tends to be higher in the fovea and the optic nerve head compared to other tissues even in a normal eye. Thus, when the two-dimensional distribution of the deviation degree is expressed as a structural abnormality map, even in a normal eye, the regions of the fovea and the optic nerve head can be depicted as regions with a higher degree of abnormality compared to other regions on the structural abnormality map of the fundus. In this case, the region with a high degree of structural abnormality corresponding to at least one of the fovea and the optic nerve head can be used as a landmark for the user to grasp the positional relationship on the structural abnormality map. On the other hand, even if an abnormality actually occurs in either the fovea or the optic nerve head, it is difficult for the user to grasp it from the structural abnormality map expressed by the two-dimensional distribution of the deviation degree. In contrast, the structural abnormality map may be a difference map between the two-dimensional distribution of the deviation degree in the eye to be examined and the two-dimensional distribution of the deviation degree in a normal eye. The two-dimensional distribution of the deviation degree in a normal eye may be created by collecting tomographic images of a plurality of normal eyes. In this case, compared to the structural abnormality map expressed as the two-dimensional distribution of the deviation degree, in the difference map, the degree of structural abnormality in tissues that inherently tend to have a high deviation degree, such as the fovea and the optic nerve head, and their surroundings, is considered to be more appropriately reflected on the map. In the fundus, in addition to the fovea and the optic nerve head, tissues that inherently tend to have a high deviation degree include blood vessels described later, and in the difference map, regardless of whether a structural abnormality actually occurs, the position of the blood vessel can also be reduced from being expressed as a place with a high degree of abnormality relative to the surroundings.

[0023] <Set the correspondence between the degree of structural abnormality and the gradation value for each tissue> Each pixel included in the structural abnormality map may be represented by a gradation value corresponding to the degree of structural abnormality. For example, in a tomographic image, blood vessels are depicted in a different manner from the surrounding tissues, so the degree of structural abnormality tends to be higher with respect to the surrounding tissues. Therefore, even in a normal eye, in tissues where blood vessels are densely distributed, regions where a larger degree of structural abnormality is output are more likely to be distributed at more positions compared to tissues with less blood vessel distribution. Thus, among the structural abnormality maps of each tissue, there may be a significant difference in the degree of noise that is expressed as a location with a high degree of abnormality with respect to the surroundings, regardless of whether an actual structural abnormality has occurred. When simultaneously displaying a plurality of structural abnormality maps with a significant difference in the degree of noise, for example, when the correspondence between the degree of structural abnormality and the gradation value in the structural abnormality map is the same among the structural abnormality maps for each tissue, there is a possibility that the user may be easily misled into thinking that the difference in the degree of noise between the structural abnormality maps represents the difference in the degree of abnormality between tissues.

[0024] In contrast, in the present embodiment, between two or more structural abnormality maps generated for each of two or more tissues, the correspondence between the degree of structural abnormality in the structural abnormality map and the gradation value of each pixel may be changeable for each tissue. Alternatively, the correspondence between the degree of structural abnormality in the structural abnormality map and the gradation value of each pixel may differ depending on the tissue. Here, the correspondence between the degree of structural abnormality and the gradation value of each pixel may be a gamma characteristic for converting from the degree of structural abnormality to the gradation value. The gamma characteristic can be represented, for example, as a gamma value. As an example, the correspondence between the degree of structural abnormality and the gradation value can be expressed using (Equation 1). The input is the degree of structural abnormality, and the output is the gradation value. γ is the gamma value.

[0025] output = input 1 / γ …(Equation 1)

[0026] Note that the input should be normalized in advance to a value in the range of 0 to 1. In this case, after the calculation of (Equation 1), integerization is appropriately performed so that the tone expression at the desired stage is obtained. For example, the output obtained by (Equation 1) may be multiplied by 255 to be imaged in 256 tones.

[0027] Here, for convenience, among two or more structural abnormality maps, the structural abnormality map in the tissue where relatively more blood vessels are distributed is referred to as the first structural abnormality map. Also, the structural abnormality map in the tissue where relatively fewer blood vessels are present with respect to the first structural abnormality map is referred to as the second structural abnormality map.

[0028] For example, if the gamma value in the first structural abnormality map is smaller than that in the second structural abnormality map, when the first structural abnormality map and the second structural abnormality map are compared, the difference in the degree of noise between the structural abnormality maps becomes less noticeable. Therefore, it becomes less likely to cause the above-mentioned misunderstanding. As a result, the structural abnormalities in each tissue can be appropriately and easily grasped by the user based on the structural abnormality map for each tissue.

[0029] For example, among the plurality of tissues included in the fundus of the eye, it is known that blood vessels are concentrated on the superficial side. Therefore, for example, appropriate gamma characteristics for each tissue (here, for each layer or each layer boundary) may be individually determined in advance.

[0030] Also, when it is desired to confirm a subtle structural abnormality in an arbitrary tissue, the gamma value of the structural abnormality map in the desired tissue may be increased based on the user's operation. Thereby, the subtle structural abnormality in the desired tissue is emphasized in the structural abnormality map, making it easier for the user to confirm.

[0031] In addition, the tissues in which characteristic structural abnormalities occur may vary depending on the type of disease. Therefore, the correspondence between the degree of structural abnormality and the tone value of each pixel for each tissue may be set according to, for example, the type of disease. The type of disease may be selectable as appropriate. For example, when the user manually inputs any one of the case name, disease name, etc. to the apparatus, the type of disease may be selected according to the input. Thereby, characteristic structural abnormalities for each disease can be easily confirmed by the user via the structural abnormality degree map. Note that it is not always necessary to use gamma correction when setting or changing the correspondence between the degree of structural abnormality and the tone value for each structural abnormality degree map. For example, at least one of the brightness and contrast of each structural abnormality degree map may be set or changed for each structural abnormality degree map. Further, processing such as histogram equalization may be performed on any one of the structural abnormality degree maps.

[0032] In addition, according to the noise level of an ophthalmic image (here, a tomographic image), the correspondence between the degree of structural abnormality and the tone value in each structural abnormality degree map may be adjusted. The degree of deviation may increase even when processing a tomographic image with poor image quality. Therefore, if the noise level of the ophthalmic image is high, a noisy structural abnormality degree map is likely to be output. Note that the noise level of the ophthalmic image may be an evaluation value for the tomographic image included in the ophthalmic image. In this case, for example, gamma correction may be automatically performed for each structural abnormality degree map according to the noise level of each tissue in the ophthalmic image. Further, for example, it may be an evaluation value for a frontal image generated for each tissue (for example, an OCT en-face image for each tissue). For example, as the evaluation value, the signal strength of the ophthalmic image or an index indicating the quality of the signal (for example, SSI (Signal Strength Index) or QI (Quality Index), etc.) can be used.

[0033] <Simultaneous display of the frontal image of the eye to be examined and the structural abnormality degree map> In this embodiment, the control unit of the ophthalmic image processing apparatus may further acquire a frontal image of the eye to be examined corresponding to the structural abnormality degree map. In other words, a frontal image of a part of the eye to be examined including the tissue indicated by the structural abnormality degree map may be acquired. The control unit may cause the acquired frontal image to be displayed on a display device together with the structural abnormality degree map. Thereby, it becomes easier for the user to grasp the position on the eye to be examined of the region with a high abnormality degree on the structural abnormality degree map. At this time, two or more structural abnormality degree maps corresponding to two or more tissues may be displayed simultaneously. A frontal image corresponding to at least one of the two or more structural abnormality degree maps may be displayed on the display device. Further, the frontal image may be displayed in parallel with the structural abnormality degree map or may be displayed in an overlapping manner. When overlapping the two, it may be possible to visually recognize both of them by making one of the frontal image and the structural abnormality degree map semi-transparent.

[0034] The frontal image can be various images. For example, the frontal image may be an OCT frontal image (specific examples include an en-face image, a C-scan image, etc.) generated from three-dimensional OCT data. Further, the frontal image may be an MC frontal image based on motion contrast data. Further, it may be a frontal image taken with a fundus camera, SLO, or the like. Note that two or more of the plurality of types of frontal images may be displayed simultaneously or may be switched and displayed.

[0035] The front image may also be an image showing the distribution of blood vessels. The image showing the distribution of blood vessels may be, for example, an MC front image. The image showing the distribution of blood vessels may also be a blood vessel density map showing a two-dimensional distribution of blood vessel density. For example, even if noise due to blood vessels is depicted on a structural abnormality map, the user can intuitively understand that the noise is caused by blood vessels through the MC front image displayed together. An MC front image of a corresponding tissue may be displayed for at least one of two or more structural abnormality maps generated for two or more tissues. This makes it easier to appropriately determine, for each tissue, whether a region with a high degree of abnormality is caused by blood vessels. Alternatively, or in addition to the MC front image for each tissue, an OCT front image for each tissue may be displayed in association with at least one of the structural abnormality maps. This makes it easier to appropriately determine whether a structural abnormality actually occurs in a region with a high degree of structural abnormality.

[0036] The front image may also be an image quality map showing the image quality of the tomographic image at each position on the structural abnormality map. The image quality map may be, for example, an SSI map that visualizes the signal intensity of each A-scan. As mentioned above, the worse the image quality, the greater the degree of abnormality. Therefore, by comparing the structural abnormality map with the image quality map, the user can easily and conveniently determine whether a region with a high degree of abnormality is due to low image quality.

[0037] <Simultaneous display of analysis image and structural abnormality map> Furthermore, instead of or in addition to the front image, an analysis image that graphically shows the analysis results for the ophthalmologic image (at least one of a tomographic image and a front image) may be displayed simultaneously with the structural abnormality map. The analysis image may show the analysis results for the thickness of the tissue. The analysis image and the front image may be displayed alternately at the same position on the screen. For at least one of two or more structural abnormality maps generated for two or more tissues, an analysis image for the corresponding tissue may be displayed.

[0038] The analysis image may be an analysis map of either a tomographic image or a frontal image, or may be an analysis chart. For example, as an analysis chart showing the thickness of tissue, a GCHART, an S / I chart, an ETDS chart, etc. may be used.

[0039] The structural abnormality degree map and the analysis image may be displayed in an overlapping manner. For example, when a thickness map is superimposed as the analysis image, either the structural abnormality degree or the thickness of the structure is expressed by contour lines, and the other is expressed by the tone of pixels, so that the structural abnormality degree and the thickness can be comprehensively confirmed, and it is easy for the user to accurately grasp the possibility of abnormality.

[0040] The above-mentioned plurality of structural abnormality degree maps may be displayed on a confirmation screen. The confirmation screen is displayed to allow the user to confirm the taken tomographic image. At this time, the control unit may cause the display device to display the tomographic image with the highest structural abnormality degree or the tomographic image with a structural abnormality degree equal to or higher than the threshold among the plurality of tomographic images of the same subject eye tissue. For example, by displaying the tomographic image with a high structural abnormality degree among the plurality of tomographic images on the confirmation screen, the tomographic image of the site with a high structural abnormality degree can be easily confirmed by the user. In addition, when the control unit activates a viewer for allowing the user to confirm the taken tomographic image, the tomographic image of the site with a high structural abnormality degree may be displayed first for the user to confirm among the plurality of tomographic images.

[0041] The control unit may execute a process of outputting a shooting instruction to shoot a site on the structural abnormality degree map where the structural abnormality degree is equal to or higher than the threshold to the ophthalmic imaging device. In addition, the control unit may execute a process of causing the display device to display a tomographic image or a magnified image of a site where the structural abnormality degree is equal to or higher than the threshold. In this case, the image of the site with a high structural abnormality degree is appropriately confirmed by the user.

[0042] When outputting an instruction to photograph a region where the degree of structural abnormality is equal to or greater than a threshold, the control unit may also output an instruction to photograph a tomographic image of the region where the degree of structural abnormality is equal to or greater than a threshold with higher image quality. For example, the control unit may output an instruction to acquire a tomographic image with higher resolution. Alternatively, the control unit may output an instruction to photograph a tomographic image of the region where the degree of structural abnormality is equal to or greater than a threshold multiple times and acquire an arithmetic average image of the photographed multiple tomographic images. In this case, a tomographic image of the region where the degree of structural abnormality is high is acquired with high image quality.

[0043] The control unit may input the structural abnormality degree map to a mathematical model that outputs an automatic diagnosis result regarding a disease of the subject's eye. In this case, the automatic diagnosis result may be output as a type of disease. In this case, it is believed that results that focus more on structural abnormalities and that can be obtained more efficiently than when searching for or identifying diseases using tomographic images are obtained. "Example" (Device configuration) A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, this embodiment uses a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image capturing devices 11A and 11B. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The constructed mathematical model outputs a probability distribution for identifying tissue in an ophthalmic image based on an input ophthalmic image. The ophthalmic image processing device 21 acquires the probability distribution using the mathematical model and acquires the degree of deviation between the acquired probability distribution and the probability distribution when the tissue is accurately identified as structural information indicating the degree of abnormality in the tissue structure. The ophthalmic image capturing devices 11A and 11B capture ophthalmic images, which are images of the tissue of the subject's eye.

[0044] As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of the present embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using the ophthalmic image (hereinafter referred to as "training ophthalmic image") acquired from the ophthalmic imaging device 11A and the training data indicating the position of at least any one of the tissues in the training ophthalmic image. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic imaging device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic imaging device 11A) may cooperate to construct a mathematical model.

[0045] Also, a PC is used as the ophthalmic image processing device 21 of the present embodiment. However, the device that can function as the ophthalmic image processing device 21 is not limited to a PC. For example, the ophthalmic imaging device 11B or a server or the like may function as the ophthalmic image processing device 21. When the ophthalmic imaging device (OCT device in the present embodiment) 11B functions as the ophthalmic image processing device 21, the ophthalmic imaging device 11B can acquire the degree of deviation from the captured ophthalmic image while capturing the ophthalmic image. Further, the ophthalmic imaging device 11B can also capture an appropriate site based on the acquired degree of deviation. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmic image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmic imaging device 11B) may cooperate to perform various processes.

[0046] Further, in the present embodiment, the case where a CPU is used as an example of the controller that performs various processes is illustrated. However, it goes without saying that a controller other than the CPU may be used for at least a part of the various devices. For example, by adopting a GPU as the controller, the processing speed may be increased.

[0047] A mathematical model construction device 1 will be described. The mathematical model construction device 1 is arranged, for example, in an ophthalmic image processing device 21 or a manufacturer or the like that provides an ophthalmic image processing program to a user. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller for controlling and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 2) described later. Further, the communication I / F 5 connects the mathematical model construction device 1 to other devices (for example, an ophthalmic image capturing device 11A and an ophthalmic image processing device 21, etc.).

[0048] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the mathematical model construction device 1. At least any one of, for example, a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 7. Note that, together with or instead of the operation unit 7, a microphone or the like for inputting various instructions may be used. The display device 8 displays various images. For the display device 8, at least any one of various devices capable of displaying images (for example, a display device, a display, a projector, etc.) can be used. Note that the "image" in the present disclosure includes both still images and moving images.

[0049] The mathematical model construction device 1 can acquire data of an ophthalmic image (hereinafter, may be simply referred to as "ophthalmic image") from the ophthalmic image capturing device 11A. The mathematical model construction device 1 may acquire data of an ophthalmic image from the ophthalmic image capturing device 11A by at least any one of, for example, wired communication, wireless communication, a removable storage medium (for example, a USB memory), etc.

[0050] The ophthalmic image processing device 21 will now be described. The ophthalmic image processing device 21 is installed, for example, in a facility (e.g., a hospital or health checkup facility) where a diagnosis or examination is performed on a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller responsible for control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see FIG. 5 ), which will be described later. The ophthalmic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (e.g., the ophthalmic image capturing device 11B and the mathematical model construction device 1).

[0051] The ophthalmologic image processing device 21 is connected to an operation unit 27 and a display device 28. As with the operation unit 7 and display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0052] The ophthalmological image processing device 21 can acquire ophthalmological images from the ophthalmological image capturing device 11B. The ophthalmological image processing device 21 may acquire ophthalmological images from the ophthalmological image capturing device 11B by at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), etc. The ophthalmological image processing device 21 may also acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

[0053] The ophthalmic image capturing devices 11A and 11B will be described. As an example, in this embodiment, a case will be described in which an ophthalmic image capturing device 11A that provides ophthalmic images to the mathematical model construction device 1 and an ophthalmic image capturing device 11B that provides ophthalmic images to the ophthalmic image processing device 21 are used. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from multiple ophthalmic image capturing devices. Alternatively, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a single common ophthalmic image capturing device. Note that the two ophthalmic image capturing devices 11A and 11B illustrated in this embodiment have the same configuration. Therefore, the two ophthalmic image capturing devices 11A and 11B will be described together below.

[0054] In this embodiment, an OCT device is exemplified as the ophthalmologic imaging device 11 (11A, 11B).

[0055] The ophthalmologic imaging device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic imaging unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that manages control, and a storage device 14 (14A, 14B) that can store programs, data, etc.

[0056] The ophthalmologic image capturing unit 16 includes various components necessary for capturing ophthalmologic images of the subject's eye. The ophthalmologic image capturing unit 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit that scans the measurement light, an optical system that irradiates the subject's eye with the measurement light, and a light receiving element that receives a composite light of the light reflected by the tissue of the subject's eye and the reference light.

[0057] The ophthalmologic imaging device 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the subject's eye. Specifically, the CPU 13 scans OCT light (measurement light) along a scan line to capture two-dimensional tomographic images of a cross section intersecting the scan line. The two-dimensional tomographic image may be an averaged image generated by performing an averaging process on multiple tomographic images of the same area. The CPU 13 can also capture three-dimensional tomographic images of tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires multiple two-dimensional tomographic images by scanning the measurement light along multiple scan lines at different positions within a two-dimensional area of the tissue when viewed from the front. Then, the CPU 13 combines the captured two-dimensional tomographic images to acquire a three-dimensional tomographic image. (Mathematical model construction process) 2 to 4, the mathematical model construction process executed by the mathematical model construction device 1 will be described. The mathematical model construction process is executed by the CPU 3 in accordance with a mathematical model construction program stored in the storage device 4. In the mathematical model construction process, the mathematical model is trained using a training data set, thereby constructing a mathematical model that outputs a probability distribution for identifying tissues in an ophthalmic image. The training data set includes input-side data (input training data) and output-side data (output training data).

[0058] 2, the CPU 3 acquires data of training ophthalmic images, which are ophthalmic images captured by the ophthalmic image capturing device 11A, as input training data (S1). In this embodiment, the data of the training ophthalmic images is generated by the ophthalmic image capturing device 11A and then acquired by the mathematical model construction device 1. However, the CPU 3 may acquire data of the training ophthalmic images by acquiring signals (e.g., OCT signals) that are the basis for generating the training ophthalmic images from the ophthalmic image capturing device 11A and generating the training ophthalmic images based on the acquired signals.

[0059] In step S1 of the present embodiment, a two-dimensional tomographic image captured by the ophthalmic imaging device 11A, which is an OCT device, is acquired as a training ophthalmic image. FIG. 3 shows an example of a training ophthalmic image 30 that is a two-dimensional tomographic image of the fundus. In the training ophthalmic image 30 illustrated in FIG. 3, a plurality of layers in the fundus are shown. In the present embodiment, the training ophthalmic images included in the training dataset may all be ophthalmic images of tissues with a low degree of structural abnormality, but are not necessarily limited to this. For example, ophthalmic images with a high degree of structural abnormality for some tissues may be included in the training dataset. Diseased eyes are not composed only of tissues with abnormal structures, but also contain many normal structures. Therefore, even if there are ophthalmic images with a high degree of structural abnormality for some tissues in the training dataset, it is considered that in the training dataset, the data for normal structures is much more abundant than the data for abnormal structures, so it is difficult for the identification of tissues using a mathematical model to be adversely affected. Also, it is considered that the accuracy of identification can be improved by appropriately having data for abnormal structures in the training dataset.

[0060] Next, the CPU 3 acquires training data indicating the position of at least any one of the tissues in the training ophthalmic image (S2). FIG. 4 shows an example of training data 31 when a two-dimensional tomographic image of the fundus is used as the training ophthalmic image 30. The training data 31 illustrated in FIG. 4 includes data of labels 32A to 32F indicating the positions of each of six boundaries among a plurality of tissues (specifically, a plurality of layers and boundaries) shown in the training ophthalmic image 30. In the present embodiment, the data of the labels 32A to 32F in the training data 31 is generated by an operator operating the operation unit 7 while viewing the boundaries in the training ophthalmic image 30. However, it is also possible to change the method of generating the label data.

[0061] It is also possible to change the training data. For example, when a two-dimensional cross-sectional image of the fundus is used as the training ophthalmic image 30, the training data may be data indicating the position of at least any one layer in the fundus. Further, the training data may be data indicating the position of a punctate site or the like in the tissue, rather than layers and boundaries.

[0062] Next, the CPU 3 executes training of a mathematical model using the training data set by a machine learning algorithm (S3). As the machine learning algorithm, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

[0063] A neural network is a method that mimics the behavior of a biological neural cell network. Examples of neural networks include feedforward (forward propagation type) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0064] Random forest is a method of learning based on randomly sampled training data to generate a large number of decision trees. When using a random forest, follow the branches of a plurality of decision trees that have been learned in advance as discriminators, and take the average (or majority vote) of the results obtained from each decision tree.

[0065] Boosting is a technique for generating a strong discriminator by combining a plurality of weak discriminators. A strong discriminator is constructed by sequentially learning simple and weak discriminators.

[0066] SVM is a method for constructing a two-class pattern classifier using linear input elements. SVM learns the parameters of the linear input elements based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

[0067] A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by training using a training dataset. As described above, a training dataset is a set of input training data and output training data. For example, the correlation data (e.g., weights) between each input and output is updated through training.

[0068] In this embodiment, a multi-layer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. In particular, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used.

[0069] As an example, the mathematical model constructed in this embodiment outputs, as a probability distribution for identifying tissue, a probability distribution in which the random variables are coordinates (one-dimensional coordinates, two-dimensional coordinates, three-dimensional coordinates, or four-dimensional coordinates) of a specific tissue (e.g., a specific boundary, a specific layer, or a specific region) within a region in an ophthalmic image (one-dimensional region, two-dimensional region, three-dimensional region, or four-dimensional region including a time axis). In this embodiment, a softmax function is applied to have the mathematical model output the probability distribution. In detail, the mathematical model constructed in S3 outputs a probability distribution in which the random variables are coordinates of a specific boundary within a one-dimensional region extending in a direction intersecting the specific boundary in the two-dimensional tomographic image (in this embodiment, the A-scan direction of the OCT).

[0070] However, the specific method by which the mathematical model outputs the probability distribution for identifying tissues can be modified as appropriate. For example, the mathematical model may output a probability distribution for identifying tissues, in which the two-dimensional or three-dimensional coordinates of a specific tissue (e.g., a characteristic site) are present in a two-dimensional or three-dimensional region as random variables. Furthermore, the mathematical model may output a probability distribution for each region (e.g., pixel) of the input ophthalmic image, in which the types of multiple tissues (e.g., multiple layers and boundaries) in the test eye are random variables. Furthermore, the ophthalmic image input to the mathematical model may be a moving image.

[0071] Other machine learning algorithms may also be used, such as generative adversarial networks (GANs) that use two competing neural networks.

[0072] The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S4: NO). When the construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. The program and data for realizing the constructed mathematical model are installed in the ophthalmologic image-processing device 21. (Ophthalmological Image Processing) 5 to 11, the ophthalmic image processing performed by the ophthalmic image processing device 21 will be described. The ophthalmic image processing is performed by the CPU 23 in accordance with an ophthalmic image processing program stored in the storage device 24.

[0073] First, the CPU 23 acquires a three-dimensional tomographic image of the tissue of the eye to be examined (the fundus in this embodiment) (S11). The three-dimensional tomographic image is taken by the ophthalmic imaging device 11B and acquired by the ophthalmic image processing device 21. As described above, the three-dimensional tomographic image is composed by combining a plurality of two-dimensional tomographic images taken by scanning measurement light on mutually different scan lines. Note that the CPU 23 may acquire a signal (for example, an OCT signal) serving as a basis for generating the three-dimensional tomographic image from the ophthalmic imaging device 11B and generate the three-dimensional tomographic image based on the acquired signal.

[0074] The CPU 23 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images constituting the acquired three-dimensional tomographic image (S12). FIG. 6 shows an example of the two-dimensional tomographic image 40. In the two-dimensional tomographic image 40, a plurality of boundaries in the fundus of the eye to be examined appear. In the example shown in FIG. 6, a plurality of boundaries including the boundary Bi which is the inner limiting membrane (ILM) and the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL) appear. Also, a plurality of one-dimensional regions A1 to AN are set in the two-dimensional tomographic image 40. In this embodiment, the one-dimensional regions A1 to AN set in the two-dimensional tomographic image 40 extend along an axis intersecting a specific boundary (in this embodiment, a plurality of boundaries including the boundary Bi and the boundary Bg). Specifically, the one-dimensional regions A1 to AN in this embodiment coincide with the regions of each of the plurality (N) of A-scans constituting the two-dimensional tomographic image 40 taken by the OCT device.

[0075] Note that it is also possible to change the method of setting the plurality of one-dimensional regions. For example, the CPU 23 may set the plurality of one-dimensional regions so that the angle between the axis of each one-dimensional region and a specific boundary approaches perpendicular as much as possible. In this case, the position and angle of each one-dimensional region may be set so that the angle approaches perpendicular based on the shape of a general tissue of the eye to be examined (the fundus in this embodiment), for example.

[0076] The CPU 23 inputs the Tth two-dimensional tomographic image into the mathematical model, thereby acquiring, as a probability distribution for identifying tissue, a probability distribution of coordinates where an Mth boundary (the initial value of M is "1") exists in each of the multiple one-dimensional regions A1 to AN (S14). FIGS. 7 and 8 show an example of a graph showing the probability distribution of coordinates where a boundary Bi exists, acquired from the one-dimensional coordinate A1. In the example shown in FIGS. 7 and 8, the one-dimensional coordinate of the one-dimensional region A1 is used as a random variable to show the probability distribution of coordinates where a boundary Bi exists. That is, in the example shown in FIGS. 7 and 8, the horizontal axis represents the random variable, and the vertical axis represents the probability of the random variable, which is the coordinate in the one-dimensional region A1 where a boundary Bi exists. In S14, a probability distribution in each of the multiple one-dimensional regions A1 to AN is acquired.

[0077] The probability distribution shown in FIG. 7 is an example of a probability distribution output when the degree of structural abnormality of the tissue (more specifically, the tissue near the boundary Bi) is low. At locations where the degree of structural abnormality is low, the mathematical model is likely to accurately identify the tissue, so the probability of the tissue location is likely to be biased. According to the graph shown in FIG. 7, it can be determined that, among the points on the one-dimensional region A1, the point at which the boundary Bi is most likely to exist is point P. The probability distribution when the mathematical model accurately identifies the tissue (i.e., an ideal probability distribution) takes the value 1 at only one point on the one-dimensional region A1, and is 0 at other points.

[0078] On the other hand, the probability distribution shown in Figure 8 is an example of a probability distribution that is output when the degree of abnormality in the tissue structure is high. As shown in Figure 8, the probability distribution is less likely to be biased at locations where the degree of structural abnormality is high. As described above, the bias in the probability distribution for identifying tissue changes depending on the degree of abnormality in the tissue structure.

[0079] Next, the CPU 23 acquires the degree of divergence of the probability distribution P regarding the M-th boundary (S15). The degree of divergence is the difference between the probability distribution P acquired in S14 and the probability distribution in the case where the organization is accurately identified. In the present embodiment, the degree of divergence is acquired as structure information indicating the degree of abnormality of the structure of the organization. In S15 of the present embodiment, the degree of divergence is acquired (calculated) for each of the plurality of probability distributions P acquired for the plurality of one-dimensional regions A1 to AN.

[0080] In the present embodiment, the entropy of the probability distribution P is calculated as the degree of divergence. The entropy is given by the following (Equation 2). The entropy H(P) takes a value of 0 ≤ H(P) ≤ log (number of events), and becomes a smaller value as the probability distribution P is more biased. That is, the lower the entropy H(P), the lower the degree of abnormality of the structure of the organization. The entropy of the probability distribution when the organization is accurately identified is 0. Also, as the degree of abnormality of the structure of the organization increases and it becomes more difficult to identify the organization, the entropy H(P) increases. Therefore, by using the entropy H(P) of the probability distribution P as the degree of divergence, the degree of abnormality of the structure of the organization is appropriately quantified.

[0081] H(P)=-Σplog(p)···(Equation 2) However, a value other than entropy may be adopted as the degree of divergence. For example, at least any one of the standard deviation, coefficient of variation, variance, etc., indicating the degree of dispersion of the acquired probability distribution P may be used as the degree of divergence. KL divergence, which is a measure for depicting the difference between probability distributions P, may be used as the degree of divergence. Also, the maximum value of the acquired probability distribution P (for example, the maximum value of the probability illustrated in FIGS. 7 and 8) may be used as the degree of divergence. Further, the difference between the maximum value of the acquired probability distribution P and the second largest value may be used as the degree of divergence.

[0082] Next, the CPU 23 determines whether or not the divergence degrees of all the boundaries to be detected in the T-th two-dimensional tomographic image have been acquired (S16). If the divergence degrees of some boundaries have not been acquired (S16: NO), "1" is added to the boundary order M (S17), and the process returns to S14, where the divergence degree of the next boundary is acquired (S14, S15). When the divergence degrees of all the boundaries have been acquired (S16: YES), the CPU 23 stores the divergence degree of the T-th two-dimensional tomographic image in the storage device 24 and also displays it on the display device 28 (S19). The CPU 23 acquires (generates in this embodiment) the structural abnormality degree graph of the T-th two-dimensional tomographic image and displays it on the display device 28 (S20).

[0083] With reference to FIGS. 9 and 10, the structural abnormality degree graph 52 will be described. FIG. 9 is an example of a display screen on which a two-dimensional tomographic image 51A with a low structural abnormality degree, a structural abnormality degree graph 52A related to the two-dimensional tomographic image 51A, and a divergence degree table 53A showing the divergence degree related to the two-dimensional tomographic image 51A are displayed. FIG. 10 is an example of a display screen on which a two-dimensional tomographic image 51B with a high structural abnormality degree, a structural abnormality degree graph 52B related to the two-dimensional tomographic image 51B, and a divergence degree table 53B showing the divergence degree related to the two-dimensional tomographic image 51B are displayed.

[0084] As shown in FIGS. 9 and 10, the two-dimensional tomographic image 51 is a two-dimensional image that extends in the X direction (the left-right direction of the drawing) and the Z direction (the up-down direction of the drawing). As described above, the divergence degree is acquired for each of a plurality of axes (in this embodiment, a plurality of A scans) that extend parallel to the Z direction on the ophthalmic image. In the structural abnormality degree graph 52 shown in FIGS. 9 and 10, the horizontal axis is the X axis, and the divergence degree at each position in the X direction is shown on the vertical axis.

[0085] As an example, the structural abnormality degree graph 52 of this embodiment shows the average value of multiple deviations (entropy in this embodiment) obtained for each of multiple boundaries for each position in the X direction. However, the deviation of one boundary may be shown by the structural abnormality degree graph 52. Also, the average value of specific multiple boundaries (for example, the IPL / INL boundary and the OPL / ONL boundary) may be shown by the structural abnormality degree graph 52. Also, various statistical values other than the average value (for example, the median, mode, maximum value, minimum value, etc.) may be used instead of the average value.

[0086] As shown in Fig. 9, when the degree of structural abnormality is low across the entire X direction, the deviation indicated by structural abnormality degree graph 52A will be a low value across the entire X direction. On the other hand, as shown in Fig. 10, at positions in the X direction where the degree of structural abnormality is high, the deviation indicated by structural abnormality degree graph 52B will be a high value. As described above, the structural abnormality degree graph 52 allows the user to appropriately grasp at which positions in the X direction the degree of abnormality is high.

[0087] Referring to FIGS. 9 and 10, an example of a method for displaying the degree of deviation will be described. As shown in FIGS. 9 and 10, in the deviation degree table 53 of the present embodiment, the obtained deviation degree (entropy in the present embodiment) is displayed for each of a plurality of boundaries. Therefore, the user can appropriately grasp the boundary with a high degree of structural abnormality based on the quantified value. The deviation degree displayed in the deviation degree table 53 of the present embodiment is the average value of a plurality of deviation degrees obtained for each of a plurality of one-dimensional regions (A-scan in the present embodiment). Also, in the deviation degree table 53 of the present embodiment, the average value of the deviation degrees for all boundaries is displayed. Therefore, the user can easily grasp whether there is a site with a high degree of structural abnormality in the tissue shown in the ophthalmic image by the average value. Further, in the deviation degree table 53 of the present embodiment, the average value of the deviation degrees for a plurality of specific boundaries among all boundaries is displayed. As an example, in the present embodiment, the average values of the boundaries (IPL / INL boundary and OPL / ONL boundary) where the structure is likely to collapse due to the influence of the disease are displayed. Therefore, the user can easily grasp whether there is a structural abnormality due to the disease. Note that, as described above, various statistical values other than the average value may be used.

[0088] Next, the CPU 23 determines whether the deviation degrees of all the two-dimensional tomographic images constituting the three-dimensional tomographic image have been obtained (S21). If the deviation degrees of some of the two-dimensional tomographic images have not been obtained (S21: NO), "1" is added to the order T of the two-dimensional tomographic images (S22), and the process returns to S12, and the deviation degree of the next two-dimensional tomographic image is obtained (S12 to S20). When the deviation degrees of all the two-dimensional tomographic images have been obtained (S21: YES), the CPU 23 acquires (generates in the present embodiment) the structural abnormality degree map and causes it to be displayed on the display device 28 (S24).

[0089] Referring to FIG. 11, the structural abnormality degree map will be described. The structural abnormality degree map shows the two-dimensional distribution of the divergence degree in the tissue. In this embodiment, in the structural abnormality degree map, the two-dimensional distribution of the divergence degree when the tissue (the fundus in this embodiment) is viewed from the front is shown. However, the direction showing the two-dimensional distribution may be changed as appropriate. The divergence degree has already been obtained for each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image, and thereby the overall divergence degree of the tissue has been obtained. The overall divergence degree of the tissue referred to here can also be rephrased as the divergence degree at the first to sixth layer boundaries corresponding to labels 32A to 32F (see FIG. 4) in this embodiment.

[0090] In this embodiment, as an example, structural abnormality degree maps 51A to 51F of the first to sixth layer boundaries respectively corresponding to labels 32A to 32F (see FIG. 4) are generated based on the divergence degree of each layer boundary. FIG. 11 shows an example of the display mode of the structural abnormality degree maps 51A to 51F on the display device 28.

[0091] The structural abnormality degree maps 51A to 51F may be graphs expressing the divergence degree at each position as colors or shades. At this time, the gradation value of each pixel in the structural abnormality degree maps 51A to 51F is converted from the divergence degree at each position of the layer boundary. For example, in the structural abnormality degree maps 51A to 51F shown in FIG. 11, the divergence degree is shown in grayscale. Here, in this embodiment, since the divergence degree (entropy in this embodiment) is calculated in the range of 0 to 1, each value in the range of 0 to 1 is converted into a gradation value in the range of 0 to 255, for example, and expressed on the map. In the structural abnormality degree maps 51A to 51F shown in FIG. 11, pixels with a higher divergence degree are expressed with smaller gradation values (that is, higher brightness). Note that the correspondence relationship between the divergence degree and the gradation value is the same among the structural abnormality degree maps 51A to 51F shown in FIG. 11. However, the specific method for showing the divergence degree at each position in the structural abnormality degree map is not limited to grayscale, and can be appropriately changed such as a color map or a three-dimensional map.

[0092] The structural abnormality maps 51A to 51F shown as an example in FIG. 11 are the processing results for the eye to be examined with detachment on the deep side. From the structural abnormality maps 51A to 51F, it can be seen that regions with a high degree of deviation appear in at least a plurality of structural abnormality maps 51B to 51F from the second layer boundary to the sixth layer boundary at the center of the plurality of maps. Therefore, the user can easily grasp, based on the plurality of structural abnormality maps 51A to 51F, the possibility of structural abnormalities affecting a plurality of layers. Also, as another example, if regions with a high degree of deviation appear only in a small number of the plurality of structural abnormality maps generated for each layer boundary, the user can be made to suspect the possibility of more local structural abnormalities compared with the examples. Thus, the display of the plurality of structural abnormality maps is useful for the user to quickly grasp the overall situation of structural abnormalities.

[0093] As shown in FIG. 11, in this embodiment, a frontal image 52 of the fundus is displayed simultaneously with the structural abnormality maps 51A to 51F. In this embodiment, the imaging range of the frontal image 52 corresponds to the structural abnormality maps 51A to 51F. The user can confirm, with reference to the frontal image 52, where on the fundus there is a location with a high degree of deviation on the structural abnormality maps 51A to 51F. The frontal image 52 in this embodiment may be, for example, an OCT frontal image. At this time, it may be an MC frontal image, which is a type of OCT frontal image. In the MC frontal image, blood vessels are depicted. Therefore, for example, it is easy for the user to grasp whether a region with a high degree of abnormality on the structural abnormality maps 51A to 51F is derived from blood vessels.

[0094] Also, when an OCT front image is displayed as the front image 52, an OCT front image regarding any boundary may be displayed as the front image 52. At this time, the OCT front images for each boundary may be selectable according to the user's operation. For example, by inputting a selection operation for any of the structural abnormality degree maps 51A to 51F, the OCT front image regarding the boundary corresponding to the selected map may be displayed as the front image 52. By simultaneously displaying the OCT front images of the layers corresponding to the desired structural abnormality degree maps, it is easier for the user to grasp the presence or absence of abnormalities at positions with a high degree of deviation in the structural abnormality degree map.

[0095] Also, in this embodiment, the correspondence relationship between the degree of deviation and the gradation value in each of the structural abnormality degree maps 51A to 51F can be individually changed for each map. In FIG. 11, as an example of a GUI widget for changing the correspondence relationship between the degree of deviation and the gradation value, sliders 53A to 53F are installed adjacent to each of the structural abnormality degree maps 51A to 51F. In each of the sliders 53A to 53F, the position of the knob can be changed based on an individual operation. In each of the structural abnormality degree maps 51A to 51F, the gamma value when converting the degree of deviation into a gradation value is changed according to the position of the knob. In this embodiment, the gamma value decreases as the position of the knob moves to the left, and the gamma value increases as the position of the knob moves to the right. By operating the sliders 53A to 53F, the sensitivity in the structural abnormality degree maps 51A to 51F can be changed retrospectively. That is, as the gamma value decreases, it becomes a low-sensitivity map in which positions with a higher degree of deviation on the map are emphasized. Conversely, as the gamma value increases, it becomes a higher-sensitivity map, and abnormalities in delicate structures can be made prominent on the map. Note that the initial position of the knob (that is, the initial value of the gamma value) may be constant or may be determined based on the noise level of the two-dimensional tomographic images constituting the three-dimensional tomographic image.

[0096] Here, for example, in the fundus of the eye, there are tissues (hereinafter referred to as specific tissues) such as the fovea, the optic disc, and blood vessels, which are inherently likely to have a high degree of divergence. In some of the structural abnormality degree maps 51A to 51F of the first to sixth layer boundaries shown in FIG. 11, the influence of specific tissues is prominently depicted. The description based on specific tissues may become noise in grasping abnormalities from the structural abnormality degree maps 51A to 51F. In such a case, in a map where the description based on specific tissues is prominent, by decreasing the gamma value to reduce the sensitivity, it may be possible to bring the description based on specific tissues closer to the background side while clarifying the positions where there may be abnormalities. Also, for example, when no prominent abnormal site is seen as in the structural abnormality degree map 51A of the first layer boundary shown in FIG. 11, by increasing the gamma value to increase the sensitivity, subtle structural abnormalities may emerge on the map.

[0097] In this embodiment, the correspondence relationship between the degree of divergence in each of the structural abnormality degree maps 51A to 51F of the first to sixth layer boundaries and the gradation value is changed for each map according to the type of disease selected by the user. In this case, the relationship between the type of disease and the gamma value (information indicating the correspondence relationship between the degree of divergence and the gradation value) in the structural abnormality degree map for each boundary may be stored in advance in the storage device 24 as a look-up table. In this embodiment, the type of disease is selected via the box 54. When the box 54 is selected, a pull-down menu listing the names of a plurality of diseases is expanded. By selecting any one of them based on an operation, the structural abnormality degree maps 51A to 51F in which the gamma value corresponding to the type of disease is reflected are displayed on the screen. For example, in RVO (retinal vein occlusion), since the overall structure of the retina is greatly disrupted, when RVO is selected, the sensitivity of each of the structural abnormality degree maps 51A to 51F may be reduced compared to the standard time. Thereby, it is considered that the whole picture of the structural abnormality due to RVO can be more easily grasped.

[0098] Also, when the disease type can be selected as described above, one or more of the structural abnormality degree maps 51A to 51F may be highlighted according to the selected disease type. For example, when CSC (central serous chorioretinopathy) is selected, the maps corresponding to IS / OS and RPE / BM, which are emphasized in the diagnosis of CSC, may be highlighted. By doing so, the amount of information that the user should check can be suitably suppressed. Note that various modes of highlighting are conceivable. As an example, it may be highlighted by changing the frame line of the map to be highlighted to a thick line.

[0099] In addition, the structural abnormality degree maps 51A to 51F based on the two-dimensional distribution of the deviation degree of the test eye, shown as an example in FIG. 11, have abnormalities near the fovea. In FIG. 11, on some maps, the range with a high deviation degree near the fovea is clearly wider than the size of the fovea, so it is possible to easily suspect an abnormality. However, if the abnormal range is narrower than the example in FIG. 11 and overlaps with the fovea, since the fovea is a tissue that is inherently likely to have a high deviation degree, it is conceivable that it may be difficult for the user to grasp the structural abnormality from the structural abnormality degree map based on the two-dimensional distribution of the deviation degree. In contrast, instead of the structural abnormality degree map based on the two-dimensional distribution of the deviation degree, a difference map between the two-dimensional distribution of the deviation degree of the test eye and the two-dimensional distribution of the deviation degree in a normal eye may be displayed. The two-dimensional distribution of the deviation degree in a normal eye may be created by collecting three-dimensional OCT data of a plurality of normal eyes. The difference map may be generated for each boundary. In the difference map, when the tissue that is inherently likely to have a high deviation degree overlaps with the abnormality, there is a possibility that the structural abnormality can be more accurately depicted.

[0100] Also, when there is a part (hereinafter referred to as "abnormal part") with a divergence degree equal to or higher than the threshold value among the tissues shown in the ophthalmic image (three-dimensional tomographic image in this embodiment), the CPU 23 causes the display device 28 to display a tomographic image (at least one of a two-dimensional tomographic image and a three-dimensional tomographic image) or an enlarged image of the abnormal part. In FIG. 11, a tomographic image 55 is shown. Specifically, the CPU 23 of this embodiment causes the display device 28 to display the tomographic image with the highest divergence degree or the tomographic image with a divergence degree equal to or higher than the threshold value among the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image. In this embodiment, as shown in FIG. 11, the tomographic image 55 is displayed together with the structural abnormality degree maps 51A to 51F. Thereby, the user can immediately confirm the actual structure of the region with a high divergence degree in the structural abnormality degree maps 51A to 51F. Further, in this embodiment, a graphic 56 indicating the acquisition position of the tomographic image 55 is displayed on either the structural abnormality degree maps 51A to 51F or the front image 52. Thereby, the user can easily grasp the location where there may be a structural abnormality.

[0101] Note that the tomographic image displayed simultaneously with the structural abnormality degree maps 51A to 51F may be changeable. For example, a tomographic image at an arbitrary acquisition position may be displayed. Also, a tomographic image with a structural abnormality degree equal to or lower than the threshold value (for example, a tomographic image acquired on the line indicated by reference numeral 57 in FIG. 11) may be displayed together with or in place of the tomographic image 55.

[0102] Also, when a layer or a layer boundary is identified, each layer or layer boundary identified in the tomographic image may be displayed in a distinguishable manner. Therefore, identification information for identifying each layer or layer boundary may be given to the tomographic image 55 shown in FIG. 11. For example, lines emphasizing the layer boundaries may be superimposed on the positions of the first to sixth layer boundaries in the tomographic image 55. Further, text associated with the first to sixth layer boundaries in the tomographic image 55 may be displayed.

[0103] <Re-photographing> These processes can be similarly performed when the ophthalmologic image capturing device 11B is performing ophthalmologic image processing. Also, for example, a plurality of structural abnormality degree maps 51A to 51F may be displayed on the confirmation screen in the display mode of Fig. 11.

[0104] The control unit may execute a process of outputting an instruction to the ophthalmologic image capturing device to capture images of regions on the structural abnormality map where the degree of structural abnormality is equal to or greater than a threshold. The control unit may also execute a process of displaying, on a display device, tomographic images or enlarged images of regions where the degree of structural abnormality is equal to or greater than a threshold. In this case, images of regions with a high degree of structural abnormality can be properly confirmed by the user.

[0105] When outputting an instruction to capture an image of a region where the degree of structural abnormality is equal to or greater than a threshold, the control unit may also output an instruction to capture a tomographic image of the region where the degree of structural abnormality is equal to or greater than a threshold with higher image quality. For example, the control unit may output an instruction to acquire a tomographic image with higher resolution. Alternatively, the control unit may output an instruction to capture a tomographic image of the region where the degree of structural abnormality is equal to or greater than a threshold multiple times and acquire an arithmetic average image of the captured multiple tomographic images. In this case, an ophthalmologic image of the region where the degree of structural abnormality is high is acquired with high image quality.

[0106] "Example of transformation" The techniques disclosed in the above embodiments are merely examples, and therefore, the techniques exemplified in the above embodiments can be modified.

[0107] <Follow-up observation> The ophthalmologic image processing device 23 may display, for follow-up observation, structural abnormality degree maps based on ophthalmologic images taken of the same subject's eye at different dates and times (hereinafter referred to as multiple time-series structural abnormality degree maps). For example, the multiple time-series structural abnormality degree maps may include multiple structural abnormality degree maps generated for each layer or boundary, and multiple time-series structural abnormality degree maps may be displayed for each layer or boundary. In addition, a trend graph showing changes in the abnormality degree for each layer or boundary may be displayed.

[0108] At this time, a difference map may be generated and displayed based on the two time-series structural anomaly maps. A two-dimensional distribution of increases and decreases in the structural anomaly may be displayed as the difference map. The difference map allows the user to easily check the temporal change in the structural anomaly at each position.

[0109] <Selectively display one or more structural abnormality maps for each layer according to the type of disease> In the above embodiment, a display mode in which multiple structural abnormality maps for each layer or boundary generated from a three-dimensional tomographic image are all simultaneously displayed on a display device is described. However, the display mode of the structural abnormality maps is not necessarily limited to this. For example, one or more structural abnormality maps corresponding to a disease type may be selectively displayed from multiple structural abnormality maps generated from a three-dimensional tomographic image. In this case, two or more structural abnormality maps corresponding to a disease type may be displayed simultaneously. This can suitably reduce the amount of information that the user needs to confirm.

[0110] <Searching and identifying types of diseases using structural abnormality maps> In the above embodiment, the disease type was manually selected by the user via the box 54. However, this is not necessarily limited to this, and the disease type may be automatically selected (classified). For example, information indicating the disease type may be obtained as a result of an automatic diagnosis. In this case, the automatic diagnosis may be performed based on the structural abnormality map. For example, the structural abnormality map may be input into a mathematical model that outputs an automatic diagnosis result regarding a disease of the subject's eye to obtain information indicating the disease type. The mathematical model may be trained using the structural abnormality map and the diagnosis result (e.g., disease type) that serves as the correct answer data for each structural abnormality map as training data. Since the structural abnormality map is a map that extracts only structural abnormalities from a three-dimensional tomographic image (information that is efficiently compressed with a focus on structural abnormalities), it is considered that a result that focuses more on structural abnormalities can be obtained compared to an automatic diagnosis result obtained based on the three-dimensional tomographic image itself. Furthermore, since the information in the structural abnormality map is compressed compared to the three-dimensional tomographic image, using the structural abnormality map allows for faster automatic diagnosis results to be obtained. Therefore, as in the above-described embodiments and examples, the structural abnormality degree map may be used not only for the purpose of identifying a map to be selectively displayed or highlighted from among multiple structural abnormality degree maps generated for each layer or boundary, but also for the automatic diagnosis of the subject's eye itself.

[0111] Furthermore, in the field of image diagnosis, a technique for searching for similar cases based on images of a subject is known. By inputting a search query into a similar case search database, similar case data corresponding to the search query can be obtained. The similar case data may be at least one of ophthalmological images of similar cases, diagnosis results for similar cases, and follow-up results for similar cases. A structural abnormality map may be used as a search query in such a similar case search. When a structural abnormality map is used as a search query, it is considered possible to obtain similar cases more quickly than when a three-dimensional tomographic image is input as a search query. [Explanation of symbols]

[0112] 11A, 11B Ophthalmic Imaging Device 13A, 13B CPU 21 Ophthalmic Image Processing Device 23 CPU 24 Memory Device 28 Display Device 30 Training Ophthalmic Image 31 Training Data 40, 51 Two-Dimensional Tomographic Image 52 Structural Abnormality Degree Graph 53 Deviation Degree Table 51A~51F Structural Abnormality Degree Map

Claims

1. An ophthalmic image processing apparatus for processing an ophthalmic image of an eye to be examined, wherein a control unit of the ophthalmic image processing apparatus acquires an ophthalmic image including tomographic images of a plurality of tomographic planes in the fundus of the eye to be examined, and by inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm, obtains a probability distribution for identifying two or more layers or layer boundaries of the fundus, which are included in a plurality of tissues in the tomographic image, generates a structural abnormality degree map representing a two-dimensional distribution of the degree of abnormality of the structure at the layer or layer boundary, such that the correspondence between the gradation value of each pixel in the structural abnormality degree map and the degree of abnormality of the structure varies according to the layer or layer boundary, for each of two or more layers or layer boundaries based on the probability distribution, An ophthalmic image processing apparatus that simultaneously arranges and displays two or more of the structural abnormality degree maps generated for each of two or more layers or layer boundaries on a display device.

2. The ophthalmic image processing apparatus according to claim 1, wherein the control unit selects a disease type and sets the correspondence for each layer or layer boundary according to the selected disease type.

3. The ophthalmic image processing apparatus according to claim 1, wherein the control unit sets the correspondence for each layer or layer boundary according to the noise level in the ophthalmic image.

4. The ophthalmic image processing apparatus according to any one of claims 1 to 3, wherein the control unit further acquires a frontal image of the eye to be examined corresponding to the structural abnormality degree map, and displays the frontal image on a display device together with the two or more structural abnormality degree maps.

5. The ophthalmic image processing apparatus according to any one of claims 1 to 4, wherein the mathematical model is trained using a training data set having, on the input side, an ophthalmic image including tomographic images of a plurality of tomographic planes in an eye to be examined taken in the past, and on the output side, data indicating a layer or layer boundary in the tomographic image on the input side.

6. An ophthalmic image processing program executed by an ophthalmic image processing apparatus for processing an ophthalmic image of an eye to be examined, wherein when the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing apparatus, an image acquisition step of acquiring an ophthalmic image including tomographic images of a plurality of tomographic planes in the fundus of the eye to be examined, as an ophthalmic image taken by an ophthalmic image photographing apparatus, An acquisition step of acquiring a probability distribution for identifying two or more layers or layer boundaries of the fundus, included in a plurality of tissues in the tomographic image, by inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm; A structural abnormality map generation step of generating, for each of two or more layers or layer boundaries, a structural abnormality map representing a two-dimensional distribution of the degree of structural abnormality at the layer or layer boundary, where the correspondence between the gradation value of each pixel in the structural abnormality map and the degree of structural abnormality is different according to the layer or layer boundary, based on the probability distribution; A display step of simultaneously arranging and displaying on a display device two or more of the structural abnormality maps generated for each of two or more layers or layer boundaries; An ophthalmic image processing program, characterized by causing the ophthalmic image processing apparatus to execute the above steps.

Images