Ophthalmic image processing device, OCT device, and ophthalmic image processing program

Abstract

The control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device (S11). The control unit inputs the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtaining a probability distribution of the coordinates of at least one of a specific boundary and a specific part of a tissue existing in a region in the ophthalmic image as a random variable (S14). The control unit detects at least one of the specific boundary and the specific part based on the obtained probability distribution (S16, S24).

Description

Technical Field

[0001] The present disclosure relates to an ophthalmologic image processing device that processes an ophthalmologic image of an eye to be inspected, an OCT device, and an ophthalmologic image processing program executed in the ophthalmologic image processing device. Background Art

[0002] In the past, various technologies have been proposed for detecting at least one of the boundaries of multiple tissues (e.g., multiple layers) shown in an ophthalmic image and specific parts on the tissue shown in the ophthalmic image (hereinafter, sometimes referred to as "boundary / specific part"). For example, in the technology disclosed in non-patent document 1, first, each pixel is mapped to which layer each pixel belongs. Then, the thickness of each layer is output based on the mapping result, thereby detecting the boundary of each layer.

[0003] As another example, various techniques for estimating abnormalities in the structure of an object shown in an image have been proposed. For example, in the technique disclosed in non-patent document 2, first, a normal image is used as training data to train a generative adversarial network (GAN). The trained GAN learns how to map the input image to coordinates in a latent space. If there is an abnormality in the structure of the input image, a difference occurs between the image generated by mapping and the input image. In the technique of non-patent document 2, the error between the image generated by mapping and the input image is taken to attempt to estimate the abnormal part of the structure.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Yufan He, Aaron Carass, et al. "Topology guaranteed segmentation of the human retina from OCT using convolutional neural networks." arXiv: 1803.05120, 14 Mar 2018

[0007] Non-patent literature 2: Thomas Schlegl, et al. "Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery" arXiv: 1703.05921, 17Mar 2017 Summary of the invention

[0008] In the method of Non-Patent Document 1, it is difficult to accurately map which tissue each pixel belongs to when the tissue structure is damaged due to a disease, etc. As a result, the detection accuracy of the tissue boundary / specific part is also reduced.

[0009] In the method of non-patent document 2, the error between the image generated by mapping and the input image is taken to attempt to estimate the abnormal part, so it is difficult to quantify the degree of structural abnormality. Therefore, it is difficult for the user to appropriately judge the abnormality of the structure of the tissue shown in the ophthalmic image.

[0010] A typical object of the present disclosure is to provide an ophthalmologic image processing device, an OCT device, and an ophthalmologic image processing program capable of appropriately detecting at least one of a boundary of a tissue and a specific part shown in an ophthalmologic image.

[0011] Another typical object of the present disclosure is to provide an ophthalmologic image processing apparatus, an OCT apparatus, and an ophthalmologic image processing program that enable a user to appropriately determine abnormalities in the structure of tissues shown in ophthalmologic images.

[0012] An ophthalmic image processing device provided in a first manner according to a typical embodiment of the present disclosure processes an ophthalmic image which is an image of a tissue of an eye to be inspected. A control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device, and inputs the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtaining a probability distribution of the coordinates of at least one of a specific boundary and a specific part of the tissue existing in an area in the ophthalmic image as a random variable, and based on the obtained probability distribution, detects at least one of the specific boundary and the specific part.

[0013] The second mode of a typical embodiment of the present disclosure provides an OCT device that captures an ophthalmic image of the tissue by processing an OCT signal generated by using reference light and reflected light of measurement light irradiated onto the tissue of the eye to be inspected. The control unit of the OCT device inputs the captured ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtaining a probability distribution of the coordinates of at least one of a specific boundary and a specific part of the tissue existing in an area in the ophthalmic image as a random variable, and detects the specific boundary and at least one of the specific part based on the obtained probability distribution.

[0014] The third mode of a typical implementation of the present disclosure provides an ophthalmic image processing program that is executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of a tissue of an eye to be inspected. The ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device, thereby causing the ophthalmic image processing device to execute the following steps: an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a probability distribution acquisition step of inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby acquiring a probability distribution of the coordinates of at least one of a specific boundary and a specific part of the tissue existing in an area in the ophthalmic image as a random variable; and a detection step of detecting at least one of the specific boundary and the specific part based on the acquired probability distribution.

[0015] According to the ophthalmologic image processing device of the first aspect, the OCT device of the second aspect, and the ophthalmologic image processing program of the third aspect, at least one of a boundary of a tissue and a specific part shown in an ophthalmologic image can be appropriately detected.

[0016] The fourth mode of a typical embodiment of the present disclosure provides an ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of an eye to be inspected. The control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device, and obtains a probability distribution for identifying tissue in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, and obtains a deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information representing the abnormality of the structure of the tissue.

[0017] The fifth mode of a typical embodiment of the present disclosure provides an OCT device that captures an ophthalmic image of the tissue by processing an OCT signal generated by using reference light and reflected light of measurement light irradiated onto the tissue of the eye to be inspected. The control unit of the OCT device obtains a probability distribution for identifying the tissue in the ophthalmic image by inputting the captured ophthalmic image into a mathematical model trained using a machine learning algorithm, and obtains a deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information representing the abnormality of the structure of the tissue.

[0018] The sixth mode of a typical embodiment of the present disclosure provides an ophthalmic image processing program which is executed by an ophthalmic image processing device which processes an ophthalmic image which is an image of tissue of an eye to be inspected. The ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device, thereby causing the ophthalmic image processing device to execute the following steps: an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a probability distribution acquisition step of acquiring a probability distribution for identifying tissue in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm; and a structure information acquisition step of acquiring a deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structure information representing the abnormality of the structure of the tissue.

[0019] According to the ophthalmologic image processing apparatus according to the fourth aspect, the OCT apparatus according to the fifth aspect, and the ophthalmologic image processing program according to the sixth aspect, it is possible for a user to appropriately determine abnormalities in tissues that appear in ophthalmologic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a block diagram showing a schematic configuration of the mathematical model construction device 1 , the ophthalmologic image processing device 21 , and the ophthalmologic image capturing devices 11A and 11B.

[0021] Figure 2 4 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1 .

[0022] Figure 3 1 is a diagram showing an example of the training ophthalmologic image 30 .

[0023] Figure 4 This is a diagram showing an example of the training data 31.

[0024] Figure 5 4 is a flowchart of the boundary detection process executed by the ophthalmologic image processing device 21 .

[0025] Figure 6 1 is a diagram schematically showing the relationship between the two-dimensional tomographic image 40 input to the mathematical model and the one-dimensional areas A1 to AN in the two-dimensional tomographic image 40 .

[0026] Figure 7 This is an example of a graph showing the probability distribution of the coordinates of the boundary Bi, using the one-dimensional coordinates of the one-dimensional area A1 as random variables.

[0027] Figure 8 This is a diagram showing an example of a probability map of a boundary Bi serving as the internal limiting membrane (ILM).

[0028] Fig. 9This is a diagram showing an example of a probability map of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL).

[0029] Fig.10 4 is a flowchart of the specific part detection process executed by the ophthalmologic image processing device 21 .

[0030] Fig.11 Schematically shows the relationship between the ophthalmologic image 50 input to the mathematical model and the coordinate system C of the two-dimensional coordinates in the ophthalmologic image 50 .

[0031] Fig.12 This is an example of a graph showing the probability distribution of the coordinates of a specific part, using two-dimensional coordinates as random variables.

[0032] Fig.13 This is a flowchart of ophthalmologic image processing according to the second embodiment executed by the ophthalmologic image processing device 21 .

[0033] Fig.14 This is an example of a graph showing a probability distribution for identifying a boundary Bi when the abnormality of the structure near the boundary Bi is high.

[0034] Fig.15 This is an example of a display screen displaying a two-dimensional tomographic image 51A, a structural abnormality degree graph 52A, and a deviation degree table 53A.

[0035] Fig.16 This is an example of a display screen displaying a two-dimensional tomographic image 51B, a structural abnormality degree graph 52B, and a deviation degree table 53B.

[0036] Fig.17 1 is a diagram showing an example of the structural abnormality degree map 55 . DETAILED DESCRIPTION

[0037] <Summary>

[0038] The control unit of the ophthalmic image processing device illustrated in one example of the present disclosure obtains an ophthalmic image captured by an ophthalmic image capturing device. The control unit inputs the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtaining a probability distribution that takes the coordinates of at least one of a specific boundary and a specific part of a tissue existing in a region in the ophthalmic image as a random variable. The control unit detects at least one of the specific boundary and the specific part based on the obtained probability distribution.

[0039] According to the ophthalmic image processing device illustrated in one example of the present disclosure, for example, even when the structure of the tissue is damaged due to the influence of a disease, or the image quality of at least a portion of the ophthalmic image is poor, it is possible to appropriately and directly detect the boundary / specific part based on the probability distribution of the coordinates as a random variable.

[0040] In addition, in one example of the above-mentioned prior art, after mapping which layer each pixel belongs to, the thickness of the layer is output based on the mapping result, and the boundary of the layer is finally detected. In this case, since multiple stages of processing are required, it is difficult to reduce the processing amount. In contrast, in the ophthalmic image processing device illustrated in one example of the present disclosure, the processing amount of detecting the boundary / specific part based on the probability distribution of the coordinates as random variables is small. Therefore, the processing amount can also be easily reduced.

[0041] In addition, the aforementioned "region" may be any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region. When the "region" is a one-dimensional region, the coordinates are one-dimensional coordinates. Similarly, when the "region" is a two-dimensional region, the coordinates are two-dimensional coordinates, and when the "region" is a three-dimensional region, the coordinates are three-dimensional coordinates.

[0042] In addition, the "specific boundary" for which the probability distribution is obtained may be one boundary or multiple boundaries. When the probability distribution of multiple boundaries is obtained, the control unit may obtain the probability distribution for each of the multiple boundaries. Similarly, the number of "specific parts" for which the probability distribution is obtained may be one or multiple.

[0043] The process of detecting the boundary / specific part based on the probability distribution may also use a mathematical model similar to the process of acquiring the probability distribution. Alternatively, the control unit itself may detect the boundary / specific part based on the acquired probability distribution without using a mathematical model.

[0044] The mathematical model can be trained using a training data set, wherein the input side is set to data of an ophthalmic image of the tissue of the subject's eye taken in the past, and the output side is set to data representing the position of at least one of a specific boundary and a specific part of the tissue in the ophthalmic image on the input side. In this case, the trained mathematical model can appropriately output the probability distribution of the specific boundary / specific part with the coordinates as a random variable by inputting the ophthalmic image.

[0045] In addition, a mathematical model construction method performed by a mathematical model construction device for constructing a mathematical model can be expressed as follows. A mathematical model construction method is performed by a mathematical model construction device for constructing a mathematical model, wherein the mathematical model is constructed by training using a machine learning algorithm, and outputs data corresponding to the input ophthalmic image by inputting an ophthalmic image, wherein the mathematical model construction method comprises the following steps: an output training data acquisition step, in which an ophthalmic image of the tissue of the eye to be inspected is acquired as a training ophthalmic image; an output training data acquisition step, in which training data is acquired, the training data representing the position of at least one of a specific boundary and a specific part of the tissue in the training ophthalmic image acquired in the input training data acquisition step; a training step, in which the mathematical model is trained by using the data of the training ophthalmic image as input training data and using the training data as output training data, thereby constructing a mathematical model that outputs a probability distribution of the coordinates of at least one of a specific boundary and a specific part of the tissue existing in a region in the input ophthalmic image as a random variable.

[0046] The control unit may input the ophthalmic image into the mathematical model, thereby obtaining a probability distribution of a random variable that a specific boundary exists in a one-dimensional region extending in a direction intersecting a specific boundary of the tissue in the ophthalmic image. The control unit may detect the specific boundary based on the obtained probability distribution. In this case, since it is easy to obtain a probability distribution with an appropriate bias, it is easy to improve the detection accuracy of the specific boundary.

[0047] In addition, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image obtained by an OCT device. In this case, a probability distribution using one-dimensional coordinates as a random variable can be obtained for a one-dimensional region extending in the direction of the optical axis of the measurement light of the OCT device (the so-called "A-scan direction"). The direction of the boundary of the tissue is close to being perpendicular to the A-scan direction. Therefore, by using the direction in which the one-dimensional region extends as the A-scan direction, the probability distribution is easily biased appropriately. In addition, the possibility that the one-dimensional region has more than two intersections with a specific boundary is also reduced. Therefore, the detection accuracy of a specific boundary is improved. In addition, the control unit may also obtain a probability distribution using one-dimensional coordinates as a random variable for a one-dimensional region extending perpendicularly to a specific boundary. In this case, the probability distribution is more likely to be biased.

[0048] The control unit may detect the two-dimensional or three-dimensional boundary based on a plurality of probability distributions obtained for each of a plurality of mutually different one-dimensional regions. In this case, the two-dimensional or three-dimensional boundary can be appropriately grasped.

[0049] In addition, after post-processing the probability distribution associated with each of the multiple one-dimensional regions, the two-dimensional or three-dimensional boundary can be detected. For example, the boundary on the Nth one-dimensional region can be detected based on the probability distribution on the Nth one-dimensional region and the probability distribution on the one-dimensional region located near the Nth one-dimensional region. In this case, for example, a well-known method such as graph cutting can be used. In addition, the two-dimensional or three-dimensional boundary can also be detected by using the theory of the shortest path search to find the path with the minimum weight.

[0050] The control unit may also obtain a two-dimensional or three-dimensional map representing the possibility of a specific boundary, which is generated based on a plurality of probability distributions obtained for each of a plurality of one-dimensional regions. That is, the ophthalmic image processing apparatus may also perform a mapping acquisition step of acquiring a two-dimensional or three-dimensional map representing the possibility of a specific boundary. In this case, the position of a specific two-dimensional or three-dimensional boundary can be appropriately grasped based on the map.

[0051] In addition, the control unit may display the acquired map on a display device. In this case, the user can appropriately grasp the position of a specific boundary of two or three dimensions by viewing the displayed map. The map may be generated by the control unit based on a plurality of probability distributions output from a mathematical model. In addition, the map may be output by a mathematical model.

[0052] The ophthalmic image may also be a three-dimensional tomographic image obtained by an OCT device. The control unit may obtain a two-dimensional frontal image (so-called "Enface image") when observing a specific layer included in the three-dimensional tomographic image from the direction along the optical axis of the measuring light based on the three-dimensional tomographic image and the detected three-dimensional boundary. That is, the ophthalmic image processing device may perform an Enface image acquisition step of acquiring an Enface image based on the three-dimensional tomographic image and the detected three-dimensional boundary. In this case, after determining a specific layer based on the appropriately detected boundary, the Enface image of the specific layer can be appropriately acquired. The Enface image may be generated by the control unit. In addition, the Enface image may also be output by a mathematical model.

[0053] In addition, the control unit may obtain a thickness map that two-dimensionally represents the thickness distribution of a specific layer included in the three-dimensional tomographic image based on the three-dimensional tomographic image and the detected three-dimensional boundary. In this case, a more accurate thickness map can be obtained after the specific layer is determined based on the appropriately detected boundary.

[0054] The control unit may also input the ophthalmic image into the mathematical model, thereby obtaining a probability distribution of the probability that the specific part exists in the two-dimensional or more area in the ophthalmic image, with the coordinates of two or more dimensions as random variables. The control unit may detect the specific part based on the obtained probability distribution. In this case, the specific part can be appropriately detected in the two-dimensional area or the three-dimensional area.

[0055] In another example of the present disclosure, a control unit of an ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device. The control unit obtains a probability distribution for identifying tissues in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm. The control unit obtains the deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating the abnormality of the structure of the tissue.

[0056] In another example of the present disclosure, when there is no abnormality in the structure of the tissue, it is easy to accurately identify the tissue through a mathematical model, so the obtained probability distribution is easy to be biased. On the other hand, when there is an abnormality in the structure of the tissue, the obtained probability distribution is difficult to be biased. Therefore, the deviation between the probability distribution when the tissue is accurately identified and the probability distribution actually obtained increases or decreases according to the abnormality of the structure. Therefore, according to the ophthalmic image processing device involved in another example of the present disclosure, the abnormality of the structure is appropriately quantified by the deviation. In addition, when training the mathematical model, even if a large number of ophthalmic images of tissues with abnormalities are not used, the abnormality of the structure of the tissue can be grasped based on the deviation. As a result, the user can appropriately judge the abnormality of the structure of the tissue shown in the ophthalmic image.

[0057] Alternatively, the degree of deviation may be output by a mathematical model. Alternatively, the degree of deviation may be calculated by the control unit based on a probability distribution output by the mathematical model.

[0058] In another example of the present disclosure, a mathematical model can be trained using a training data set, wherein the input side is set to data of an ophthalmic image of tissue of a subject's eye taken in the past, and the output side is set to data representing the tissue in the ophthalmic image on the input side. In this case, the trained mathematical model can appropriately output a probability distribution for identifying the tissue by inputting an ophthalmic image.

[0059] In addition, in another example of the present disclosure, the specific method of the mathematical model for outputting the probability distribution can be appropriately selected. For example, the mathematical model can output the coordinates of at least one of the specific boundary and specific part of the tissue in the region in the input ophthalmic image as a probability distribution of a random variable. In this case, the tissue in the ophthalmic image (in this case, at least one of the boundary and the specific position) can be appropriately and directly identified based on the probability distribution output by the mathematical model. In addition, in this case, the "specific boundary" as the object of obtaining the probability distribution can be one boundary or multiple boundaries. In the case of obtaining the probability distribution of multiple boundaries, the control unit can obtain the probability distribution for each of the multiple boundaries. Similarly, the number of "specific parts" as the object of obtaining the probability distribution can also be one or more. In addition, the region in the ophthalmic image as the unit for obtaining the probability distribution can be any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region. The dimension of the coordinates as a random variable can be consistent with the dimension of the region as the unit for obtaining the probability distribution. In addition, the mathematical model can output the probability distribution of the types of each tissue in the subject's eye as a random variable for each region (for example, each pixel) of the input ophthalmic image. In this case, it is also possible to identify tissue in the ophthalmic image based on the probability distribution output by the mathematical model.

[0060] The degree of deviation may include the entropy (mean information content) of the obtained probability distribution. Entropy indicates the degree of uncertainty, randomness, and disorder. In the present disclosure, the entropy of the probability distribution output when the tissue is accurately identified is 0. In addition, the more the abnormality of the structure of the tissue increases, the more difficult it is to identify the tissue, and the more the entropy increases. Therefore, by using the entropy of the probability distribution as the degree of deviation, the abnormality of the structure of the tissue can be quantified more appropriately.

[0061] However, a value other than entropy may be used as the deviation. For example, at least one of the standard deviation, coefficient of variation, variance, etc., which represents the dispersion of the obtained probability distribution, may be used as the deviation. The KL divergence, which is a measure of the difference between the probability distributions, may also be used as the deviation. In addition, the maximum value of the obtained probability distribution may also be used as the deviation.

[0062] In another example of the present disclosure, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image of a tissue. The control unit may obtain a probability distribution of one or more layers or boundaries included in a plurality of layers and layer boundaries for identifying the ophthalmic image by inputting the ophthalmic image into a mathematical model. The control unit may obtain a deviation for the one or more layers or boundaries. In this case, the abnormality of the structure of a specific layer or boundary can be appropriately grasped based on the deviation. For example, in the case where only the deviation of a layer or boundary that is prone to structural abnormalities due to the influence of a disease is obtained, the obtained deviation may be more useful information for the disease.

[0063] In another example of the present disclosure, the control unit may obtain at least one of a graph and a map indicating the magnitude of the deviation for a position in the tissue. The user can appropriately understand which position in the tissue has a high degree of abnormality based on at least one of the graph and the map.

[0064] In another example of the present disclosure, for example, the ophthalmic image may be a two-dimensional image extending in the XZ direction. The control unit may obtain a probability distribution with a one-dimensional coordinate as a random variable for each of a plurality of one-dimensional regions extending parallel to the Z direction on the ophthalmic image by inputting the ophthalmic image into a mathematical model, and obtain a deviation for each of the plurality of one-dimensional regions. The control unit may also obtain a structural abnormality chart indicating the magnitude of the deviation at each position in the X direction based on the deviation obtained for each of the plurality of one-dimensional regions. In this case, the user can appropriately grasp which position in the X direction in the two-dimensional ophthalmic image has a high degree of abnormality based on the structural abnormality chart.

[0065] In this case, the two-dimensional image may be a two-dimensional tomographic image of the tissue or a two-dimensional frontal image of the tissue. In the case of a two-dimensional tomographic image, the deviation degree obtained for each of the multiple one-dimensional regions may be obtained for each layer or each boundary, or may be obtained as an average value of multiple layers and boundaries. In addition, the structural abnormality degree graph may be output by a mathematical model. In addition, the control unit may generate a structural abnormality degree graph based on the deviation degree obtained for each of the multiple axes. The control unit may also display the obtained structural abnormality degree graph on a display device.

[0066] In another example of the present disclosure, the control unit can obtain a structural abnormality map representing a two-dimensional distribution of deviations in the tissue. In this case, the user can accurately grasp the abnormality of the structure at each position in the two-dimensional region based on the structural abnormality map. In addition, the structural abnormality map can be output by a mathematical model. In addition, the control unit can also generate a structural abnormality map based on the obtained deviation. The control unit can also display the obtained structural abnormality map on a display device.

[0067] In another example of the present disclosure, the user can also understand the abnormality of the structure of the tissue without using a graph or a map. For example, the control unit may notify the user of the acquired deviation itself as structural information indicating the abnormality of the structure.

[0068] In another example of the present disclosure, the control unit may execute a process of outputting a photographing instruction to photograph a portion of the tissue having a deviation degree greater than a threshold value to an ophthalmic image photographing device. In addition, the control unit may also execute a process of displaying a tomographic image or an enlarged image of the portion having a deviation degree greater than the threshold value on a display device. In this case, the user can appropriately confirm the image of the portion having a high degree of structural abnormality.

[0069] In another example of the present disclosure, the control unit may output an instruction to take multiple ophthalmic images of a portion having a degree of deviation greater than a threshold value and obtain an additive average image of the multiple ophthalmic images taken, when outputting an instruction to take a photograph of a portion having a degree of deviation greater than a threshold value. In this case, an ophthalmic image of a portion having a high degree of structural abnormality can be obtained with high quality.

[0070] In another example of the present disclosure, the control unit can display an ophthalmic image with the highest deviation or an ophthalmic image with a deviation greater than a threshold value among a plurality of ophthalmic images obtained by photographing tissue of the same eye to be examined on a display device. For example, the control unit can make it easy for a user to confirm an ophthalmic image obtained by photographing a portion with a high degree of structural abnormality by displaying an ophthalmic image with a high degree of deviation among a plurality of ophthalmic images on a photographing confirmation screen for allowing a user to confirm the photographed ophthalmic image. In addition, when the control unit starts a viewer for allowing a user to confirm the photographed ophthalmic image, the control unit can make it possible for a user to first confirm an ophthalmic image obtained by photographing a portion with a high degree of structural abnormality by displaying an ophthalmic image with a high degree of deviation among a plurality of ophthalmic images.

[0071] In another example of the present disclosure, the control unit may input a two-dimensional tomographic image with the highest deviation or a two-dimensional tomographic image with a deviation greater than a threshold value among a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image into a mathematical model, and the mathematical model outputs an automatic diagnosis result related to a disease of the eye to be inspected. In this case, it is possible to efficiently obtain an automatic diagnosis result by using a two-dimensional tomographic image with a high degree of structural abnormality among a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image.

[0072] In another example of the present disclosure, the control unit may store the obtained deviation in a storage device. The control unit may also display multiple deviations associated with each of multiple ophthalmic images obtained by photographing the tissue of the same eye to be examined at different times on a display device. In this case, the user can appropriately grasp the development status of structural abnormalities, etc. by comparing multiple deviations. In addition, the control unit may display the values ​​of multiple deviations themselves, or may display multiple of the above-mentioned structural abnormality charts side by side. In addition, the control unit may display multiple of the above-mentioned structural abnormality maps side by side.

[0073] In another example of the present disclosure, the control unit can generate image quality evaluation information for evaluating the image quality of the ophthalmic image based on the deviation degree obtained for the ophthalmic image. The deviation degree may be high not only when there is an abnormality in the structure of the tissue, but also when the image quality of the ophthalmic image is poor. Therefore, by generating the image quality evaluation information based on the deviation degree, the image quality of the ophthalmic image can be appropriately grasped.

[0074] In addition, the specific method for generating the image quality evaluation information can be appropriately selected. For example, the value of the deviation itself can be used as the image quality evaluation information. In addition, when there is an abnormality in a part of the structure of the tissue, the deviation of the abnormal part is higher than the deviation of other parts. On the other hand, in the case of poor image quality, the deviation of each part of the ophthalmic image becomes higher as a whole. Therefore, when the deviation is obtained for each of the parts of the ophthalmic image, the control unit can generate image quality evaluation information indicating poor image quality when the deviation of each part is all high (for example, when the deviation of each part is all above the threshold, etc.). In addition, the control unit can also simultaneously consider the intensity of the signal of the captured ophthalmic image or the index of the goodness of the signal (for example, SSI (Signal Strength Index: signal strength index) or QI (Quality Index: quality index) etc.) and the obtained deviation to generate the image quality evaluation information.

[0075] In addition, in another example of the present disclosure, the control unit can generate information indicating the abnormality of the structure of the tissue based on the index of the strength or goodness of the signal representing the ophthalmic image and the obtained deviation. For example, when the index of the signal of the image is above the threshold and the deviation is above the threshold, the control unit can generate information indicating that there is a high possibility of abnormality in the structure. In this case, the abnormality of the structure can be more appropriately determined based on the image quality of the ophthalmic image.

[0076] In addition, the deviation can also be used for image quality evaluation information instead of being used as structural information representing the degree of abnormality of the structure of the tissue. In this case, the ophthalmic image processing device can be expressed as follows. An ophthalmic image processing device processes an ophthalmic image that is an image of the tissue of an eye to be inspected, characterized in that a control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device, and obtains a probability distribution for identifying the tissue in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, and obtains the deviation of the obtained probability distribution relative to the probability distribution when the tissue is accurately identified as image quality evaluation information for evaluating the image quality of the ophthalmic image.

[0077] In addition, in the present disclosure, various images can be used as ophthalmic images input to the mathematical model. For example, the ophthalmic image can be a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue of the eye to be inspected obtained by an OCT device. The tomographic image can also be taken by a device other than the OCT device (such as a Scheimpflug camera, etc.). In addition, the ophthalmic image can also be a two-dimensional frontal image taken by a fundus camera, a two-dimensional frontal image taken by a laser scanning fundus examination device (SLO), etc. The ophthalmic image can also be a two-dimensional frontal image (so-called "Enface image") generated based on the data of the three-dimensional tomographic image taken by the OCT device. In addition, the ophthalmic image can also be a two-dimensional frontal image (so-called "motion contrast image") generated based on motion contrast data obtained by processing multiple OCT data obtained from the same position at different times. The so-called two-dimensional frontal image is a two-dimensional image obtained by photographing the tissue from the optical axis direction of the shooting light. In addition, the tissue as the object of shooting can also be appropriately selected. For example, an image obtained by photographing any one of the fundus, anterior eye, and corner of the eye to be inspected can be used as an ophthalmic image.

[0078] <Implementation Method>

[0079] (Device Structure)

[0080] Hereinafter, typical embodiments of the present disclosure will be described with reference to the accompanying drawings. Figure 1 As shown, in this embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image capturing devices 11A and 11B are used. The ophthalmic image capturing devices 11A and 11B capture ophthalmic images that are images of tissues of the eye to be inspected. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm.

[0081] In the first embodiment described later, the constructed mathematical model outputs the coordinates of a specific boundary / specific part in the region of the ophthalmic image as a probability distribution of a random variable based on the input ophthalmic image. The ophthalmic image processing device 21 uses the mathematical model to obtain the probability distribution and detects the specific boundary / specific part based on the probability distribution.

[0082] In a second embodiment described later, the constructed mathematical model outputs a probability distribution for identifying tissues in the ophthalmic image based on the input ophthalmic image. The ophthalmic image processing device 21 obtains the probability distribution using the mathematical model, and obtains the deviation between the obtained probability distribution and the probability distribution when the tissue is accurately identified as structural information indicating the abnormality of the structure of the tissue.

[0083] As an example, the mathematical model construction device 1 of the present embodiment uses a personal computer (hereinafter referred to as a "PC"). However, the device that can function as the mathematical model construction device 1 is not limited to the PC. For example, the ophthalmic image capturing device 11A can also function as the mathematical model construction device 1. In addition, the control units of multiple devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic image capturing device 11A) can also cooperate to construct the mathematical model.

[0084] The details will be described later. As an example, the mathematical model construction device 1 can train the mathematical model by using the ophthalmic image obtained from the ophthalmic image shooting device 11A (hereinafter referred to as the "training ophthalmic image") and the training data representing the specific boundary / specific part of the tissue in the training ophthalmic image, thereby constructing the mathematical model. In addition, as another example, the mathematical model construction device 1 can also train the mathematical model by using the training ophthalmic image and the training data representing the position of at least any tissue in the training ophthalmic image, thereby constructing the mathematical model.

[0085] In addition, the ophthalmic image processing device 21 of the present embodiment uses a PC. However, the device that can function as the ophthalmic image processing device 21 is not limited to the PC. For example, the ophthalmic image capturing device 11B or a server or the like can also function as the ophthalmic image processing device 21. In addition, a mobile terminal such as a tablet terminal or a smart phone can also function as the ophthalmic image processing device 21. The control units of multiple devices (for example, the CPU of a PC and the CPU 13B of the ophthalmic image capturing device 11B) can also cooperate to perform various processes.

[0086] When the ophthalmic image capturing device (OCT device in this embodiment) 11B functions as the ophthalmic image processing device 21, as an example, the ophthalmic image capturing device 11B can appropriately detect a specific boundary / specific part in the tissue of the captured ophthalmic image while capturing the ophthalmic image. When the ophthalmic image capturing device (OCT device in this embodiment) 11B functions as the ophthalmic image processing device 21, as another example, the ophthalmic image capturing device 11B can obtain a deviation from the captured ophthalmic image while capturing the ophthalmic image.

[0087] In addition, in this embodiment, a case where a CPU is used as an example of a controller for performing various processes is illustrated. However, at least a portion of the various devices may also use a controller other than a CPU. For example, a GPU may be used as a controller to achieve high-speed processing.

[0088] The mathematical model construction device 1 is described. The mathematical model construction device 1 is configured, for example, in an ophthalmic image processing device 21 or a manufacturer that provides an ophthalmic image processing program to a user. The mathematical model construction device 1 includes a control unit 2 and a communication I / F 5 that perform various control processes. The control unit 2 includes a CPU 3 as a controller responsible for control and a storage device 4 that can store programs and data. The storage device 4 stores data for executing the mathematical model construction process described later (see Figure 2 ) of the mathematical model building program. In addition, the communication I / F 5 connects the mathematical model building device 1 to other devices (for example, the ophthalmologic image capturing device 11A and the ophthalmologic image processing device 21, etc.).

[0089] The mathematical model building device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user so that the user can input various instructions to the mathematical model building device 1. The operation unit 7 can use at least any one of a keyboard, a mouse, a touch panel, etc., for example. In addition, a microphone for inputting various instructions can also be used together with or instead of the operation unit 7. The display device 8 displays various images. The display device 8 can use various devices that can display images (for example, at least any one of a monitor, a display, a projector, etc.). In addition, the "image" in the present disclosure includes both static images and dynamic images.

[0090] The mathematical model construction device 1 can obtain ophthalmic image data (hereinafter sometimes referred to as "ophthalmic image") from the ophthalmic image capturing device 11A. The mathematical model construction device 1 can obtain ophthalmic image data from the ophthalmic image capturing device 11A through at least one of wired communication, wireless communication, and a removable storage medium (such as a USB memory).

[0091] The ophthalmic image processing device 21 is described. The ophthalmic image processing device 21 is, for example, disposed in a facility (e.g., a hospital or health diagnosis facility) that performs diagnosis or examination of a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller in charge of control and a storage device 24 that can store programs and data. The storage device 24 stores programs for performing ophthalmic image processing (e.g., for the first embodiment, Figure 5 The illustrated boundary detection process and Fig.10 For the second embodiment, refer to Fig.13 ) of the ophthalmologic image processing program. The ophthalmologic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmologic image processing device 21 with other devices (for example, the ophthalmologic image capturing device 11B and the mathematical model construction device 1, etc.).

[0092] The ophthalmologic image processing apparatus 21 is connected to an operation unit 27 and a display device 28. As the operation unit 27 and the display device 28, various devices can be used, similarly to the operation unit 7 and the display device 8 described above.

[0093] The ophthalmic image processing device 21 can obtain ophthalmic images from the ophthalmic image capturing device 11B. The ophthalmic image processing device 21 can obtain ophthalmic images from the ophthalmic image capturing device 11B, for example, through at least one of wired communication, wireless communication, and a removable storage medium (e.g., a USB memory). In addition, the ophthalmic image processing device 21 can also obtain a program or the like for realizing the mathematical model constructed by the mathematical model construction device 1 through communication or the like.

[0094] The ophthalmic image capturing devices 11A and 11B are described. As an example, in the present embodiment, a case where an ophthalmic image capturing device 11A that provides an ophthalmic image to the mathematical model building device 1 and an ophthalmic image capturing device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used is described. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model building device 1 and the ophthalmic image processing device 21 may also obtain ophthalmic images from a plurality of ophthalmic image capturing devices. In addition, the mathematical model building device 1 and the ophthalmic image processing device 21 may also obtain ophthalmic images from a common ophthalmic image capturing device. In addition, the two ophthalmic image capturing devices 11A and 11B illustrated in the present embodiment have the same structure. Therefore, the two ophthalmic image capturing devices 11A and 11B are collectively described below.

[0095] In addition, in this embodiment, an OCT device is exemplified as the ophthalmic image capturing device 11 (11A, 11B). However, an ophthalmic image capturing device other than the OCT device (for example, a laser scanning fundus ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may also be used.

[0096] The ophthalmic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) for performing various control processes and an ophthalmic image capturing unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) as a controller in charge of control and a storage device 14 (14A, 14B) capable of storing programs, data, etc.

[0097] The ophthalmic image capturing unit 16 has various structures required for capturing an ophthalmic image of the eye to be inspected. The ophthalmic image capturing unit 16 of this embodiment includes an OCT light source, a branching optical element for branching the OCT light emitted from the OCT light source into a measurement light and a reference light, a scanning unit for scanning the measurement light, an optical system for irradiating the measurement light to the eye to be inspected, and a light receiving element for receiving a composite light of the light reflected by the tissue of the eye to be inspected and the reference light.

[0098] The ophthalmic image capturing device 11 is capable of capturing two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the eye to be inspected. In detail, the CPU 13 captures a two-dimensional tomographic image of a cross section intersecting the scan line by scanning the OCT light (measurement light) on the scan line. The two-dimensional tomographic image may also be an averaged image generated by performing an addition averaging process on a plurality of tomographic images of the same part. In addition, the CPU 13 is capable of capturing a three-dimensional tomographic image in the tissue by scanning the OCT light two-dimensionally. For example, the CPU 13 obtains a plurality of two-dimensional tomographic images by scanning the measurement light on a plurality of scan lines at different positions in a two-dimensional region when the tissue is observed from the front. Next, the CPU 13 obtains a three-dimensional tomographic image by combining the captured plurality of two-dimensional tomographic images.

[0099] (Mathematical model building process)

[0100] Reference Figure 2 to Figure 4, the mathematical model building process performed by the mathematical model building device 1 is explained. The mathematical model building process is executed by the CPU 3 according to the mathematical model building program stored in the storage device 4. In the mathematical model building process, the mathematical model is trained by using the training data set, thereby constructing a mathematical model of the probability distribution related to the tissue in the ophthalmic image. In more detail, in a first embodiment, in the mathematical model building process, a mathematical model of the probability distribution that outputs the coordinates of at least one of the specific boundaries and specific parts of the tissue in the area in the ophthalmic image is constructed as a random variable. In a second embodiment, in the mathematical model building process, a mathematical model of the probability distribution for identifying the tissue in the ophthalmic image is constructed. The training data set includes data on the input side (input training data) and data on the output side (output training data).

[0101] like Figure 2 As shown, the CPU 3 obtains the data of the ophthalmic image captured by the ophthalmic image capturing device 11A, that is, the data of the training ophthalmic image as input training data (S1). In the present embodiment, the data of the training ophthalmic image is generated by the ophthalmic image capturing device 11A and then obtained by the mathematical model building device 1. However, the CPU 3 may also obtain the data of the training ophthalmic image by obtaining a signal (e.g., an OCT signal) that is the basis for generating the training ophthalmic image from the ophthalmic image capturing device 11A and generating the training ophthalmic image based on the obtained signal. Figure 3 An example of a training ophthalmological image 30 which is a two-dimensional tomographic image of the fundus. Figure 3 In the illustrated ophthalmic training image 30 , a plurality of layers of the fundus are displayed.

[0102] In the first embodiment, the ophthalmic image processing device 21 performs the boundary detection process described later (see Figure 5 ), in S1, a two-dimensional tomographic image of the tissue of the eye to be inspected (for example, the fundus) is obtained as an ophthalmological image for training.

[0103] In the first embodiment, the ophthalmic image processing device 21 performs a specific part detection process (see Fig.10 ), in S1, a two-dimensional frontal image (for example, an Enface image captured by an ophthalmic image capturing device 11A as an OCT device, etc.) of the tissue of the eye to be examined (for example, the fundus) is obtained as a training ophthalmic image.

[0104] However, in the first embodiment, the ophthalmic image used as the training ophthalmic image can also be changed. For example, when a two-dimensional frontal image is used as the training ophthalmic image, the training ophthalmic image can be a two-dimensional frontal image captured by a device other than the OCT device (for example, an SLO device or a fundus camera, etc.). A two-dimensional image other than the frontal image (for example, a two-dimensional tomographic image) can also be used as the training ophthalmic image. In addition, in the specific part detection processing described later (see Fig.10 ), when a specific part is detected in three dimensions, in S1, a three-dimensional tomographic image can also be obtained as an ophthalmic image for training.

[0105] In addition, in S1 of the second embodiment, a two-dimensional tomographic image captured by an ophthalmic image capturing device 11A as an OCT device is obtained as a training ophthalmic image. In addition, in the second embodiment, an ophthalmic image of a tissue with a low degree of structural abnormality is obtained as a training ophthalmic image. In this case, it is difficult for the mathematical model trained by the training data set to recognize a tissue with a high degree of structural abnormality. As a result, the deviation degree obtained for an ophthalmic image with a high degree of structural abnormality is likely to become large.

[0106] However, in the second embodiment, the ophthalmic image used as the training ophthalmic image can also be changed. For example, a two-dimensional frontal image of the tissue of the eye to be examined can also be used as the training ophthalmic image. In this case, the ophthalmic image shooting device for shooting the training ophthalmic image can use various devices (for example, at least any one of an OCT device, an SLO device, a fundus camera, an infrared camera, a corneal endothelial cell shooting device, etc.). In addition, a three-dimensional image can also be used as the training ophthalmic image.

[0107] Next, the CPU 3 obtains training data (S2). In the first embodiment, the training data indicates the position of at least one of a specific boundary and a specific part of a tissue in the training ophthalmologic image. In the second embodiment, the training data indicates the position of at least one of the tissues in the training ophthalmologic image. Figure 4 An example of the training data 31 is shown when a two-dimensional tomographic image of the fundus is used as the training ophthalmologic image 30. As an example, the training data 31 may indicate the position of a specific boundary. Figure 4 The illustrated training data 31 includes data of labels 32A to 32F indicating the positions of the six boundaries in the training ophthalmic image 30. As an example, Figure 4The illustrated training data 31 includes data of labels 32A to 32F indicating the positions of six boundaries in a plurality of tissues (specifically, a plurality of layers and boundaries) shown in the training ophthalmologic image 30. In the present embodiment, the data of the labels 32A to 32F in the training data 31 are generated by the operator operating the operation unit 7 while observing the boundaries in the training ophthalmologic image 30. However, the method of generating the label data may be changed.

[0108] In the first embodiment, the ophthalmic image processing device 21 performs a specific part detection process (see Fig.10 ), in S2, data indicating the position of a specific part in the tissue of the two-dimensional tomographic image or the three-dimensional tomographic image as the training ophthalmic image is obtained. For example, the position of the fovea is detected in the specific part detection processing described later. In this case, in S2, data indicating the position of the fovea in the two-dimensional tomographic image or the three-dimensional tomographic image is obtained.

[0109] In the second embodiment, the training data can also be changed. For example, when a two-dimensional tomographic image of the fundus is used as the training ophthalmic image 30, the training data may be data indicating the position of at least one layer in the fundus. In addition, the training data may be data indicating the position of a point-like part in the tissue instead of the data indicating the position of the layer and the boundary.

[0110] Next, the CPU 3 executes training of the mathematical model using the training data set using a machine learning algorithm (S3). As machine learning algorithms, generally known ones include neural networks, random forests, boosting, and support vector machines (SVM).

[0111] Neural network is a method of simulating the behavior of biological neural cell networks. Examples of neural networks include feedforward (forward propagation) neural networks, RBF networks (radial basis functions), spike neural networks, convolutional neural networks, recursive neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0112] Random forest is a method for generating a large number of decision trees by learning based on randomly sampled training data. When using random forest, the branches of multiple decision trees that have been pre-learned as classifiers are tracked, and the average (or majority vote) of the results obtained from each decision tree is taken.

[0113] Boosting is a method of generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by learning only the weak classifiers one by one.

[0114] SVM is a method of constructing two types of pattern recognizers using linear input elements. SVM learns the parameters of linear input elements based on, for example, the principle of finding a margin-maximizing hyperplane that maximizes the distance to each data point based on training data (hyperplane separating theorem).

[0115] Mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. Mathematical model is constructed by training using training data set. As mentioned above, training data set is a collection of input training data and output training data. For example, relevant data (e.g., weight) of each input and output can be updated through training.

[0116] In this embodiment, a multi-layer neural network is used as a machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. Multiple nodes (also called neurons) are configured in each layer. In detail, in this embodiment, a convolutional neural network (CNN) is used as a type of multi-layer neural network.

[0117] In addition, the mathematical model constructed in the first embodiment outputs the coordinates (one of the one-dimensional coordinates, two-dimensional coordinates, and three-dimensional coordinates) of at least one of the specific boundary and specific part of the tissue existing in the region (one of the one-dimensional region, two-dimensional region, and three-dimensional region) in the ophthalmic image as a probability distribution of a random variable. In the first embodiment, in order to make the mathematical model output the probability distribution, a normalized exponential (Softmax) function is applied.

[0118] In the first embodiment, the ophthalmic image processing device 21 performs a boundary detection process (see Figure 5 ), the mathematical model constructed in S3 outputs the probability distribution of the coordinates of a specific boundary within a one-dimensional region on a line extending along a direction intersecting the specific boundary (in this embodiment, the A-scan direction of OCT) in the two-dimensional tomographic image.

[0119] In the first embodiment, the ophthalmic image processing device 21 performs a specific part detection process (see Fig.10 ), the mathematical model constructed in S3 outputs the two-dimensional coordinates of a specific part (for example, the fovea in this embodiment) existing in a two-dimensional region in the two-dimensional ophthalmic image as a probability distribution of a random variable.

[0120] As an example, the mathematical model constructed in the second embodiment outputs the coordinates (one-dimensional coordinates, two-dimensional coordinates, three-dimensional coordinates, and four-dimensional coordinates) of a specific tissue (e.g., a specific boundary, a specific layer, or a specific part, etc.) in an area (one-dimensional area, two-dimensional area, three-dimensional area, and any area in a four-dimensional area including a time axis) in an ophthalmic image as a probability distribution of a random variable, as a probability distribution for identifying tissue. In the second embodiment, in order to make the mathematical model output a probability distribution, a normalized exponential function is applied. In detail, the mathematical model constructed in S3 outputs the coordinates of a specific boundary in a one-dimensional area extending in a direction intersecting a specific boundary (in this embodiment, the A-scan direction of OCT) in a two-dimensional tomographic image as a probability distribution of a random variable.

[0121] However, in the second embodiment, the specific method of the mathematical model outputting the probability distribution for identifying the tissue can be appropriately changed. For example, the mathematical model may also output the probability distribution of the two-dimensional coordinates or three-dimensional coordinates of a specific tissue (e.g., a characteristic part, etc.) in a two-dimensional region or a three-dimensional region as a random variable, as a probability distribution for identifying the tissue. In addition, the mathematical model may also output the probability distribution of the types of multiple tissues (e.g., multiple layers and boundaries) in the eye to be examined as a random variable for each region (e.g., each pixel) of the input ophthalmic image. In addition, the ophthalmic image input to the mathematical model may also be a dynamic image.

[0122] In addition, other machine learning algorithms may also be used. For example, a generative adversarial network (GAN) using two competing neural networks may be used as a machine learning algorithm.

[0123] The processes of S1 to S3 are repeated until the mathematical model is constructed (S4: No). When the mathematical model is constructed (S4: Yes), the mathematical model construction process ends. The program and data for realizing the constructed mathematical model are incorporated into the ophthalmic image processing device 21.

[0124] (Ophthalmic Image Processing)

[0125] Reference Figures 5 to 17 , the ophthalmologic image processing performed by the ophthalmologic image processing device 21 will be described. The ophthalmologic image processing is performed by the CPU 23 according to the ophthalmologic image processing program stored in the storage device 21 .

[0126] (Boundary Detection Processing)

[0127] Reference Figures 5 to 9, a boundary detection process as an example of ophthalmic image processing involved in the first embodiment is described. In the boundary detection process, a specific boundary (a two-dimensional boundary and a three-dimensional boundary in the first embodiment) is detected based on the probability distribution of the one-dimensional coordinates of a specific boundary in each of a plurality of one-dimensional regions as a random variable. And, based on the detected three-dimensional boundary, an Enface image of a specific layer in the tissue is obtained.

[0128] First, the CPU 23 obtains a three-dimensional tomographic image of the tissue of the eye to be inspected (the fundus in the first embodiment) (S11). The three-dimensional tomographic image is captured by the ophthalmic image capturing device 11B and obtained by the ophthalmic image processing device 21. As described above, the three-dimensional tomographic image is formed by combining a plurality of two-dimensional tomographic images captured by scanning the measurement light on mutually different scanning lines. In addition, the CPU 23 may also obtain a signal (e.g., an OCT signal) that is the basis for generating the three-dimensional tomographic image from the ophthalmic image capturing device 11B, and generate the three-dimensional tomographic image based on the obtained signal.

[0129] The CPU 23 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images constituting the acquired three-dimensional tomographic image (S12). Figure 6 1 shows an example of a two-dimensional tomographic image 40. The two-dimensional tomographic image 40 shows a plurality of boundaries in the fundus of the eye to be inspected. Figure 6 In the example shown, a plurality of boundaries including the boundary Bi as the internal limiting membrane (ILM) and the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL) are shown. In addition, a plurality of one-dimensional areas A1 to AN are set in the two-dimensional tomographic image 40. In the first embodiment, the plurality of one-dimensional areas A1 to AN set in the two-dimensional tomographic image 40 extend along an axis intersecting with a specific boundary (in the first embodiment, a plurality of boundaries including the boundary Bi and the boundary Bg). In detail, the one-dimensional areas A1 to AN of the first embodiment coincide with the respective areas of the plurality of (N) A scans constituting the two-dimensional tomographic image 40 captured by the OCT device.

[0130] In addition, the method of setting multiple one-dimensional areas can also be changed. For example, CPU23 can also set multiple one-dimensional areas so that the angle between the axis of each one-dimensional area and a specific boundary is as close to vertical as possible. In this case, for example, based on the shape of the tissue of a common eye to be inspected (fundus in the first embodiment), the position and angle of each one-dimensional area can be set to an angle close to vertical.

[0131] The CPU 23 inputs the T-th two-dimensional tomographic image to the mathematical model, thereby obtaining the probability distribution of the coordinates of the M-th (initial value of M is "1") boundary in each of the plurality of one-dimensional areas A1 to AN (S14). Figure 7 An example of a graph showing the probability distribution of the coordinates of the boundary Bi obtained from the one-dimensional area A1 is shown. Figure 7 In the example shown, the one-dimensional coordinates of the one-dimensional area A1 are used as random variables, and the probability distribution of the coordinates of the boundary Bi is shown. Figure 7 In the example shown, the horizontal axis represents a random variable, and the vertical axis represents the probability of the random variable, and the random variable is the coordinate of the existence boundary Bi in the one-dimensional area A1. In S14, the probability distribution in each of the plurality of one-dimensional areas A1 to AN is obtained.

[0132] according to Figure 7 From the graph shown, it can be determined that among the points on the one-dimensional area A1, the point with the highest probability of the existence of the boundary Bi is point P. In addition, even when the structure of the tissue is destroyed due to the influence of a disease, or when the image quality of at least a part of the ophthalmic image is poor, the probability distribution output by the mathematical model is likely to form a peak at any position. Therefore, by using the probability distribution output in S14, the boundary can be appropriately and directly detected.

[0133] The CPU 23 obtains a probability map that two-dimensionally indicates the proportion of the existence of the Mth boundary in the Tth two-dimensional tomographic image. The CPU 23 displays the obtained probability map on the display device 28 (S15). The probability map is generated by two-dimensionally arranging the multiple probability distributions obtained for each of the multiple one-dimensional areas A1 to AN. The probability map can be generated by the CPU 23 based on the multiple probability distributions. In addition, the probability map can also be generated by a mathematical model.

[0134] Figure 8 This is an example of a probability map 41 of a boundary Bi of the internal limiting membrane (ILM). Fig. 9 is an example of a probability map 42 of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL). Figure 8 and Fig. 9 In the illustrated probability maps 41 and 42, the value for each coordinate is represented by brightness, and the position that is more similar to a specific boundary is brighter. Therefore, the user can appropriately grasp the position of the boundary Bi and Bg in a two-dimensional manner by viewing the displayed probability maps 41 and 42. In addition, it is self-evident that the expression method of the probability map can be appropriately changed.

[0135] Next, the CPU 23 detects the Mth boundary of the two-dimensional tomographic image based on the probability distribution associated with the Mth boundary obtained for each of the plurality of one-dimensional regions A1 to AN (S16). In the first embodiment, the CPU 23 detects the two-dimensional boundary by combining the plurality of probability distributions obtained for each of the plurality of one-dimensional regions A1 to AN. The two-dimensional boundary may be detected based on the plurality of probability distributions obtained in S14, or may be detected based on the probability map obtained in S15. In addition, the detection result of the boundary may be output by a mathematical model, or the CPU 23 may perform the detection process of the boundary.

[0136] In addition, in the first embodiment, after the probability distribution associated with the plurality of one-dimensional regions A1 to AN is post-processed, the generation of the probability map and the detection of the boundary are performed. As an example, the boundary on the n-th one-dimensional region can be detected based on the probability distribution on the n-th one-dimensional region and the probability distribution on the one-dimensional region located near the n-th one-dimensional region. In this case, for example, a known method such as graph cutting can be used. In addition, the boundary can also be detected by using the theory of the shortest path search for finding the path with the minimum weight.

[0137] In addition, CPU23 may also display the probability maps 41, 42 or the detected specific boundaries superimposed on the two-dimensional tomographic image (i.e., the original image) as the object of the detected boundary. In this case, the positional relationship between the actually photographed tissue and the boundary detected by the probability distribution can be properly grasped. In addition, CPU23 may input the user's instructions for specifying the position of the boundary in the tissue while displaying the probability maps 41, 42 or the detected boundary superimposed on the original image. In this case, the user can compare the boundary detected by the probability distribution with the image of the actually photographed tissue and appropriately specify the position of the boundary.

[0138] Next, the CPU 23 determines whether the detection of all the boundaries to be detected in the T-th two-dimensional tomographic image is completed (S18). If the detection of a part of the boundaries is not completed (S18: No), "1" is added to the order M of the boundaries (S19), and the process returns to S14 to perform the next boundary detection process (S14 to S16). When the detection of all the boundaries is completed (S18: Yes), the CPU 23 determines whether the detection of the boundaries of all the two-dimensional tomographic images constituting the three-dimensional tomographic image is completed (S21). If the detection of the boundaries of a part of the two-dimensional tomographic images is not completed (S21: No), "1" is added to the order T of the two-dimensional tomographic images (S22), and the process returns to S12 to detect the boundary of the next two-dimensional tomographic image (S12 to S19).

[0139] When the detection of the boundaries of all two-dimensional tomographic images is completed (S21: Yes), the detection of the three-dimensional boundaries is completed. The CPU 23 integrates the multiple two-dimensional boundaries detected in S16 for at least one of the M boundaries, thereby obtaining a three-dimensional boundary (S24). In addition, the CPU 23 can generate image data of the detected three-dimensional boundary and display it on the display device 28. Therefore, the user can appropriately grasp the three-dimensional shape of the boundary. In addition, based on the acquisition results of the probability distribution for T two-dimensional tomographic images, a probability map representing the probability distribution in the three-dimensional region can be generated.

[0140] Next, CPU23 obtains an Enface image (S25) of a specific layer (or boundary) included in the three-dimensional tomographic image based on the detected three-dimensional boundary and the three-dimensional tomographic image obtained in S11. The so-called Enface image is a two-dimensional frontal image when a specific layer is observed from the direction along the optical axis of the measurement light of OCT. Since the Enface image obtained in S25 is generated based on the boundary detected appropriately, the quality is high. In addition, in S25, an Enface image of a specific one of the multiple layers and boundaries included in the three-dimensional image can also be obtained. In addition, it is also possible to obtain Enface images of multiple layers and boundaries included in the three-dimensional image.

[0141] In addition, the CPU 23 may obtain a thickness map that two-dimensionally represents the thickness distribution of a specific layer included in the three-dimensional tomographic image based on the three-dimensional tomographic image obtained in S11 and the detected three-dimensional boundary. In this case, a more accurate thickness map can be obtained after the specific layer is determined based on the appropriately detected boundary.

[0142] In addition, Figure 5 In the example shown, after obtaining a three-dimensional tomographic image in S11, a specific boundary is detected in two dimensions and three dimensions. Then, based on the detected three-dimensional boundary, an Enface image is obtained. However, only a two-dimensional boundary may be detected. In this case, a two-dimensional ophthalmic image may also be obtained in S11. In addition, Figure 5 In the example shown, in S14, a plurality of probability distributions associated with each of the plurality of one-dimensional regions A1 to AN are obtained. As a result, a two-dimensional boundary is detected. However, for example, in the case where only one point on a specific boundary is detected, in S14, only the probability distribution in one one-dimensional region may be obtained.

[0143] (Specific part detection and processing)

[0144] Reference Figures 10 to 12, a specific part detection process as an example of ophthalmic image processing involved in the first embodiment is described. In the specific part detection process, the specific part is detected based on the probability distribution of the coordinates of the specific part. The following is an example of detecting a specific part from a two-dimensional ophthalmic image.

[0145] First, CPU23 obtains an ophthalmic image of the tissue of the eye to be inspected (fundus in the first embodiment) (S31). As an example, in S31 of the first embodiment, a two-dimensional frontal image (for example, an Enface image or a motion contrast image) obtained by an OCT device is obtained. However, the ophthalmic image obtained in S31 can be appropriately changed. For example, a two-dimensional frontal image obtained by an SLO device or a fundus camera can also be obtained in S31. In addition, an ophthalmic image obtained by photographing tissues other than the fundus can also be obtained in S31.

[0146] Fig.11 FIG. 5 shows an example of the ophthalmologic image 50 acquired in S31 of the first embodiment. Fig.11 In the illustrated two-dimensional ophthalmologic image 50 , the optic disc 51 , macula 52 , and fundus blood vessels 53 of the fundus are visualized. The center of the macula 52 is the fovea F. In addition, a two-dimensional coordinate system (XY coordinates) is set in a two-dimensional region C in the two-dimensional ophthalmologic image 50 .

[0147] CPU23 inputs the ophthalmic image 50 into the mathematical model, thereby obtaining the probability distribution of the two-dimensional coordinates of the specific part existing in the two-dimensional area in the ophthalmic image 50 as a random variable (S32). As an example, the specific part in the first embodiment is the fovea F. However, it is self-evident that the specific part is not limited to the fovea F. For example, the specific part may also be the optic nerve head, etc. In addition, the specific part may be a point or a part having a certain area or volume. Based on the probability distribution obtained in S32, CPU23 detects the specific part in the two-dimensional ophthalmic image 50 (S33).

[0148] Fig.12 An example of a graph showing the probability distribution of the coordinates of a specific part obtained from a two-dimensional coordinate system as a random variable is shown. Fig.12 In the example shown, the X-axis and the Y-axis are random variables, and the axis orthogonal to the X-axis and the Y-axis is the probability of the random variable, and the random variable is the coordinate of the fovea F in the two-dimensional region C. Fig.12From the graph shown, it can be determined that the coordinates of the point with the highest probability of the presence of the fovea F among the points in the two-dimensional area C are (X', Y'). In addition, even when the structure of the tissue is damaged due to the influence of a disease, or when the image quality of at least a part of the ophthalmic image is poor, the probability distribution output by the mathematical model is likely to form a peak at any position. Therefore, a specific part can be appropriately and directly detected.

[0149] In addition, in the specific part detection process, the specific part can also be detected from the three-dimensional ophthalmic image. In this case, a three-dimensional ophthalmic image (for example, a three-dimensional tomographic image taken by an OCT device, etc.) is obtained in S31. In S32, a probability distribution of the three-dimensional coordinates of the specific part existing in the three-dimensional region in the three-dimensional ophthalmic image as a random variable is obtained. In S33, based on the probability distribution obtained in S32, the specific part in the three-dimensional ophthalmic image is detected.

[0150] Next, refer to Figures 13 to 17 , the ophthalmologic image processing according to the second embodiment will be described. In addition, the description of the same processing as the ophthalmologic image processing according to the first embodiment will be omitted or simplified.

[0151] First, the CPU 23 obtains a three-dimensional tomographic image of the tissue of the eye to be inspected (the fundus in the second embodiment) in the same manner as in S11 of the boundary detection process as an example of the ophthalmic image processing involved in the first embodiment (S31). Then, in the same manner as in S12 of the boundary detection process, the CPU 23 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images constituting the obtained three-dimensional tomographic image (S32).

[0152] The CPU 23 inputs the T-th two-dimensional tomographic image to the mathematical model, thereby obtaining the probability distribution of the coordinates of the M-th (initial value of M is "1") boundary in each of the plurality of one-dimensional regions A1 to AN as a probability distribution for identifying tissues (S34). Figure 7 and Fig.14 An example of a graph showing the probability distribution of coordinates where the boundary Bi exists, obtained from the one-dimensional coordinate A1, is shown. Figure 7 and Fig.14 In the example shown, the one-dimensional coordinates of the one-dimensional area A1 are used as random variables, and the probability distribution of the coordinates of the boundary Bi is shown. Figure 7 and Fig.14 In the example shown, the horizontal axis represents a random variable, and the vertical axis represents the probability of the random variable, and the random variable is the coordinate of the existence boundary Bi in the one-dimensional area A1. In S34, the probability distribution in each of the plurality of one-dimensional areas A1 to AN is obtained.

[0153] In the second embodiment, Figure 7 The probability distribution shown is an example of a probability distribution output when the structural abnormality of the tissue (specifically, the tissue near the boundary Bi) is low. At a location where the structural abnormality is low, it is easy to accurately identify the tissue using a mathematical model, so the probability of the location of the tissue is likely to be biased. Figure 7 From the graph shown, it can be determined that among the points on the one-dimensional area A1, the point with the highest probability of the existence of the boundary Bi is point P. The probability distribution when the mathematical model accurately identifies the tissue (i.e., the ideal probability distribution) takes the value of 1 at only one point on the one-dimensional area A1 and takes the value of 0 at other points.

[0154] on the other hand, Fig.14 The probability distribution shown is an example of a probability distribution output when the abnormality of the structure of the tissue is high. Fig.14 As shown, the probability distribution is less likely to be biased at a position where the abnormality of the structure is high. As described above, the bias of the probability distribution for identifying the tissue changes depending on the abnormality of the structure of the tissue.

[0155] Next, the CPU 23 obtains the deviation of the probability distribution P associated with the Mth boundary (S35). The so-called deviation refers to the difference between the probability distribution P obtained in S14 and the probability distribution P when the tissue is accurately identified. In the second embodiment, the deviation is obtained as structural information indicating the abnormality of the structure of the tissue. In S35 of the second embodiment, the deviation is obtained (calculated) for each of the multiple probability distributions P obtained for the multiple one-dimensional areas A1 to AN.

[0156] In a second embodiment, the entropy of the probability distribution P is calculated as the degree of deviation. The entropy is given by the following (Mathematical Formula 1). The entropy H(P) takes a value of 0≤H(P)≤log(number of events), and the more biased the probability distribution P is, the smaller its value is. That is, the smaller the entropy H(P), the lower the abnormality of the structure of the tissue. The entropy of the probability distribution when the tissue is accurately identified is 0. In addition, the more the abnormality of the structure of the tissue increases, the more difficult it is to identify the tissue, and the larger the entropy H(P). Therefore, by using the entropy H(P) of the probability distribution P as the degree of deviation, the abnormality of the structure of the tissue can be appropriately quantified.

[0157] H(P)=-∑plog(p)……(Mathematical formula 1)

[0158] However, a value other than entropy may be used as the deviation. For example, at least one of the standard deviation, coefficient of variation, variance, etc., which represents the dispersion of the obtained probability distribution P, may be used as the deviation. KL divergence, which is a measure of the difference between the probability distributions P, may also be used as the deviation. In addition, the maximum value of the obtained probability distribution P (for example, Figure 7 and Fig.14 The maximum value of the probability illustrated as an example) is used as the deviation. In addition, the difference between the maximum value and the second largest value of the probability distribution P obtained can also be used as the deviation.

[0159] Next, the CPU 23 determines whether the deviations of all the boundaries to be detected in the T-th two-dimensional tomographic image have been obtained (S36). If the deviations of some boundaries have not been obtained (S16: No), "1" is added to the order M of the boundaries (S37), and the process returns to S14 to obtain the deviation of the next boundary (S34, S35). When the deviations of all boundaries have been obtained (S36: Yes), the CPU 23 stores the deviations of the T-th two-dimensional tomographic image in the storage device 24 and displays it on the display device 28 (S39). The CPU 23 obtains (generates in the second embodiment) the structural abnormality graph of the T-th two-dimensional tomographic image and displays it on the display device 28 (S40).

[0160] Reference Fig.15 and Fig.16 , the structural abnormality degree graph 52 is explained. Fig.15 This is an example of a display screen showing a two-dimensional tomographic image 51A with a low structural abnormality, a structural abnormality graph 52A related to the two-dimensional tomographic image 51A, and a deviation table 53A showing the deviation related to the two-dimensional tomographic image 51A. Fig.16 This is an example of a display screen showing a two-dimensional tomographic image 51B having a high degree of structural abnormality, a structural abnormality graph 52B related to the two-dimensional tomographic image 51B, and a deviation table 53B indicating the degree of deviation related to the two-dimensional tomographic image 51B.

[0161] like Fig.15 and Fig.16 As shown, the two-dimensional tomographic image 51 is a two-dimensional image extended in the X direction (left-right direction of the drawing) and the Z direction (up-down direction of the drawing). As described above, the deviation is obtained for each of the multiple axes (in the second embodiment, multiple A scans) extending parallel to the Z direction on the ophthalmic image. Fig. 9 and Fig.10 In the structural abnormality degree graph 52 shown, the horizontal axis is the X axis, and the vertical axis represents the deviation at each position in the X direction.

[0162] As an example, in the structural abnormality chart 52 of the second embodiment, the average value of multiple deviations (entropy in the second embodiment) obtained for each of the multiple boundaries is shown for each position in the X direction. However, the deviation of one boundary can also be represented by the structural abnormality chart 52. In addition, the average value of a specific plurality of boundaries (for example, the boundary of IPL / INL, the boundary of OPL / ONL) can also be represented by the structural abnormality chart 52. In addition, various statistical values ​​other than the average value (for example, median, mode, maximum or minimum value, etc.) can also be used instead of the average value.

[0163] like Fig.15 As shown in FIG. 5A , when the structural abnormality is low in the entire X direction, the deviation represented by the structural abnormality graph 52A is a low value in the entire X direction. Fig.16 As shown, at the position in the X direction where the structural abnormality is high, the deviation indicated by the structural abnormality graph 52B is a high value. As described above, the structural abnormality graph 52 allows the user to appropriately understand which position in the X direction has a high abnormality.

[0164] Reference Fig.15 and Fig.16 , an example of a method for displaying the degree of deviation is described below. Fig.15 and Fig.16 As shown, in the deviation table 53 of the present embodiment, the obtained deviation (entropy in the present embodiment) is displayed for each of the multiple boundaries. Therefore, the user can properly grasp the boundary with high structural abnormality based on the quantified value. The deviation displayed in the deviation table 53 of the second embodiment is the average value of multiple deviations obtained for each of the multiple one-dimensional regions (A scans in the present embodiment). In addition, in the deviation table 53 of the present embodiment, the average value of the deviations related to all boundaries is displayed. Therefore, the user can easily grasp whether there is a part with high structural abnormality in the tissue shown in the ophthalmic image based on the average value. Moreover, in the deviation table 53 of the second embodiment, the average value of the deviations related to multiple specific boundaries in all boundaries is displayed. As an example, in the second embodiment, the average value of the boundary (IPL / INL boundary, OPL / ONL boundary) where the structure is easily destroyed due to the influence of the disease is displayed. Therefore, the user can easily grasp whether there is structural abnormality caused by the disease. In addition, as described above, various statistical values ​​other than the average value can also be used.

[0165] Next, the CPU 23 determines whether the deviations of all the two-dimensional tomographic images constituting the three-dimensional tomographic image have been obtained (S41). If the deviations of some two-dimensional tomographic images have not been obtained (S41: No), the order T of the two-dimensional tomographic images is incremented by "1" (S42), and the process returns to S12 to obtain the deviation of the next two-dimensional tomographic image (S32 to S40). When the deviations of all the two-dimensional tomographic images have been obtained (S41: Yes), the CPU 23 obtains (generates in the second embodiment) a structural abnormality map and displays it on the display device 28 (S44).

[0166] Reference Fig.17 , the structural abnormality map 55 is explained. The structural abnormality map 55 is a map representing the two-dimensional distribution of the deviation in the tissue. As described above, in the second embodiment, the deviation is obtained for each of the multiple two-dimensional tomographic images that constitute the three-dimensional tomographic image. That is, the overall deviation of the tissue shown in the three-dimensional tomographic image is obtained. In the structural abnormality map 55 of the second embodiment, the two-dimensional distribution of the deviation when the tissue is observed from the front (in the second embodiment, the fundus) is shown. However, the direction representing the two-dimensional distribution can be appropriately changed. In addition, in the structural abnormality map 55 of the second embodiment, the higher the deviation, the lower the brightness displayed. However, the specific method used to represent the deviation of each position in the structural abnormality map 55 can also be appropriately changed. In addition, in the structural abnormality map 55 of the second embodiment, the average value of the deviation associated with multiple boundaries is used. However, the deviation associated with a specific boundary can also be used.

[0167] Next, when there is a part (hereinafter referred to as "abnormal part") whose deviation is greater than or equal to a threshold value in the tissue displayed in the ophthalmic image (three-dimensional tomographic image in the second embodiment), the CPU 23 displays the tomographic image (at least one of the two-dimensional tomographic image and the three-dimensional tomographic image) or the enlarged image of the abnormal part on the display device 28 (S45). Therefore, the image of the abnormal part is appropriately confirmed by the user.

[0168] In detail, the CPU 23 of the second embodiment causes the ophthalmic image with the highest deviation or the ophthalmic image with a deviation greater than a threshold value to be displayed on the display device 28 among the multiple ophthalmic images (in the present embodiment, multiple two-dimensional tomographic images constituting a three-dimensional tomographic image) obtained by photographing the tissue of the same eye to be examined. For example, the CPU 23 may also cause the ophthalmic image with a high deviation among the multiple ophthalmic images to be displayed on a photographing confirmation screen for the user to confirm the photographed ophthalmic image. In addition, the CPU 23 may also cause the ophthalmic image with a high deviation among the multiple ophthalmic images to be displayed when starting a viewer for displaying the photographed ophthalmic image. In this case, the user can easily confirm that the ophthalmic image of the part with a high degree of structural abnormality has been photographed. This processing can also be performed in the same manner when the ophthalmic image photographing device 11B is performing ophthalmic image processing.

[0169] In addition, when the ophthalmic image capturing device 11B is performing ophthalmic image processing, the CPU 13B of the ophthalmic image capturing device 11B outputs a shooting instruction (S25) for capturing an image of a portion having a deviation degree greater than a threshold value (at least one of a two-dimensional tomographic image, a three-dimensional tomographic image, and a motion contrast image in the second embodiment). In addition, when capturing an image of a portion having a deviation degree greater than a threshold value, the CPU 13B may capture ophthalmic images of the same portion multiple times and obtain an additive average image of the captured multiple ophthalmic images. In this case, an ophthalmic image of a portion having a high degree of structural abnormality can be obtained with high quality. In addition, the ophthalmic image processing device 23 may output a shooting instruction to the ophthalmic image capturing device 11B.

[0170] Next, CPU23 performs follow-up processing (S46). Follow-up processing is a process for allowing the user to observe the eye to be inspected. As described above, the obtained deviation is stored in the storage device 24 (S39). In the follow-up processing, CPU23 displays multiple deviations associated with each of multiple ophthalmic images obtained by photographing the tissue of the same eye to be inspected at different times on the display device 28. Therefore, by comparing multiple deviations, the user can appropriately grasp the development status of structural abnormalities, etc. In addition, CPU23 can display multiple deviation values ​​themselves (for example, deviation table 53), can display multiple structural abnormality charts 52, and can display multiple structural abnormality maps 55.

[0171] Next, CPU23 generates image quality evaluation information (S47) for evaluating the image quality of the ophthalmic image based on the deviation obtained for the ophthalmic image. Sometimes, when the image quality of the ophthalmic image is poor, the deviation also becomes high. Therefore, by generating image quality evaluation information based on the deviation, the image quality of the ophthalmic image can be properly grasped. The specific method for generating image quality evaluation information can be appropriately selected. As an example, in the second embodiment, CPU23, i.e., the control unit, obtains an index (e.g., SSI (Signal Strength Index) or QI (Quality Index)) representing the strength of the signal of the ophthalmic image or the goodness of the signal. When the obtained index is below the threshold and the average value of the deviation is above the threshold, CPU23 generates image quality evaluation information indicating poor image quality. In addition, CPU23 can also generate image quality evaluation information indicating poor image quality when the deviations of all parts of the ophthalmic image are all high (e.g., when the deviations of all parts are all above the threshold, etc.). In addition, at least any one of the deviation value itself, the structural abnormality degree graph 52 and the structural abnormality degree map 55 may be used as the image quality evaluation information.

[0172] The technology disclosed in the above embodiment is only an example. Therefore, the technology exemplified in the above embodiment can also be changed. First, only part of the multiple technologies exemplified in the above embodiment can also be executed.

[0173] For example, in the first embodiment, the ophthalmologic image processing device 21 may perform only the boundary detection process (see Figure 5 ) and specific part detection processing (refer to Fig.10 ) in the case of a party.

[0174] For example, in the second embodiment, the ophthalmic image processing device 21 may only perform the process of obtaining the deviation of the two-dimensional tomographic image (S31 to S40), and omit the process of obtaining the deviation of the entire three-dimensional tomographic image. In this case, the ophthalmic image obtained in S31 may also be a two-dimensional tomographic image. In addition, the ophthalmic image processing device 21 may use the deviation as image quality evaluation information instead of using the deviation as structural information indicating the abnormality of the tissue structure.

[0175] In addition, in the second embodiment, the method of using the obtained deviation can also be changed. For example, the CPU 23 can also input the two-dimensional tomographic image with the highest deviation or the two-dimensional tomographic image with a deviation greater than a threshold value among the multiple two-dimensional tomographic images constituting the three-dimensional tomographic image into a mathematical model, and the mathematical model outputs an automatic diagnosis result related to the disease of the eye to be inspected. The mathematical model is pre-trained by a machine learning algorithm to output an automatic diagnosis result based on the input ophthalmic image. In this case, the two-dimensional tomographic image with a high degree of structural abnormality among the multiple two-dimensional tomographic images constituting the three-dimensional tomographic image can be used to efficiently obtain the automatic diagnosis result.

[0176] In addition, regarding the first aspect, the second aspect, and the third aspect, Figure 2 The process of obtaining the training ophthalmic image in S1 is an example of the "input training data acquisition step". Figure 2 The process of obtaining training data in S2 is an example of the "output training data acquisition step". Figure 2 The process of training the mathematical model in S3 is an example of a "training step". Figure 5 S11 and Fig.10 The process of acquiring an ophthalmic image in S31 is an example of an "image acquisition step". Figure 5 S14 and Fig.10 The process of obtaining the probability distribution in S32 is an example of the "probability distribution obtaining step". Figure 5 S16, S24 and Fig.10 The process of detecting a boundary or a specific part in S33 is an example of a "detection step". Figure 5 The process of obtaining the probability map in S15 is an example of the "mapping acquisition step". Figure 5 The process of obtaining the Enface image in S25 is an example of the “Enface image obtaining step”.

[0177] Regarding the fourth, fifth and sixth methods, Fig.13 The process of acquiring an ophthalmic image in S21 is an example of an "image acquisition step". Fig.13 The process of obtaining the probability distribution in S24 is an example of the "probability distribution obtaining step". Fig.13 The process of obtaining the degree of deviation as structural information in S25 is an example of the “structural information obtaining step”.

Claims

1. An ophthalmic image processing device for processing an ophthalmic image which is an image of tissue of an eye to be inspected, characterized in that: The control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device, and obtains a probability distribution for identifying tissues in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, and obtains a deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality in the structure of the tissue. The abnormality of the structure is quantified by the deviation.

2. The ophthalmic image processing device according to claim 1, characterized in that: The mathematical model is trained using a training data set having an input side as data of an ophthalmologic image of a tissue of a subject's eye captured in the past and an output side as data representing the tissue in the ophthalmologic image on the input side.

3. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The deviation includes the obtained entropy of the probability distribution.

4. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The ophthalmic image is a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue, The control unit inputs the ophthalmic image into the mathematical model, thereby obtaining the probability distribution for identifying one or more layers or boundaries included in a plurality of layers and boundaries of the layers in the ophthalmic image, and obtaining the degree of deviation for the one or more layers or boundaries.

5. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The control unit acquires at least one of a graph and a map indicating the magnitude of the deviation with respect to the position in the tissue.

6. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The control unit performs at least one of outputting a photographing instruction to photograph the portion of the tissue where the deviation is greater than a threshold to the ophthalmic image photographing device, and displaying a tomographic image or an enlarged image of the portion where the deviation is greater than the threshold on a display device.

7. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The control unit stores the acquired deviation in a storage device, and displays a plurality of deviations associated with each of a plurality of ophthalmologic images obtained by imaging the tissue of the same subject's eye at different times on a display device.

8. The ophthalmic image processing device according to claim 1 or 2, characterized in that: The control unit generates image quality evaluation information for evaluating image quality of the ophthalmologic image based on the deviation degree acquired with respect to the ophthalmologic image.

9. An OCT apparatus for capturing an ophthalmic image of a tissue of an eye to be inspected by processing an OCT signal generated by using reference light and reflected light of a measurement light irradiated to the tissue of the eye to be inspected, characterized in that: The control unit of the OCT device inputs the captured ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtaining a probability distribution for identifying tissues in the ophthalmic image, and obtains a deviation of the obtained probability distribution from the probability distribution when the tissue is accurately identified as structural information representing the degree of abnormality of the structure of the tissue. The abnormality of the structure is quantified by the deviation.

10. A computer program product comprising an ophthalmologic image processing program, the ophthalmologic image processing program being executed by an ophthalmologic image processing device that processes an ophthalmologic image that is an image of tissue of an eye to be inspected, characterized in that: The ophthalmologic image processing program is executed by the control unit of the ophthalmologic image processing device, thereby causing the ophthalmologic image processing device to execute the following steps: An image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a probability distribution acquisition step, wherein the ophthalmic image is input into a mathematical model trained by a machine learning algorithm, thereby acquiring a probability distribution for identifying tissues in the ophthalmic image; and a structural information acquisition step of acquiring a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality of the structure of the tissue, The abnormality of the structure is quantified by the deviation.

Images