Ophthalmic image processing apparatus, OCT apparatus, and computer program product

Abstract

The invention provides an ophthalmic image processing apparatus, an OCT apparatus, and a computer program product. A control unit of an ophthalmic image processing device acquires an ophthalmic image captured by an ophthalmic image capturing device (S11). A control unit inputs an ophthalmic image into a mathematical model trained by a machine learning algorithm, thereby acquiring a probability distribution in which coordinates of at least one of a specific boundary and a specific site in which tissue is present in a region in the ophthalmic image are used as random variables (S14). The control unit detects at least one of a specific boundary and a specific site on the basis of the acquired probability distribution (S16, S24).

Description

[0001] This application is a divisional application of an application with an international filing date of April 15, 2019, an international application number of PCT / JP2019 / 016209, a national application number of 201980051471.6, and an invention title of "Ophthalmic Image Processing Device, OCT Device, and Ophthalmic Image Processing Program". Technical Field

[0002] The present disclosure relates to an ophthalmic image processing device that processes ophthalmic images of an eye to be examined, an OCT device, and an ophthalmic image processing program executed in the ophthalmic image processing device. Background Art

[0003] Conventionally, various techniques have been proposed for detecting at least one of the boundaries of a plurality of tissues (e.g., a plurality of layers) shown in an ophthalmic image and specific sites on the tissues shown in the ophthalmic image (hereinafter, sometimes simply referred to as "boundary / specific site"). For example, in the technique disclosed in Non-Patent Document 1, first, for each pixel, it is mapped to which layer each pixel belongs. Then, by outputting the thickness of each layer based on the mapping result, the boundaries of each layer are detected.

[0004] As another example, various techniques have been proposed for estimating abnormalities in the structure of an object shown in an image. For example, in the technique disclosed in Non-Patent Document 2, first, a normal image is used as training data to train a generative adversarial network (GAN). The trained GAN learns the case of mapping the input image to coordinates in the latent space. If there is an abnormality in the structure of the input image, a difference occurs between the image generated by the mapping and the input image. In the technique of Non-Patent Document 2, the error between the image generated by the mapping and the input image is taken to attempt to estimate the abnormal part of the structure.

[0005] Prior Art Documents

[0006] Non-Patent Documents

[0007] Non-Patent Document 1: Yufan He, Aaron Carass, et al. "Topology guaranteed segmentation of the human retina from OCT using convolutional neural networks." arXiv:1803.05120, 14 Mar 2018

[0008] Non-Patent Document 2: Thomas Schlegl, et al. “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery” arXiv:1703.05921, 17 Mar 2017 Summary of the Invention

[0009] In the method of Non-Patent Document 1, for example, when the structure of the tissue is damaged due to the influence of a disease, it is difficult to map with high precision which tissue each pixel belongs to. As a result, the detection accuracy of the boundary of the tissue / specific part also decreases.

[0010] In the method of Non-Patent Document 2, the error between the image generated by mapping and the input image is taken to attempt to estimate the abnormal part, so it is difficult to quantify the degree of abnormality of the structure. Therefore, it is difficult for the user to appropriately judge the abnormality of the structure of the tissue shown in the ophthalmic image.

[0011] A typical object of the present disclosure is to provide an ophthalmic image processing apparatus, an OCT apparatus, and an ophthalmic image processing program that can appropriately detect at least one of the boundary and specific part of the tissue shown in the ophthalmic image.

[0012] Another typical object of the present disclosure is to provide an ophthalmic image processing apparatus, an OCT apparatus, and an ophthalmic image processing program that can enable a user to appropriately judge the abnormality of the structure of the tissue shown in the ophthalmic image.

[0013] The ophthalmic image processing apparatus provided by the first aspect of the typical embodiment of the present disclosure processes an ophthalmic image that is an image of the tissue of an eye to be examined. The control unit of the ophthalmic image processing apparatus acquires the ophthalmic image captured by the ophthalmic image capturing device, and by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtains a probability distribution in which the coordinates of at least one of a specific boundary and a specific part where the tissue exists in the region in the ophthalmic image are used as random variables, and based on the obtained probability distribution, detects at least one of the specific boundary and the specific part.

[0014] The OCT device provided by the second aspect of the typical embodiment of the present disclosure captures an ophthalmic image of the tissue by processing an OCT signal generated from the reflected light of the reference light and the measurement light irradiated onto the tissue of the eye to be examined. The control unit of the OCT device inputs the captured ophthalmic image into a mathematical model obtained by training using a machine learning algorithm, thereby obtaining a probability distribution with the coordinates of at least one of the specific boundaries and specific parts of the tissue existing in the region in the ophthalmic image as random variables, and detecting at least one of the specific boundaries and the specific parts based on the obtained probability distribution.

[0015] The ophthalmic image processing program provided by the third aspect of the typical embodiment of the present disclosure is executed by an ophthalmic image processing device that processes an ophthalmic image, which is an image of the tissue of the eye to be examined. By being executed by the control unit of the ophthalmic image processing device through the ophthalmic image processing program, the ophthalmic image processing device is caused to execute the following steps: an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a probability distribution acquisition step of obtaining a probability distribution with the coordinates of at least one of the specific boundaries and specific parts of the tissue existing in the region in the ophthalmic image as random variables by inputting the ophthalmic image into a mathematical model obtained by training using a machine learning algorithm; and a detection step of detecting at least one of the specific boundaries and the specific parts based on the obtained probability distribution.

[0016] According to the ophthalmic image processing device according to the first aspect, the OCT device according to the second aspect, and the ophthalmic image processing program according to the third aspect, at least one of the boundaries and specific parts of the tissue shown in the ophthalmic image can be appropriately detected.

[0017] The ophthalmic image processing device provided by the fourth aspect of the typical embodiment of the present disclosure processes an ophthalmic image, which is an image of the tissue of the eye to be examined. The control unit of the ophthalmic image processing device acquires an ophthalmic image captured by an ophthalmic image capturing device, obtains a probability distribution for identifying the tissue in the ophthalmic image by inputting the ophthalmic image into a mathematical model obtained by training using a machine learning algorithm, and obtains the degree of deviation of the obtained probability distribution from the probability distribution in the case of accurately identifying the tissue as structure information indicating the degree of abnormality of the structure of the tissue.

[0018] The OCT device provided by the fifth aspect of the typical embodiment of the present disclosure captures an ophthalmic image of the tissue by processing an OCT signal generated from the reflected light of the reference light and the measurement light irradiated to the tissue of the eye to be examined. The control unit of the OCT device inputs the captured ophthalmic image into a mathematical model obtained by training using a machine learning algorithm, thereby obtaining a probability distribution for identifying the tissue in the ophthalmic image, and taking the degree of deviation of the obtained probability distribution from the probability distribution in the case of accurately identifying the tissue as structural information indicating the degree of abnormality of the structure of the tissue.

[0019] The ophthalmic image processing program provided by the sixth aspect of the typical embodiment of the present disclosure is executed by an ophthalmic image processing device that processes an ophthalmic image, which is an image of the tissue of the eye to be examined. By being executed by the control unit of the ophthalmic image processing device through the ophthalmic image processing program, the ophthalmic image processing device is caused to execute the following steps: an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a probability distribution acquisition step of obtaining a probability distribution for identifying the tissue in the ophthalmic image by inputting the ophthalmic image into a mathematical model obtained by training using a machine learning algorithm; and a structural information acquisition step of taking the degree of deviation of the obtained probability distribution from the probability distribution in the case of accurately identifying the tissue as structural information indicating the degree of abnormality of the structure of the tissue.

[0020] According to the ophthalmic image processing device according to the fourth aspect, the OCT device according to the fifth aspect, and the ophthalmic image processing program according to the sixth aspect, a user can appropriately judge the abnormality of the tissue shown in the ophthalmic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a block diagram showing the schematic configuration of the mathematical model construction device 1, the ophthalmic image processing device 21, and the ophthalmic image capturing devices 11A and 11B.

[0022] Figure 2 It is a flowchart of the mathematical model construction process executed by the mathematical model construction device 1.

[0023] Figure 3 It is a diagram showing an example of the training ophthalmic image 30.

[0024] Figure 4 It is a diagram showing an example of the training data 31.

[0025] Figure 5 It is a flowchart of the boundary detection process executed by the ophthalmic image processing device 21.

[0026] Figure 6It is a diagram schematically showing the relationship between the two-dimensional tomographic image 40 input to the mathematical model and the one-dimensional regions A1 to AN in the two-dimensional tomographic image 40.

[0027] Figure 7 It is an example of a graph showing the probability distribution of the coordinates of the boundary Bi where the one-dimensional coordinates of the one-dimensional region A1 are used as random variables.

[0028] Figure 8 It is a diagram showing an example of a probability map of the boundary Bi as the inner limiting membrane (ILM).

[0029] Figure 9 It is a diagram showing an example of a probability map of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL).

[0030] Figure 10 It is a flowchart of the specific site detection process executed by the ophthalmic image processing device 21.

[0031] Figure 11 It is a diagram schematically showing the relationship between the ophthalmic image 50 input to the mathematical model and the coordinate system C of the two-dimensional coordinates in the ophthalmic image 50.

[0032] Figure 12 It is an example of a graph showing the probability distribution of the coordinates of the specific site where the two-dimensional coordinates are used as random variables.

[0033] Figure 13 It is a flowchart of the ophthalmic image processing related to the second embodiment executed by the ophthalmic image processing device 21.

[0034] Figure 14 It is an example of a graph showing the probability distribution for identifying the boundary Bi when the abnormality degree of the structure near the boundary Bi is high.

[0035] Figure 15 It is an example of a display screen showing the two-dimensional tomographic image 51A, the structure abnormality degree graph 52A, and the deviation degree table 53A.

[0036] Figure 16 It is an example of a display screen showing the two-dimensional tomographic image 51B, the structure abnormality degree graph 52B, and the deviation degree table 53B.

[0037] Figure 17 It is a diagram showing an example of the structure abnormality degree map 55. Detailed implementation mode

[0038] <Summary>

[0039] In one example of the present disclosure, the control unit of the ophthalmic image processing device acquires an ophthalmic image captured by the ophthalmic image capturing device. By inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, the control unit thereby obtains a probability distribution having as random variables the coordinates of at least any one of a specific boundary and a specific part existing within a region in the ophthalmic image. Based on the obtained probability distribution, the control unit detects at least any one of the specific boundary and the specific part.

[0040] According to the ophthalmic image processing device exemplified in one example of the present disclosure, for example, even when the structure of the tissue is damaged due to the influence of a disease, or when the image quality of at least a part of the ophthalmic image is poor, etc., it is possible to appropriately and directly detect the boundary / specific part based on the probability distribution having the coordinates as random variables.

[0041] In addition, in one example of the above prior art, after mapping which layer each pixel belongs to, the thickness of the layer is output based on the mapping result, and finally the boundary of the layer is detected. In this case, since multiple stages of processing are required, it is difficult to reduce the processing amount. In contrast, in the ophthalmic image processing device exemplified in one example of the present disclosure, the processing amount for detecting the boundary / specific part based on the probability distribution having the coordinates as random variables is small. Therefore, the processing amount can also be easily reduced.

[0042] In addition, the aforementioned "region" can be any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region. When the "region" is a one-dimensional region, the coordinate is a one-dimensional coordinate. Similarly, when the "region" is a two-dimensional region, the coordinate is a two-dimensional coordinate, and when the "region" is a three-dimensional region, the coordinate is a three-dimensional coordinate.

[0043] In addition, the "specific boundary" that is the object for obtaining the probability distribution can be one boundary or multiple boundaries. When obtaining the probability distribution of multiple boundaries, the control unit can obtain the probability distribution for each of the multiple boundaries separately. Similarly, the number of "specific parts" that are the objects for obtaining the probability distribution can also be one or multiple.

[0044] The processing for detecting the boundary / specific part based on the probability distribution can also utilize a mathematical model in the same manner as the processing for obtaining the probability distribution. In addition, the control unit itself can also detect the boundary / specific part based on the obtained probability distribution without using a mathematical model.

[0045] The mathematical model can be trained using a training data set that sets the input side to data of an ophthalmic image of the tissue of the eye to be examined taken in the past, and sets the output side to data representing the position of at least one of a specific boundary and a specific part of the tissue in the ophthalmic image on the input side. In this case, the trained mathematical model can appropriately output the probability distribution of a specific boundary / specific part with coordinates as random variables by inputting an ophthalmic image.

[0046] In addition, the mathematical model construction method executed by the mathematical model construction device for constructing the mathematical model can be expressed as follows. A mathematical model construction method, executed by the mathematical model construction device for constructing the mathematical model, the mathematical model being constructed by being trained using a machine learning algorithm and outputting data corresponding to the input ophthalmic image by inputting the ophthalmic image, the mathematical model construction method including the following steps: an output training data acquisition step of acquiring an ophthalmic image of the tissue of the eye to be examined as a training ophthalmic image; an output training data acquisition step of acquiring training data that represents the position of at least one of a specific boundary and a specific part of the tissue in the training ophthalmic image acquired in the input training data acquisition step; a training step of training the mathematical model by using the data of the training ophthalmic image as input training data and the training data as output training data, thereby constructing a mathematical model that outputs the probability distribution of the coordinates of at least one of a specific boundary and a specific part of the tissue existing in the region in the input ophthalmic image as random variables.

[0047] The control unit can also obtain the probability distribution of the one-dimensional coordinates where a specific boundary exists in a one-dimensional region extending in a direction intersecting the specific boundary of the tissue in the ophthalmic image by inputting the ophthalmic image into the mathematical model. The control unit can also detect a specific boundary based on the obtained probability distribution. In this case, since it is easy to obtain a properly biased probability distribution, it is easy to improve the detection accuracy of the specific boundary.

[0048] In addition, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image captured by an OCT device. In this case, for a one-dimensional region extending along the optical axis direction of the measurement light of the OCT device (the so-called "A-scan direction"), a probability distribution with the one-dimensional coordinate as a random variable can be obtained. The direction of the boundary of the tissue is likely to be close to perpendicular to the A-scan direction. Therefore, by taking the direction in which the one-dimensional region extends as the A-scan direction, the probability distribution is likely to be appropriately biased. In addition, the possibility that there are two or more intersections of the one-dimensional region and a specific boundary is also reduced. Therefore, the detection accuracy of a specific boundary is improved. In addition, the control unit may also obtain a probability distribution with the one-dimensional coordinate as a random variable for a one-dimensional region extending perpendicular to a specific boundary. In this case, the probability distribution is more likely to be biased.

[0049] The control unit may also detect a two-dimensional or three-dimensional boundary based on the multiple probability distributions obtained for each of the multiple one-dimensional regions that are different from each other. In this case, the two-dimensional or three-dimensional boundary can be appropriately grasped.

[0050] In addition, the two-dimensional or three-dimensional boundary may be detected after post-processing the probability distributions related to each of the multiple one-dimensional regions. For example, the boundary on the Nth one-dimensional region may be detected based on the probability distribution on the Nth one-dimensional region and the probability distributions on the one-dimensional regions near the Nth one-dimensional region. In this case, for example, known methods such as graph cut can be adopted. In addition, the two-dimensional or three-dimensional boundary may also be detected by using the theory of shortest path search for finding the path with the minimum weight.

[0051] The control unit may also obtain a two-dimensional or three-dimensional map representing the possibility of a specific boundary generated based on the multiple probability distributions obtained for each of the multiple one-dimensional regions. That is, the ophthalmic image processing device may also execute a map acquisition step of acquiring a two-dimensional or three-dimensional map representing the possibility of a specific boundary. In this case, the position of the two-dimensional or three-dimensional specific boundary can be appropriately grasped based on the map.

[0052] In addition, the control unit may also display the acquired map on the display device. In this case, the user can appropriately grasp the position of the two-dimensional or three-dimensional specific boundary by viewing the displayed map. The map may be generated by the control unit based on the multiple probability distributions output from the mathematical model. In addition, the map may also be output by the mathematical model.

[0053] The ophthalmic image may also be a three-dimensional tomographic image captured by an OCT device. The control unit can obtain a two-dimensional frontal image (so-called "Enface image") when observing a specific layer included in the three-dimensional tomographic image in the direction along the optical axis of the measurement light based on the three-dimensional tomographic image and the detected three-dimensional boundary. That is, the ophthalmic image processing device can perform an Enface image acquisition step of acquiring an Enface image based on the three-dimensional tomographic image and the detected three-dimensional boundary. In this case, after determining a specific layer based on the appropriately detected boundary, an Enface image of the specific layer can be appropriately obtained. The Enface image can be generated by the control unit. In addition, the Enface image can also be output by a mathematical model.

[0054] In addition, the control unit can also obtain a thickness map representing the thickness distribution of a specific layer included in the three-dimensional tomographic image in a two-dimensional manner based on the three-dimensional tomographic image and the detected three-dimensional boundary. In this case, after determining a specific layer based on the appropriately detected boundary, a more accurate thickness map can be obtained.

[0055] The control unit can also obtain a probability distribution in which the probability of the presence of a specific part in a two-dimensional or more region in the ophthalmic image and two-dimensional or more coordinates are random variables by inputting the ophthalmic image into a mathematical model. The control unit can detect a specific part based on the obtained probability distribution. In this case, a specific part can be appropriately detected in a two-dimensional region or a three-dimensional region.

[0056] In another example of the present disclosure, the control unit of the ophthalmic image processing device obtains an ophthalmic image captured by an ophthalmic image capturing device. The control unit obtains a probability distribution for identifying tissues in the ophthalmic image by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm. The control unit obtains the degree of deviation of the obtained probability distribution from the probability distribution in the case of accurately identifying tissues as structural information indicating the abnormality of the structure of the tissues.

[0057] In another example of the present disclosure, when the structure of the tissue is normal, it is easy to accurately identify the tissue through a mathematical model, so the obtained probability distribution is likely to be biased. On the other hand, when the structure of the tissue is abnormal, the obtained probability distribution is less likely to be biased. Therefore, the degree of deviation between the probability distribution in the case of accurately identifying the tissue and the actually obtained probability distribution increases or decreases according to the degree of abnormality of the structure. Therefore, according to the ophthalmic image processing apparatus according to another example of the present disclosure, the degree of abnormality of the structure is appropriately quantified by the degree of deviation. In addition, when training the mathematical model, even if a large number of ophthalmic images of tissues with abnormalities are not used, the degree of abnormality of the structure of the tissue can be grasped according to the degree of deviation. Thus, the user can appropriately judge the abnormality of the structure of the tissue shown in the ophthalmic image.

[0058] In addition, the degree of deviation can be output by the mathematical model. Alternatively, the control unit may calculate the degree of deviation based on the probability distribution output by the mathematical model.

[0059] In another example of the present disclosure, the mathematical model can be trained using a training data set that sets the input side as data of ophthalmic images of tissues of an eye to be examined taken in the past, and sets the output side as data representing the tissues in the input ophthalmic images. In this case, the trained mathematical model can appropriately output a probability distribution for identifying the tissue by inputting an ophthalmic image.

[0060] In addition, in another example of the present disclosure, the specific manner of the mathematical model that outputs the probability distribution can be appropriately selected. For example, the mathematical model can output a probability distribution having as a random variable the coordinates of at least any one of a specific boundary and a specific part where a tissue exists in a region in the input ophthalmic image. In this case, based on the probability distribution output by the mathematical model, the tissue (in this case, at least any one of the boundary and the specific position) in the ophthalmic image can be appropriately and directly identified. In addition, in this case, the "specific boundary" as the object for obtaining the probability distribution can be one boundary or multiple boundaries. When obtaining the probability distributions of multiple boundaries, the control unit can obtain the probability distribution for each of the multiple boundaries separately. Similarly, the number of "specific parts" as the object for obtaining the probability distribution can be one or multiple. In addition, the region in the ophthalmic image as the unit for obtaining the probability distribution can be any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region. The dimension of the coordinates as the random variable can be the same as the dimension of the region as the unit for obtaining the probability distribution. In addition, the mathematical model can output a probability distribution having as a random variable the type of each tissue in the eye to be examined for each region (for example, each pixel) of the input ophthalmic image. In this case, the tissue in the ophthalmic image can also be identified based on the probability distribution output by the mathematical model.

[0061] The degree of deviation may include the entropy (average information amount) of the obtained probability distribution. Entropy represents the degree of uncertainty, randomness, and disorder. In the present disclosure, the entropy of the probability distribution output when an organization is accurately identified is 0. Additionally, as the degree of abnormality of the structure of the organization increases, the identification of the organization becomes more difficult and the entropy increases. Therefore, by using the entropy of the probability distribution as the degree of deviation, the degree of abnormality of the structure of the organization can be more appropriately quantified.

[0062] However, a value other than entropy may also be adopted as the degree of deviation. For example, at least any one of the standard deviation, coefficient of variation, variance, etc., which represent the dispersion degree of the obtained probability distribution, may be used as the degree of deviation. The KL divergence, etc., which is a scale for realizing the difference between probability distributions, may also be used as the degree of deviation. In addition, the maximum value of the obtained probability distribution may be used as the degree of deviation.

[0063] In another example of the present disclosure, the ophthalmic image may be a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue. The control unit can obtain a probability distribution for identifying one or more layers or boundaries included in multiple layers and layer boundaries in the ophthalmic image by inputting the ophthalmic image into a mathematical model. The control unit can obtain the degree of deviation for the one or more layers or boundaries. In this case, the degree of abnormality of the structure of a specific layer or boundary can be appropriately grasped based on the degree of deviation. For example, when only the degree of deviation of a layer or boundary that is likely to have a structural abnormality due to the influence of a disease is obtained, the obtained degree of deviation can be more useful information for the disease.

[0064] In another example of the present disclosure, the control unit can obtain at least any one of a chart and a map indicating the magnitude of the degree of deviation for positions in the tissue. The user can appropriately grasp which position in the tissue has a high degree of abnormality based on at least any one of the chart and the map.

[0065] In another example of the present disclosure, for example, the ophthalmic image may be a two-dimensional image extended in the XZ direction. The control unit can, by inputting the ophthalmic image into a mathematical model, obtain, for each of a plurality of one-dimensional regions extending parallel to the Z direction on the ophthalmic image, a probability distribution having a one-dimensional coordinate as a random variable, and obtain the degree of deviation for each of the plurality of one-dimensional regions. The control unit can also obtain a structural abnormality chart indicating the magnitude of the degree of deviation at each position in the X direction based on the degree of deviation obtained for each of the plurality of one-dimensional regions. In this case, the user can appropriately grasp which position in the X direction of the two-dimensional ophthalmic image has a high degree of abnormality based on the structural abnormality chart.

[0066] In this case, the two-dimensional image can be a two-dimensional tomographic image of the tissue or a two-dimensional frontal image of the tissue. In the case of a two-dimensional tomographic image, the degree of deviation obtained for each of the plurality of one-dimensional regions can be obtained for each layer or each boundary, or can be obtained as an average of a plurality of layers and boundaries. In addition, the structural abnormality degree chart can be output by a mathematical model. In addition, the control unit can also generate a structural abnormality degree chart based on the degree of deviation obtained for each of the plurality of axes. The control unit can also cause the obtained structural abnormality degree chart to be displayed on the display device.

[0067] In another example of the present disclosure, the control unit can obtain a structural abnormality map representing the two-dimensional distribution of the degree of deviation in the tissue. In this case, the user can accurately grasp the degree of abnormality of the structure at each position within the two-dimensional region based on the structural abnormality map. In addition, the structural abnormality map can be output by a mathematical model. In addition, the control unit can also generate a structural abnormality map based on the obtained degree of deviation. The control unit can also cause the obtained structural abnormality map to be displayed on the display device.

[0068] In another example of the present disclosure, it is also possible to enable the user to grasp the degree of abnormality of the structure of the tissue without using a chart and a map. For example, the control unit can also notify the user of the obtained degree of deviation itself as structural information indicating the degree of abnormality of the structure.

[0069] In another example of the present disclosure, the control unit can perform a process of outputting a photographing instruction for photographing a part where the degree of deviation in the tissue is equal to or greater than a threshold value to the ophthalmic image photographing device. In addition, the control unit can also perform a process of causing a tomographic image or a magnified image of a part where the degree of deviation is equal to or greater than the threshold value to be displayed on the display device. In this case, the user can appropriately confirm an image of a part with a high degree of abnormality of the structure.

[0070] In another example of the present disclosure, when outputting a photographing instruction for photographing a part where the degree of deviation is equal to or greater than a threshold value, the control unit can output an instruction to photograph the ophthalmic image of the part where the degree of deviation is equal to or greater than the threshold value a plurality of times and obtain an additive average image of the plurality of photographed ophthalmic images. In this case, it is possible to obtain an ophthalmic image of a part with a high degree of abnormality of the structure with high quality.

[0071] In another example of the present disclosure, the control unit may cause an ophthalmic image with the highest deviation degree or an ophthalmic image with a deviation degree equal to or higher than a threshold value among a plurality of ophthalmic images obtained by photographing the tissue of the same eye to be examined to be displayed on the display device. For example, by displaying, on a photographing confirmation screen for allowing the user to confirm the photographed ophthalmic images, an ophthalmic image with a high deviation degree among the plurality of ophthalmic images, the control unit can enable the user to easily confirm the ophthalmic image obtained by photographing a part with a high degree of abnormality in the structure. In addition, when a viewer for allowing the user to confirm the photographed ophthalmic images is started, by displaying an ophthalmic image with a high deviation degree among the plurality of ophthalmic images, the control unit can enable the user to first confirm the ophthalmic image obtained by photographing a part with a high degree of abnormality in the structure.

[0072] In another example of the present disclosure, the control unit may input a two-dimensional tomographic image with the highest deviation degree or a two-dimensional tomographic image with a deviation degree equal to or higher than a threshold value among a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image into a mathematical model, and the mathematical model outputs an automatic diagnosis result related to the disease of the eye to be examined. In this case, it is possible to efficiently obtain an automatic diagnosis result by using a two-dimensional tomographic image with a high degree of abnormality in the structure among the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image.

[0073] In another example of the present disclosure, the control unit may cause the obtained deviation degree to be stored in the storage device. The control unit may also cause a plurality of deviation degrees related to each of a plurality of ophthalmic images obtained by photographing the tissue of the same eye to be examined at different times to be displayed on the display device. In this case, by comparing the plurality of deviation degrees, the user can appropriately grasp the development status of the abnormality in the structure and the like. In addition, the control unit may display the values of the plurality of deviation degrees themselves, or may display the plurality of above-mentioned structure abnormality degree charts side by side. In addition, the control unit may also display the plurality of above-mentioned structure abnormality degree maps side by side.

[0074] In another example of the present disclosure, the control unit may generate image quality evaluation information for evaluating the image quality of an ophthalmic image based on the deviation degree obtained for the ophthalmic image. There are cases where the deviation degree becomes high not only when there is an abnormality in the structure of the tissue but also when the image quality of the ophthalmic image is poor. Therefore, by generating image quality evaluation information based on the deviation degree, it is possible to appropriately grasp the image quality of the ophthalmic image.

[0075] In addition, the specific method for generating the image quality evaluation information can be appropriately selected. For example, the value of the deviation itself can be used as the image quality evaluation information. Additionally, when there is an abnormality in a part of the structure of the tissue, the deviation of the abnormal part is higher than that of other parts. On the other hand, when the image quality is poor, the deviation of each part of the ophthalmic image as a whole becomes higher. Therefore, when the deviation is obtained for each of the parts of the ophthalmic image, the control unit can generate image quality evaluation information indicating poor image quality when the deviations of all parts are high (for example, when the deviations of all parts are above a threshold value, etc.). Additionally, the control unit can also generate the image quality evaluation information by considering both an index indicating the strength or goodness of the signal of the captured ophthalmic image (such as SSI (Signal Strength Index) or QI (Quality Index), etc.) and the obtained deviation.

[0076] Furthermore, in another example of the present disclosure, the control unit can generate information indicating the degree of abnormality of the structure of the tissue based on an index indicating the strength or goodness of the signal of the ophthalmic image and the obtained deviation. For example, the control unit can generate information indicating a high possibility of the existence of an abnormality in the structure when the index of the signal of the image is above a threshold value and the deviation is above a threshold value. In this case, it is possible to more appropriately determine the abnormality of the structure while considering the image quality of the ophthalmic image.

[0077] Alternatively, the deviation can be used for the image quality evaluation information instead of being used as the structure information indicating the degree of abnormality of the structure of the tissue. In this case, the ophthalmic image processing device can be configured as follows. An ophthalmic image processing device that processes an ophthalmic image, which is an image of the tissue of an eye to be examined, characterized in that the control unit of the ophthalmic image processing device acquires the ophthalmic image captured by an ophthalmic image capturing device, and by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtains a probability distribution for identifying the tissue in the ophthalmic image, and takes the deviation of the obtained probability distribution from the probability distribution in the case of accurately identifying the tissue as the image quality evaluation information for evaluating the image quality of the ophthalmic image.

[0078] In addition, in the present disclosure, various ophthalmic images can be used as the ophthalmic images input to the mathematical model. For example, the ophthalmic image can be a two-dimensional tomographic image or a three-dimensional tomographic image of the tissue of the eye to be examined taken by an OCT device. The tomographic image can also be taken by a device other than the OCT device (such as a Scheimpflug camera, etc.). In addition, the ophthalmic image can also be a two-dimensional frontal image taken by a fundus camera, a two-dimensional frontal image taken by a laser scanning ophthalmoscope (SLO), etc. The ophthalmic image can also be a two-dimensional frontal image (so-called "Enface image") generated based on the data of the three-dimensional tomographic image taken by the OCT device. In addition, the ophthalmic image can also be a two-dimensional frontal image (so-called "motion contrast image") generated based on the motion contrast data obtained by processing a plurality of OCT data taken from the same position at different times. The so-called two-dimensional frontal image is a two-dimensional image taken of the tissue from the direction of the optical axis of the imaging light. In addition, the tissue to be imaged can also be appropriately selected. For example, an image taken of any one of the fundus, anterior eye segment, and cornea of the eye to be examined can be used as the ophthalmic image.

[0079] <Embodiment>

[0080] (Device Structure)

[0081] Hereinafter, typical embodiments of the present disclosure will be described with reference to the accompanying drawings. As Figure 1 shown, in the present embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image capturing devices 11A and 11B are used. The ophthalmic image capturing devices 11A and 11B capture ophthalmic images that are images of the tissue of the eye to be examined. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm.

[0082] In the first embodiment described later, the constructed mathematical model outputs a probability distribution in which the coordinates of a specific boundary / specific part existing within a region in the ophthalmic image are used as random variables based on the input ophthalmic image. The ophthalmic image processing device 21 uses the mathematical model to obtain the probability distribution and detects a specific boundary / specific part based on the probability distribution.

[0083] In the second embodiment described later, the constructed mathematical model outputs a probability distribution for identifying the tissue in the ophthalmic image based on the input ophthalmic image. The ophthalmic image processing device 21 uses the mathematical model to obtain the probability distribution and obtains the degree of deviation between the obtained probability distribution and the probability distribution in the case of accurately identifying the tissue as structural information indicating the degree of abnormality of the structure of the tissue.

[0084] As an example, the mathematical model construction device 1 of the present embodiment uses a personal computer (hereinafter referred to as "PC"). However, the device capable of functioning as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic image capturing device 11A can also function as the mathematical model construction device 1. In addition, the control units of multiple devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic image capturing device 11A) can also cooperate to construct a mathematical model.

[0085] Details will be described later. As an example, the mathematical model construction device 1 can train a mathematical model by using the ophthalmic images (hereinafter referred to as "training ophthalmic images") obtained from the ophthalmic image capturing device 11A and the training data representing specific boundaries / specific parts of tissues in the training ophthalmic images, thereby constructing a mathematical model. Additionally, as another example, the mathematical model construction device 1 can also train a mathematical model by using the training ophthalmic images and the training data representing the positions of at least any one of the tissues in the training ophthalmic images, thereby constructing a mathematical model.

[0086] In addition, the ophthalmic image processing device 21 of the present embodiment uses a PC. However, the device capable of functioning as the ophthalmic image processing device 21 is not limited to a PC. For example, the ophthalmic image capturing device 11B or a server, etc. can also function as the ophthalmic image processing device 21. Additionally, mobile terminals such as tablet terminals or smartphones can also function as the ophthalmic image processing device 21. The control units of multiple devices (for example, the CPU of the PC and the CPU 13B of the ophthalmic image capturing device 11B) can also cooperate to perform various processes.

[0087] When the ophthalmic image capturing device (OCT device in the present embodiment) 11B functions as the ophthalmic image processing device 21, as an example, the ophthalmic image capturing device 11B can appropriately detect specific boundaries / specific parts in the tissues of the captured ophthalmic images while capturing the ophthalmic images. When the ophthalmic image capturing device (OCT device in the present embodiment) 11B functions as the ophthalmic image processing device 21, as another example, the ophthalmic image capturing device 11B can obtain the deviation degree from the captured ophthalmic images while capturing the ophthalmic images.

[0088] In addition, in the present embodiment, an example in which a CPU is used as the controller for performing various processes is illustrated. However, of course, at least a part of various devices can also use a controller other than the CPU. For example, the high-speedization of processing can also be achieved by adopting a GPU as the controller.

[0089] A description is given of the mathematical model construction device 1. The mathematical model construction device 1 is arranged, for example, in an ophthalmic image processing device 21 or a manufacturer that provides an ophthalmic image processing program to users, etc. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I / F 5. The control unit 2 includes a CPU 3 that serves as a controller responsible for control and a storage device 4 that can store programs, data, etc. A mathematical model construction program for executing the mathematical model construction process described later (refer to Figure 2 ) is stored in the storage device 4. In addition, the communication I / F 5 connects the mathematical model construction device 1 to other devices (for example, an ophthalmic image capturing device 11A and an ophthalmic image processing device 21, etc.).

[0090] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user so that the user can input various instructions into the mathematical model construction device 1. The operation unit 7 can use, for example, at least any one of a keyboard, a mouse, a touch panel, etc. In addition, a microphone, etc. for inputting various instructions can be used together with or instead of the operation unit 7. The display device 8 displays various images. The display device 8 can use various devices capable of displaying images (for example, at least any one of a monitor, a display, a projector, etc.). In addition, the "image" in the present disclosure includes both still images and moving images.

[0091] The mathematical model construction device 1 can acquire data of an ophthalmic image (hereinafter, sometimes simply referred to as an "ophthalmic image") from the ophthalmic image capturing device 11A. The mathematical model construction device 1 can acquire data of an ophthalmic image from the ophthalmic image capturing device 11A, for example, by at least any one of wired communication, wireless communication, a detachable storage medium (for example, a USB memory), etc.

[0092] A description is given of the ophthalmic image processing device 21. The ophthalmic image processing device 21 is arranged, for example, in a facility for diagnosing or examining a subject (for example, a hospital or a health examination facility, etc.). The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23 that serves as a controller responsible for control and a storage device 24 that can store programs, data, etc. An ophthalmic image processing program for executing the ophthalmic image processing described later (for example, for the first embodiment, the boundary detection process exemplified by Figure 5 and the specific part detection process shown by Figure 10 , etc. For the second embodiment, refer to Figure 13 ) is stored in the storage device 24. The ophthalmic image processing program includes a program for implementing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (for example, an ophthalmic image capturing device 11B and the mathematical model construction device 1, etc.).

[0093] The ophthalmic image processing device 21 is connected to the operation unit 27 and the display device 28. Similar to the above-described operation unit 7 and display device 8, the operation unit 27 and the display device 28 can use various devices.

[0094] The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B, for example, by at least any one of wired communication, wireless communication, a detachable storage medium (e.g., a USB memory), etc. In addition, the ophthalmic image processing device 21 can also acquire a program, etc., for implementing the mathematical model constructed by the mathematical model construction device 1 through communication, etc.

[0095] The ophthalmic image capturing devices 11A and 11B will be described. As an example, in the present embodiment, the case where the ophthalmic image capturing device 11A that provides an ophthalmic image to the mathematical model construction device 1 and the ophthalmic image capturing device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used will be described. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 can also acquire ophthalmic images from a plurality of ophthalmic image capturing devices. In addition, the mathematical model construction device 1 and the ophthalmic image processing device 21 can also acquire ophthalmic images from a common ophthalmic image capturing device. Additionally, the two ophthalmic image capturing devices 11A and 11B illustrated in the present embodiment have the same structure. Therefore, the two ophthalmic image capturing devices 11A and 11B will be described collectively below.

[0096] In addition, in the present embodiment, an OCT device is exemplified as the ophthalmic image capturing device 11 (11A, 11B). However, an ophthalmic image capturing device other than the OCT device (e.g., a laser scanning ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM), etc.) can also be used.

[0097] The ophthalmic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes and an ophthalmic image capturing unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller responsible for control and a storage device 14 (14A, 14B) that can store programs, data, etc.

[0098] The ophthalmic image capturing unit 16 includes various structures necessary for capturing an ophthalmic image of an eye to be examined. The ophthalmic image capturing unit 16 of the present embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, an optical system for irradiating the measurement light onto the eye to be examined, a light receiving element that receives the combined light of the light reflected by the tissue of the eye to be examined and the reference light, and the like.

[0099] The ophthalmic image capturing device 11 is capable of capturing two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the eye to be examined. Specifically, the CPU 13 captures two-dimensional tomographic images of the cross-section intersecting the scanning line by scanning the OCT light (measurement light) on the scanning line. The two-dimensional tomographic image may also be an addition-averaged image generated by performing addition-averaging processing on a plurality of tomographic images of the same part. In addition, the CPU 13 can capture three-dimensional tomographic images in the tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 obtains a plurality of two-dimensional tomographic images by scanning the measurement light on a plurality of scanning lines with different positions in a two-dimensional region when observing the tissue from the front. Then, the CPU 13 obtains a three-dimensional tomographic image by combining the plurality of captured two-dimensional tomographic images.

[0100] (Mathematical model construction process)

[0101] Refer to Figures 2 to 4 , and the mathematical model construction process executed by the mathematical model construction device 1 will be described. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4. In the mathematical model construction process, a mathematical model that outputs a probability distribution related to the tissue in the ophthalmic image is constructed by training the mathematical model using a training data set. More specifically, in the first embodiment, in the mathematical model construction process, a mathematical model that outputs a probability distribution having at least one of the coordinates of a specific boundary and a specific part where the tissue exists in the region in the ophthalmic image as a random variable is constructed. In the second embodiment, in the mathematical model construction process, a mathematical model that outputs a probability distribution for identifying the tissue in the ophthalmic image is constructed. The training data set includes input-side data (input training data) and output-side data (output training data).

[0102] As Figure 2As shown, the CPU 3 acquires, as input training data, data of an ophthalmic image (i.e., a training ophthalmic image) captured by the ophthalmic image capturing device 11A (S1). In the present embodiment, after the data of the training ophthalmic image is generated by the ophthalmic image capturing device 11A, it is acquired by the mathematical model construction device 1. However, the CPU 3 can also acquire a signal (e.g., an OCT signal) that is the basis for generating the training ophthalmic image from the ophthalmic image capturing device 11A, and generate the training ophthalmic image based on the acquired signal, thereby acquiring the data of the training ophthalmic image. Figure 3 An example of the training ophthalmic image 30, which is a two-dimensional tomographic image of the fundus, is shown. In Figure 3 The illustrated training ophthalmic image 30 shows multiple layers of the fundus.

[0103] In addition, in the first embodiment, when the boundary detection process (refer to Figure 5 ) described later is executed by the ophthalmic image processing device 21, in S1, a two-dimensional tomographic image of the tissue (as an example, the fundus) of the eye to be examined is acquired as the training ophthalmic image.

[0104] Furthermore, in the first embodiment, when the specific site detection process (refer to Figure 10 ) described later is executed by the ophthalmic image processing device 21, in S1, a two-dimensional frontal image (e.g., an Enface image captured by the ophthalmic image capturing device 11A which is an OCT device, etc.) of the tissue (as an example, the fundus) of the eye to be examined is acquired as the training ophthalmic image.

[0105] However, in the first embodiment, the ophthalmic image used as the training ophthalmic image can also be changed. For example, when using a two-dimensional frontal image as the training ophthalmic image, the training ophthalmic image can be a two-dimensional frontal image captured by a device other than the OCT device (e.g., an SLO device or a fundus camera, etc.). A two-dimensional image other than the frontal image (e.g., a two-dimensional tomographic image) can also be used as the training ophthalmic image. In addition, in the specific site detection process (refer to Figure 10 ) described later, when detecting a specific site in three dimensions, in S1, a three-dimensional tomographic image can also be acquired as the training ophthalmic image.

[0106] In addition, in S1 of the second embodiment, a two-dimensional tomographic image captured by the ophthalmic image capturing device 11A which is an OCT device is acquired as the training ophthalmic image. In addition, in the second embodiment, an ophthalmic image of a tissue with a low degree of structural abnormality is acquired as the training ophthalmic image. In this case, it is difficult for the mathematical model trained by the training data set to recognize a tissue with a high degree of structural abnormality. As a result, the deviation obtained for an ophthalmic image with a high degree of structural abnormality is likely to become larger.

[0107] However, in the second embodiment, the ophthalmic image used as the training ophthalmic image can also be changed. For example, a two-dimensional frontal image of the tissue of the eye to be examined can also be used as the training ophthalmic image. In this case, the ophthalmic image capturing device for capturing the training ophthalmic image can use various devices (for example, at least any one of an OCT device, an SLO device, a fundus camera, an infrared camera, a corneal endothelial cell capturing device, etc.). In addition, a three-dimensional image can also be used as the training ophthalmic image.

[0108] Next, the CPU 3 acquires the training data (S2). In the first embodiment, the training data represents the position of at least one of a specific boundary and a specific part in the tissue in the training ophthalmic image. In the second embodiment, it represents the position of at least any one tissue among the tissues in the training ophthalmic image. Figure 4 An example of the training data 31 when using a two-dimensional tomographic image of the fundus as the training ophthalmic image 30 is shown. As an example, the training data 31 can represent the position of a specific boundary. Figure 4 The illustrated training data 31 includes data of labels 32A to 32F representing the positions of six boundaries in the training ophthalmic image 30 respectively. As an example, Figure 4 The illustrated training data 31 includes data of labels 32A to 32F representing the positions of six boundaries among a plurality of tissues (specifically, a plurality of layers and boundaries) shown in the training ophthalmic image 30. In the present embodiment, the data of the labels 32A to 32F in the training data 31 is generated by an operator operating the operation unit 7 while observing the boundaries in the training ophthalmic image 30. However, the method of generating the label data can also be changed.

[0109] In addition, in the first embodiment, when the specific part detection process (refer to Figure 10 ) described later is executed by the ophthalmic image processing device 21, data representing the position of a specific part in the tissue of the two-dimensional tomographic image or three-dimensional tomographic image used as the training ophthalmic image is acquired in S2. For example, the position of the fovea is detected in the specific part detection process described later. In this case, in S2, data representing the position of the fovea in the two-dimensional tomographic image or three-dimensional tomographic image is acquired.

[0110] In addition, in the second embodiment, the training data can also be changed. For example, when using a two-dimensional tomographic image of the fundus as the training ophthalmic image 30, the training data can be data representing the position of at least any one layer in the fundus. In addition, the training data may not be data representing the positions of layers and boundaries, but data representing the positions of dot-like parts in the tissue, etc.

[0111] Next, the CPU 3 uses a machine learning algorithm to perform training of a mathematical model using the training data set (S3). As machine learning algorithms, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

[0112] A neural network is a method of simulating the behavior of a biological neural cell network. Neural networks include, for example, feedforward (forward propagation) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0113] A random forest is a method of generating a large number of decision trees by learning based on randomly sampled training data. In the case of using a random forest, the branches of multiple decision trees pre-learned as classifiers are traced, and the average value (or majority vote) of the results obtained from each decision tree is taken.

[0114] Boosting is a method of generating a strong recognizer by combining multiple weak recognizers. A strong recognizer is constructed by successively making only the weak recognizers learn.

[0115] SVM is a method of using linear input elements to construct a two-class pattern recognizer. For example, SVM learns the parameters of the linear input elements based on a criterion (hyperplane separation theorem) such as finding a maximum margin hyperplane with the largest distance between each data point according to the training data.

[0116] A mathematical model is, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by training using a training data set. As described above, the training data set is a set of input training data and output training data. For example, the relevant data (e.g., weights) of each input and output can be updated through training.

[0117] In this embodiment, as the machine learning algorithm, a multi-layer neural network is used. The neural network includes an input layer for input data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. Multiple nodes (also called neurons) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used.

[0118] In addition, the mathematical model constructed in the first embodiment outputs a probability distribution of coordinates (any one of one-dimensional coordinates, two-dimensional coordinates, and three-dimensional coordinates) of at least any one of a specific boundary and a specific part where tissue exists within a region (any one of a one-dimensional region, a two-dimensional region, and a three-dimensional region) in an ophthalmic image as random variables. In the first embodiment, in order for the mathematical model to output a probability distribution, a normalized exponential (Softmax) function is applied.

[0119] In the first embodiment, when the boundary detection process (see Figure 5 ) described later is executed by the ophthalmic image processing device 21, the mathematical model constructed in S3 outputs a probability distribution of coordinates where a specific boundary exists within a one-dimensional region on a line extending in a direction (in this embodiment, the A-scan direction of OCT) intersecting the specific boundary in the two-dimensional tomographic image.

[0120] In addition, in the first embodiment, when the specific part detection process (see Figure 10 ) described later is executed by the ophthalmic image processing device 21, the mathematical model constructed in S3 outputs a probability distribution of two-dimensional coordinates where a specific part (for example, the fovea in this embodiment) exists within a two-dimensional region in the two-dimensional ophthalmic image as random variables.

[0121] As an example, the mathematical model constructed in the second embodiment outputs a probability distribution of coordinates (any one of one-dimensional coordinates, two-dimensional coordinates, three-dimensional coordinates, and four-dimensional coordinates) of a specific tissue (for example, a specific boundary, a specific layer, or a specific part, etc.) existing within a region (any one of a one-dimensional region, a two-dimensional region, a three-dimensional region, and a four-dimensional region including a time axis) in an ophthalmic image as a probability distribution for identifying the tissue. In the second embodiment, in order for the mathematical model to output a probability distribution, a normalized exponential function is applied. Specifically, the mathematical model constructed in S3 outputs a probability distribution of coordinates where a specific boundary exists within a one-dimensional region on a line extending in a direction (in this embodiment, the A-scan direction of OCT) intersecting the specific boundary in the two-dimensional tomographic image as random variables.

[0122] However, in the second embodiment, the specific method by which the mathematical model outputs the probability distribution for identifying tissues can be appropriately changed. For example, the mathematical model may also output the probability distribution with the two-dimensional coordinates or three-dimensional coordinates of the presence of a specific tissue (such as a characteristic part, etc.) in a two-dimensional region or a three-dimensional region as random variables, as the probability distribution for identifying tissues. In addition, the mathematical model may also output, for each region (e.g., each pixel) of the input ophthalmic image, the probability distribution with the types of multiple tissues (e.g., multiple layers and boundaries) in the eye to be examined as random variables. In addition, the ophthalmic image input to the mathematical model may also be a dynamic image.

[0123] In addition, other machine learning algorithms may also be used. For example, a generative adversarial network (GAN) that utilizes two competing neural networks may be adopted as the machine learning algorithm.

[0124] The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S4: No). When the construction of the mathematical model is completed (S4: Yes), the mathematical model construction process ends. The program and data for implementing the constructed mathematical model are incorporated into the ophthalmic image processing device 21.

[0125] (Ophthalmic Image Processing)

[0126] Refer to Figures 5 to 17 , and the ophthalmic image processing performed by the ophthalmic image processing device 21 will be described. The ophthalmic image processing is executed by the CPU 23 according to the ophthalmic image processing program stored in the storage device 21.

[0127] (Boundary Detection Processing)

[0128] Refer to Figures 5 to 9 , and the boundary detection processing, which is an example of the ophthalmic image processing according to the first embodiment, will be described. In the boundary detection processing, a specific boundary (in the first embodiment, a two-dimensional boundary and a three-dimensional boundary) is detected based on the probability distribution with the one-dimensional coordinates of the presence of a specific boundary in each of multiple one-dimensional regions as random variables. Moreover, based on the detected three-dimensional boundary, an Enface image of a specific layer in the tissue is obtained.

[0129] First, the CPU 23 acquires a three-dimensional tomographic image of the tissue of the eye to be examined (the fundus in the first embodiment) (S11). The three-dimensional tomographic image is captured by the ophthalmic image capturing device 11B and acquired by the ophthalmic image processing device 21. As described above, the three-dimensional tomographic image is constituted by combining a plurality of two-dimensional tomographic images, and the plurality of two-dimensional tomographic images are captured by scanning measurement light on mutually different scan lines. Additionally, the CPU 23 may also acquire a signal (such as an OCT signal) that serves as the basis for generating the three-dimensional tomographic image from the ophthalmic image capturing device 11B, and generate the three-dimensional tomographic image based on the acquired signal.

[0130] The CPU 23 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image among the plurality of two-dimensional tomographic images that constitute the acquired three-dimensional tomographic image (S12). Figure 6 Shows an example of the two-dimensional tomographic image 40. A plurality of boundaries in the fundus of the eye to be examined are shown in the two-dimensional tomographic image 40. In Figure 6 the example shown, a plurality of boundaries including the boundary Bi as the inner limiting membrane (ILM) and the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL) are shown. Additionally, a plurality of one-dimensional regions A1 to AN are set in the two-dimensional tomographic image 40. In the first embodiment, the plurality of one-dimensional regions A1 to AN set in the two-dimensional tomographic image 40 extend along an axis intersecting a specific boundary (in the first embodiment, a plurality of boundaries including the boundary Bi and the boundary Bg). Specifically, the one-dimensional regions A1 to AN in the first embodiment coincide with the regions of each of the plurality of (N) A-scans that constitute the two-dimensional tomographic image 40 captured by the OCT device.

[0131] In addition, the method of setting the plurality of one-dimensional regions can also be changed. For example, the CPU 23 may set the plurality of one-dimensional regions such that the angle between the axis of each one-dimensional region and the specific boundary is as close to perpendicular as possible. In this case, for example, based on the shape of the tissue of a normal eye to be examined (the fundus in the first embodiment), the position and angle of each one-dimensional region can be set such that the angle is close to perpendicular.

[0132] The CPU 23 obtains a probability distribution of the coordinates where the M-th (the initial value of M is "1") boundary exists in each of the plurality of one-dimensional regions A1 to AN by inputting the T-th two-dimensional tomographic image into the mathematical model (S14). Figure 7 Shows an example of a graph representing the probability distribution of the coordinates where the boundary Bi exists obtained from the one-dimensional region A1. In Figure 7 the example shown, the one-dimensional coordinate of the one-dimensional region A1 is used as a random variable, and the probability distribution of the coordinates where the boundary Bi exists is shown. That is, in Figure 7In the example shown, the horizontal axis is a random variable, and the vertical axis is the probability of the random variable. The random variable is the coordinate of the boundary Bi existing in the one-dimensional region A1. In S14, the probability distribution in each of the multiple one-dimensional regions A1 to AN is obtained.

[0133] According to Figure 7 the chart shown, it can be judged that among the points on the one-dimensional region A1, the point with the highest possibility of the existence of the boundary Bi is point P. In addition, even when the structure of the tissue is damaged due to the influence of a disease, or when the image quality of at least a part of the ophthalmic image is poor, etc., the probability distribution output by the mathematical model is likely to form a peak at any position. Therefore, by using the probability distribution output in S14, the boundary can be detected appropriately and directly.

[0134] The CPU 23 obtains a probability map that two-dimensionally represents the ratio of the existence of the M-th boundary in the T-th two-dimensional tomographic image for the M-th boundary. The CPU 23 causes the obtained probability map to be displayed on the display device 28 (S15). The probability map is generated by two-dimensionally arranging the multiple probability distributions obtained for each of the multiple one-dimensional regions A1 to AN. The probability map can be generated by the CPU 23 based on the multiple probability distributions. In addition, the probability map can also be generated by the mathematical model.

[0135] Figure 8 is an example of the probability map 41 of the boundary Bi as the inner limiting membrane (ILM). Figure 9 is an example of the probability map 42 of the boundary Bg between the nerve fiber layer (NFL) and the ganglion cell layer (GCL). In Figure 8 and Figure 9 In the exemplified probability maps 41 and 42, the value for each coordinate is represented by brightness, and the position more similar to a specific boundary is brighter. Therefore, by viewing the displayed probability maps 41 and 42, the user can appropriately grasp the positions of the boundaries Bi and Bg in a two-dimensional manner. In addition, it goes without saying that the display method of the probability map can be appropriately changed.

[0136] Next, the CPU 23 detects the M-th boundary of the two-dimensional tomographic image based on the probability distributions related to the M-th boundary obtained for each of the multiple one-dimensional regions A1 to AN (S16). In the first embodiment, the CPU 23 detects the two-dimensional boundary by combining the multiple probability distributions obtained for each of the multiple one-dimensional regions A1 to AN. The two-dimensional boundary can be detected based on the multiple probability distributions obtained in S14, or can be detected based on the probability map obtained in S15. In addition, the detection result of the boundary can be output by the mathematical model, or the CPU 23 can perform the boundary detection process.

[0137] In addition, in the first embodiment, after post-processing the probability distributions related to the plurality of one-dimensional regions A1 to AN, generation of a probability map and detection of a boundary are performed. As an example, a boundary on the n-th one-dimensional region can be detected based on the probability distribution on the n-th one-dimensional region and the probability distributions on the one-dimensional regions near the n-th one-dimensional region. In this case, for example, a known method such as graph cut can be employed. In addition, the boundary can also be detected by using the theory of shortest path search for finding the path with the minimum weight.

[0138] In addition, the CPU 23 can also overlap and display the probability maps 41 and 42 or the detected specific boundary on the two-dimensional tomographic image (i.e., the original image) that is the object of boundary detection. In this case, the positional relationship between the actually photographed tissue and the boundary detected from the probability distribution can be appropriately grasped. In addition, the CPU 23 can also input an instruction from the user for specifying the position of the boundary in the tissue in a state where the probability maps 41 and 42 or the detected boundary are overlapped and displayed on the original image. In this case, the user can appropriately specify the position of the boundary while comparing the boundary detected from the probability distribution with the image of the actually photographed tissue.

[0139] Next, the CPU 23 determines whether the detection of all the boundaries to be detected in the T-th two-dimensional tomographic image is completed (S18). If the detection of some of the boundaries is not completed (S18: No), then "1" is added to the order M of the boundaries (S19), and the process returns to S14 to perform the detection process for the next boundary (S14 to S16). When the detection of all the boundaries is completed (S18: Yes), the CPU 23 determines whether the detection of the boundaries of all the two-dimensional tomographic images constituting the three-dimensional tomographic image is completed (S21). If the detection of the boundaries of some of the two-dimensional tomographic images is not completed (S21: No), then "1" is added to the order T of the two-dimensional tomographic images (S22), and the process returns to S12 to detect the boundaries of the next two-dimensional tomographic image (S12 to S19).

[0140] When the detection of the boundaries of all the two-dimensional tomographic images is completed (S21: Yes), the detection of the three-dimensional boundary is completed. The CPU 23 integrates the plurality of two-dimensional boundaries detected in S16 for at least any one of the M boundaries, thereby obtaining the three-dimensional boundary (S24). In addition, the CPU 23 can generate image data of the detected three-dimensional boundary and display it on the display device 28. Therefore, the user can appropriately grasp the three-dimensional shape of the boundary. In addition, a probability map representing the probability distribution in the three-dimensional region can also be generated based on the acquisition result of the probability distributions for the T two-dimensional tomographic images.

[0141] Next, the CPU 23 obtains an Enface image (S25) of a specific layer (which may also be a boundary) included in the three-dimensional tomographic image based on the detected three-dimensional boundary and the three-dimensional tomographic image obtained in S11. The Enface image is a two-dimensional front image when observing the specific layer in the direction along the optical axis of the measurement light of the OCT. Since the Enface image obtained in S25 is generated based on the appropriately detected boundary, the quality is high. In addition, in S25, an Enface image of a specific one of the multiple layers and boundaries included in the three-dimensional image may also be obtained. Further, Enface images of multiple ones of the multiple layers and boundaries included in the three-dimensional image may also be obtained.

[0142] In addition, the CPU 23 may also obtain a thickness map representing the thickness distribution of a specific layer included in the three-dimensional tomographic image in a two-dimensional manner based on the three-dimensional tomographic image obtained in S11 and the detected three-dimensional boundary. In this case, after determining the specific layer based on the appropriately detected boundary, a more accurate thickness map can be obtained.

[0143] In addition, in Figure 5 the example shown, after obtaining the three-dimensional tomographic image in S11, a specific boundary is detected in two-dimensional and three-dimensional manners. Then, an Enface image is obtained based on the detected three-dimensional boundary. However, it is also possible to detect only a two-dimensional boundary. In this case, a two-dimensional ophthalmic image may also be obtained in S11. In addition, in Figure 5 the example shown, in S14, a plurality of probability distributions related to each of the plurality of one-dimensional regions A1 to AN are obtained. As a result, a two-dimensional boundary is detected. However, for example, in a case where it is sufficient to detect only one point on a specific boundary, in S14, only the probability distribution in one one-dimensional region may also be obtained.

[0144] (Specific part detection process)

[0145] Refer to Figures 10 to 12 to describe a specific part detection process as an example of ophthalmic image processing according to the first embodiment. In the specific part detection process, a specific part is detected based on the probability distribution of the coordinates where the specific part exists. Hereinafter, an example of detecting a specific part from a two-dimensional ophthalmic image will be illustrated.

[0146] First, the CPU 23 acquires an ophthalmic image of the tissue of the eye to be examined (the fundus in the first embodiment) (S31). As an example, in S31 of the first embodiment, a two-dimensional frontal image (e.g., an Enface image or a motion contrast image) captured by an OCT device is acquired. However, the ophthalmic image acquired in S31 can be appropriately changed. For example, a two-dimensional frontal image captured by an SLO device, a fundus camera, or the like can also be acquired in S31. Additionally, an ophthalmic image of a tissue other than the fundus can be acquired in S31.

[0147] Figure 11 FIG. 4 shows an example of the ophthalmic image 50 acquired in S31 of the first embodiment. In Figure 11 the illustrated two-dimensional ophthalmic image 50, the optic nerve head 51, the macula 52, and the fundus blood vessels 53 of the fundus are visualized. The center of the macula 52 is the fovea F. Additionally, a two-dimensional coordinate system (XY coordinates) is set in the two-dimensional region C in the two-dimensional ophthalmic image 50.

[0148] The CPU 23 inputs the ophthalmic image 50 into the mathematical model, thereby obtaining a probability distribution in which the two-dimensional coordinates of a specific part existing within the two-dimensional region in the ophthalmic image 50 are random variables (S32). As an example, the specific part in the first embodiment is the fovea F. However, it goes without saying that the specific part is not limited to the fovea F. For example, the specific part can also be the optic nerve head or the like. Additionally, the specific part can be a single point or a part having a certain area or volume. The CPU 23 detects the specific part in the two-dimensional ophthalmic image 50 based on the probability distribution obtained in S32 (S33).

[0149] Figure 12 FIG. 5 shows an example of a graph representing a probability distribution in which the coordinates of the existence of a specific part obtained from the two-dimensional coordinate system are random variables. In Figure 12 the example shown, the X-axis and the Y-axis are random variables, and the axis orthogonal to the X-axis and the Y-axis is the probability of the random variable, and the random variable is the coordinate of the existence of the fovea F in the two-dimensional region C. According to Figure 12 the graph shown, it can be determined that the coordinates of the point with the highest possibility of the existence of the fovea F among the points in the two-dimensional region C are (X', Y'). Furthermore, even when the structure of the tissue is damaged due to the influence of a disease, or when the image quality of at least a part of the ophthalmic image is poor, etc., it is highly likely that the probability distribution output by the mathematical model forms a peak at any position. Therefore, the specific part can be appropriately and directly detected.

[0150] In addition, in the detection process of a specific part, it is also possible to detect a specific part from a three-dimensional ophthalmic image. In this case, a three-dimensional ophthalmic image (for example, a three-dimensional tomographic image taken by an OCT device, etc.) is acquired in S31. In S32, a probability distribution with the three-dimensional coordinates of the specific part existing within the three-dimensional region in the three-dimensional ophthalmic image as a random variable is acquired. In S33, based on the probability distribution acquired in S32, the specific part in the three-dimensional ophthalmic image is detected.

[0151] Next, with reference to Figures 13 to 17 , the ophthalmic image processing according to the second embodiment will be described. In addition, for the processing that is the same as the ophthalmic image processing according to the first embodiment, the description will be omitted or simplified.

[0152] First, in the same manner as S11 of the boundary detection process, which is an example of the ophthalmic image processing according to the first embodiment, the CPU 23 acquires a three-dimensional tomographic image of the tissue of the eye to be examined (the fundus in the second embodiment) (S31). Next, in the same manner as S12 of the boundary detection process, the CPU 23 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images that make up the acquired three-dimensional tomographic image (S32).

[0153] The CPU 23 inputs the T-th two-dimensional tomographic image into the mathematical model, thereby acquiring, as the probability distribution for identifying the tissue, the probability distribution of the coordinates where the M-th (the initial value of M is "1") boundary exists in each of the plurality of one-dimensional regions A1 to AN (S34). Figure 7 and Figure 14 shows an example of a graph representing the probability distribution of the coordinates where the boundary Bi exists, which is obtained from the one-dimensional coordinate A1. In Figure 7 and Figure 14 In the example shown, taking the one-dimensional coordinate of the one-dimensional region A1 as a random variable, the probability distribution of the coordinates where the boundary Bi exists is shown. That is, in Figure 7 and Figure 14 In the example shown, the horizontal axis is the random variable, and the vertical axis is the probability of the random variable, and the random variable is the coordinate where the boundary Bi exists in the one-dimensional region A1. In S34, the probability distribution in each of the plurality of one-dimensional regions A1 to AN is acquired.

[0154] In the second embodiment, Figure 7 the probability distribution shown is an example of the probability distribution output when the abnormality degree of the structure of the tissue (specifically, the tissue near the boundary Bi) is low. At a position where the abnormality degree of the structure is low, it is easy to accurately identify the tissue through the mathematical model, so the probability of the position of the tissue tends to be biased. According to Figure 7In the shown graph, it can be judged that among the points on the one-dimensional region A1, the point with the highest possibility of the existence of the boundary Bi is the point P. The probability distribution (i.e., the ideal probability distribution) when the mathematical model accurately identifies the tissue takes the value of 1 at only one point on the one-dimensional region A1 and 0 at other points.

[0155] On the other hand, Figure 14 The shown probability distribution is an example of the probability distribution output in the case of a high degree of abnormality in the structure of the tissue. As Figure 14 shown, at the position with a high degree of abnormality in the structure, the probability distribution is difficult to be biased. As described above, the bias of the probability distribution for identifying the tissue changes according to the degree of abnormality in the structure of the tissue.

[0156] Next, the CPU 23 obtains the degree of deviation (S35) of the probability distribution P related to the Mth boundary. The degree of deviation refers to the difference between the probability distribution P obtained in S14 and the probability distribution P in the case of accurately identifying the tissue. In the second embodiment, the degree of deviation is obtained as the structure information indicating the degree of abnormality in the structure of the tissue. In S35 of the second embodiment, the degree of deviation is obtained (calculated) for each of the multiple probability distributions P obtained for the multiple one-dimensional regions A1 to AN.

[0157] In the second embodiment, the entropy of the probability distribution P is calculated as the degree of deviation. The entropy is given by the following (mathematical formula 1). The entropy H(P) takes the value of 0 ≤ H(P) ≤ log(number of events), and the more biased the probability distribution P is, the smaller its value. That is, the smaller the entropy H(P) is, the lower the degree of abnormality in the structure of the tissue. The entropy of the probability distribution in the case of accurately identifying the tissue is 0. In addition, the more the degree of abnormality in the structure of the tissue increases, the more difficult it is to identify the tissue, and the greater the entropy H(P) becomes. Therefore, by using the entropy H(P) of the probability distribution P as the degree of deviation, the degree of abnormality in the structure of the tissue can be appropriately quantified.

[0158] H(P)=-∑plog(p)……(mathematical formula 1)

[0159] However, a value other than entropy can also be used as the degree of deviation. For example, at least any one of the standard deviation, coefficient of variation, variance, etc. indicating the degree of dispersion of the obtained probability distribution P can be used as the degree of deviation. The KL divergence, etc., which is a scale for realizing the difference between probability distributions P, can also be used as the degree of deviation. In addition, the maximum value of the obtained probability distribution P (for example, Figure 7 and Figure 14 the maximum value of the probabilities exemplified) can be used as the degree of deviation. In addition, the difference between the maximum value and the second largest value of the obtained probability distribution P can also be used as the degree of deviation.

[0160] Next, the CPU 23 determines whether the deviation degrees of all the boundaries to be detected in the T-th two-dimensional tomographic image have been obtained (S36). If the deviation degrees of some boundaries have not been obtained (S16: No), the order M of the boundaries is incremented by "1" (S37), and the process returns to S14 to obtain the deviation degree of the next boundary (S34, S35). When the deviation degrees of all the boundaries have been obtained (S36: Yes), the CPU 23 stores the deviation degree of the T-th two-dimensional tomographic image in the storage device 24 and displays it on the display device 28 (S39). The CPU 23 obtains (generates in the second embodiment) the structural abnormality degree chart of the T-th two-dimensional tomographic image and displays it on the display device 28 (S40).

[0161] Refer to Figure 15 and Figure 16 to describe the structural abnormality degree chart 52. Figure 15 FIG. is an example of a display screen showing a two-dimensional tomographic image 51A with a low degree of structural abnormality, a structural abnormality degree chart 52A related to the two-dimensional tomographic image 51A, and a deviation degree table 53A showing the deviation degrees related to the two-dimensional tomographic image 51A. In addition, Figure 16 FIG. is an example of a display screen showing a two-dimensional tomographic image 51B with a high degree of structural abnormality, a structural abnormality degree chart 52B related to the two-dimensional tomographic image 51B, and a deviation degree table 53B showing the deviation degrees related to the two-dimensional tomographic image 51B.

[0162] As Figure 15 and Figure 16 shown, the two-dimensional tomographic image 51 is a two-dimensional image extending in the X direction (the left-right direction in the drawing) and the Z direction (the up-down direction in the drawing). As described above, the deviation degree is obtained for each of a plurality of axes (a plurality of A-scans in the second embodiment) extending parallel to the Z direction on the ophthalmic image. In Figure 9 and Figure 10 the structural abnormality degree chart 52 shown, the horizontal axis is set as the X-axis, and the vertical axis represents the deviation degree at each position in the X direction.

[0163] As an example, in the structural abnormality degree chart 52 of the second embodiment, the average value of the plurality of deviation degrees (entropy in the second embodiment) obtained for each of the plurality of boundaries is shown for each position in the X direction. However, the deviation degree of one boundary may be represented by the structural abnormality degree chart 52. In addition, the average value of specific multiple boundaries (for example, the boundary between IPL / INL, the boundary between OPL / ONL) may be represented by the structural abnormality degree chart 52. In addition, various statistical values other than the average value (for example, median, mode, maximum value, or minimum value, etc.) may be used instead of the average value.

[0164] As Figure 15As shown, when the overall abnormality degree in the X direction is low, the deviation degree represented by the structure abnormality degree chart 52A is a low value overall in the X direction. On the other hand, as Figure 16 shown, at the position in the X direction where the abnormality degree of the structure is high, the deviation degree represented by the structure abnormality degree chart 52B is a high value. As described above, according to the structure abnormality degree chart 52, the user can appropriately grasp which position in the X direction has a high abnormality degree.

[0165] Refer to Figure 15 and Figure 16 , and an example of the display method of the deviation degree will be described. As Figure 15 and Figure 16 shown, in the deviation degree table 53 of the present embodiment, the obtained deviation degree (entropy in the present embodiment) is displayed for each of a plurality of boundaries. Therefore, the user can appropriately grasp the boundary with a high abnormality degree of the structure based on the quantified value. The deviation degree displayed in the deviation degree table 53 of the second embodiment is the average value of a plurality of deviation degrees obtained for each of a plurality of one-dimensional regions (A-scan in the present embodiment). In addition, in the deviation degree table 53 of the present embodiment, the average value of the deviation degrees related to all boundaries is displayed. Therefore, the user can easily grasp whether there is a part with a high abnormality degree of the structure in the tissue shown in the ophthalmic image. Moreover, in the deviation degree table 53 of the second embodiment, the average value of the deviation degrees related to a plurality of specific boundaries among all boundaries is displayed. As an example, in the second embodiment, the average values of the boundaries (IPL / INL boundary, OPL / ONL boundary) where the structure is likely to be damaged due to the influence of the disease are displayed. Therefore, the user can easily grasp whether there is a structural abnormality caused by the disease. In addition, as described above, various statistical values other than the average value can also be used.

[0166] Next, the CPU 23 determines whether the deviation degrees of all two-dimensional tomographic images constituting the three-dimensional tomographic image have been obtained (S41). If the deviation degrees of some two-dimensional tomographic images have not been obtained (S41: No), the order T of the two-dimensional tomographic images is incremented by "1" (S42), and the process returns to S12 to obtain the deviation degree of the next two-dimensional tomographic image (S32 - S40). When the deviation degrees of all two-dimensional tomographic images have been obtained (S41: Yes), the CPU 23 obtains (generates in the second embodiment) the structure abnormality degree map and displays it on the display device 28 (S44).

[0167] Refer to Figure 17, the structural abnormality degree map 55 will be described. The structural abnormality degree map 55 is a map representing the two-dimensional distribution of the deviation degree in the tissue. As described above, in the second embodiment, the deviation degree is obtained for each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image. That is, the deviation degree of the entire tissue shown in the three-dimensional tomographic image is obtained. In the structural abnormality degree map 55 of the second embodiment, the two-dimensional distribution of the deviation degree when observing the tissue (the fundus in the second embodiment) from the front is shown. However, the direction representing the two-dimensional distribution can be appropriately changed. In addition, in the structural abnormality degree map 55 of the second embodiment, the higher the deviation degree, the lower the displayed brightness. However, the specific method for representing the deviation degree at each position in the structural abnormality degree map 55 can also be appropriately changed. In addition, in the structural abnormality degree map 55 of the second embodiment, the average value of the deviation degrees related to a plurality of boundaries is used. However, the deviation degree related to a specific boundary can also be used.

[0168] Next, when there is a part (hereinafter referred to as "abnormal part") in the tissue shown in the ophthalmic image (the three-dimensional tomographic image in the second embodiment) whose deviation degree is equal to or higher than the threshold value, the CPU 23 causes the tomographic image (at least one of the two-dimensional tomographic image and the three-dimensional tomographic image) or the enlarged image of the abnormal part to be displayed on the display device 28 (S45). Therefore, the image of the abnormal part is appropriately confirmed by the user.

[0169] Specifically, the CPU 23 of the second embodiment causes the ophthalmic image with the highest deviation degree or the ophthalmic image with a deviation degree equal to or higher than the threshold value among the plurality of ophthalmic images (in this embodiment, the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image) obtained by photographing the tissue of the same subject eye to be displayed on the display device 28. For example, the CPU 23 can also cause the ophthalmic image with a high deviation degree among the plurality of ophthalmic images to be displayed on the shooting confirmation screen for the user to confirm the photographed ophthalmic image. In addition, when starting the viewer for displaying the photographed ophthalmic image, the CPU 23 can also cause the ophthalmic image with a high deviation degree among the plurality of ophthalmic images to be displayed. In this case, the user can easily confirm the ophthalmic image of the part with a high structural abnormality degree that has been photographed. This process can also be performed in the same manner when the ophthalmic image photographing device 11B is performing ophthalmic image processing.

[0170] In addition, when the ophthalmic image capturing device 11B is performing ophthalmic image processing, the CPU 13B of the ophthalmic image capturing device 11B outputs a shooting instruction (S25) for shooting an image of a part with a deviation degree equal to or higher than a threshold value (in the second embodiment, at least any one of a two-dimensional tomographic image, a three-dimensional tomographic image, a motion contrast image, etc.). In addition, when shooting an image of a part with a deviation degree equal to or higher than a threshold value, the CPU 13B may shoot ophthalmic images of the same part multiple times and obtain an addition average image of the multiple captured ophthalmic images. In this case, an ophthalmic image of a part with a high degree of structural abnormality can be obtained with high quality. In addition, the shooting instruction may be output from the ophthalmic image processing device 23 to the ophthalmic image capturing device 11B.

[0171] Next, the CPU 23 performs follow-up processing (S46). The follow-up processing is for enabling the user to perform follow-up observation of the eye to be examined. As described above, the obtained deviation degree is stored in the storage device 24 (S39). In the follow-up processing, the CPU 23 causes a plurality of deviation degrees related to each of a plurality of ophthalmic images obtained by photographing the tissue of the same eye to be examined at different times to be displayed on the display device 28. Therefore, the user can appropriately grasp the development status of structural abnormalities, etc. by comparing the plurality of deviation degrees. In addition, the CPU 23 may display the values of the plurality of deviation degrees themselves (for example, the deviation degree table 53), may display a plurality of structural abnormality degree charts 52, or may display a plurality of structural abnormality degree maps 55.

[0172] Next, the CPU 23 generates image quality evaluation information for evaluating the image quality of the ophthalmic image based on the deviation degree obtained for the ophthalmic image (S47). Sometimes, when the image quality of the ophthalmic image is poor, the deviation degree also becomes high. Therefore, by generating the image quality evaluation information based on the deviation degree, the image quality of the ophthalmic image can be appropriately grasped. The specific method for generating the image quality evaluation information can be appropriately selected. As an example, in the second embodiment, the CPU 23, i.e., the control unit, obtains an index indicating the intensity or goodness of the signal of the ophthalmic image (for example, SSI (Signal Strength Index) or QI (Quality Index), etc.). The CPU 23 generates image quality evaluation information indicating poor image quality when the obtained index is below the threshold value and the average value of the deviation degree is above the threshold value. In addition, the CPU 23 may generate image quality evaluation information indicating poor image quality when the deviation degrees of all parts of the ophthalmic image are high (for example, when the deviation degrees of all parts are above the threshold value, etc.). In addition, at least any one of the deviation degree value itself, the structural abnormality degree chart 52, and the structural abnormality degree map 55 may be used as the image quality evaluation information.

[0173] The technology disclosed in the above-described embodiments is merely an example. Therefore, the technology exemplified in the above-described embodiments can also be changed. First, it is also possible to execute only a part of the multiple technologies exemplified in the above-described embodiments.

[0174] For example, in the first embodiment, the ophthalmic image processing apparatus 21 may also execute only one of the boundary detection process (see Figure 5 ) and the specific part detection process (see Figure 10 ).

[0175] For example, in the second embodiment, the ophthalmic image processing apparatus 21 may also execute only the process of obtaining the deviation degree of the two-dimensional tomographic image (S31 to S40), and omit the process of obtaining the deviation degree of the entire three-dimensional tomographic image. In this case, the ophthalmic image obtained in S31 may also be a two-dimensional tomographic image. In addition, the ophthalmic image processing apparatus 21 may use the deviation degree as the image quality evaluation information, instead of using the deviation degree as the structure information indicating the abnormality degree of the tissue structure.

[0176] In addition, in the second embodiment, the method of using the obtained deviation degree can also be changed. For example, the CPU 23 may also input the two-dimensional tomographic image with the highest deviation degree or the two-dimensional tomographic image with a deviation degree equal to or higher than the threshold value among the multiple two-dimensional tomographic images constituting the three-dimensional tomographic image into the mathematical model, and the mathematical model outputs an automatic diagnosis result related to the disease of the eye to be examined. The mathematical model is pre-trained by a machine learning algorithm to output an automatic diagnosis result based on the input ophthalmic image. In this case, it is possible to use the two-dimensional tomographic image with a high degree of structural abnormality among the multiple two-dimensional tomographic images constituting the three-dimensional tomographic image to efficiently obtain an automatic diagnosis result.

[0177] In addition, regarding the first method, the second method, and the third method, the process of obtaining the training ophthalmic image in Figure 2 S1 is an example of the "input training data acquisition step". The process of obtaining the training data in Figure 2 S2 is an example of the "output training data acquisition step". The process of training the mathematical model in Figure 2 S3 is an example of the "training step". The process of obtaining the ophthalmic image in Figure 5 S11 and Figure 10 S31 in Figure 5 is an example of the "image acquisition step". The process of obtaining the probability distribution in Figure 10 S14 and Figure 5 S32 in Figure 10 is an example of the "probability distribution acquisition step". The process of detecting the boundary or specific part in Figure 5The process of obtaining the probability map in S15 is an example of the "map obtaining step". In Figure 5 The process of obtaining the Enface image in S25 is an example of the "Enface image obtaining step".

[0178] Regarding the fourth mode, the fifth mode, and the sixth mode, in Figure 13 The process of obtaining the ophthalmic image in S21 is an example of the "image obtaining step". In Figure 13 The process of obtaining the probability distribution in S24 is an example of the "probability distribution obtaining step". In Figure 13 The process of obtaining the degree of deviation as the structure information in S25 is an example of the "structure information obtaining step".

Claims

1. An ophthalmic image processing device processes an ophthalmic image which is an image of the tissue of an eye to be examined. It is characterized in that the control unit of the ophthalmic image processing device acquires the ophthalmic image captured by an ophthalmic image capturing device, and by inputting the ophthalmic image into a mathematical model trained using a machine learning algorithm, thereby obtains a probability distribution taking at least one of one-dimensional or more coordinates of a specific boundary and a specific part where the tissue exists within a region in the ophthalmic image as a random variable, and based on the obtained probability distribution, detects at least one of the specific boundary and the specific part.

2. The ophthalmic image processing device according to claim 1, It is characterized in that the mathematical model is trained using a training data set, the training data set sets the input side as data of ophthalmic images of the tissue of the eye to be examined captured in the past, and sets the output side as data indicating the position of at least one of a specific boundary and a specific part of the tissue in the ophthalmic image on the input side.

3. The ophthalmic image processing device according to claim 1, It is characterized in that the control unit, by inputting the ophthalmic image into the mathematical model, thereby obtains a probability distribution taking a one-dimensional coordinate where the specific boundary exists in a one-dimensional region extending in a direction intersecting with the specific boundary of the tissue in the ophthalmic image as a random variable, and based on the obtained probability distribution, detects the specific boundary.

4. The ophthalmic image processing device according to claim 3, It is characterized in that the control unit detects a two-dimensional or three-dimensional boundary based on a plurality of the probability distributions obtained for each of a plurality of mutually different one-dimensional regions.

5. The ophthalmic image processing device according to claim 3 or 4, It is characterized in that the control unit obtains a two-dimensional or three-dimensional map representing the possibility of the specific boundary, which is generated based on a plurality of the probability distributions obtained for each of a plurality of the one-dimensional regions.

6. The ophthalmic image processing device according to claim 3 or 4, It is characterized in that the ophthalmic image is a three-dimensional tomographic image captured by an OCT device, the control unit, based on the three-dimensional tomographic image and the detected three-dimensional boundary, obtains a two-dimensional front view image when observing a specific layer included in the three-dimensional tomographic image from a direction along the optical axis of the measurement light.

7. The ophthalmic image processing device according to claim 5, It is characterized in that the ophthalmic image is a three-dimensional tomographic image captured by an OCT device, the control unit, based on the three-dimensional tomographic image and the detected three-dimensional boundary, obtains a two-dimensional front view image when observing a specific layer included in the three-dimensional tomographic image from a direction along the optical axis of the measurement light.

8. The ophthalmic image processing device according to claim 1 or 2, It is characterized in that The control unit obtains a probability distribution in which the two-dimensional or higher-dimensional coordinates of the specific part existing in the two-dimensional or higher-dimensional region in the ophthalmic image are used as random variables by inputting the ophthalmic image into the mathematical model, and detects the specific part based on the obtained probability distribution.

9. An OCT device that captures an ophthalmic image of the tissue by processing an OCT signal generated from reflected light of reference light and measurement light irradiated to the tissue of the eye to be examined. Characterized in that The control unit of the OCT device obtains a probability distribution in which the one-dimensional or higher-dimensional coordinates of at least one of the specific boundary and the specific part of the tissue existing in the region in the ophthalmic image are used as random variables by inputting the captured ophthalmic image into a mathematical model obtained by training using a machine learning algorithm, and detects at least one of the specific boundary and the specific part based on the obtained probability distribution.

10. A computer program product including an ophthalmic image processing program, which is executed by an ophthalmic image processing device that processes an ophthalmic image that is an image of the tissue of the eye to be examined. Wherein By the control unit of the ophthalmic image processing device executing the ophthalmic image processing program, the ophthalmic image processing device is caused to execute the following steps: An image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; A probability distribution acquisition step of obtaining a probability distribution in which the one-dimensional or higher-dimensional coordinates of at least one of the specific boundary and the specific part of the tissue existing in the region in the ophthalmic image are used as random variables by inputting the ophthalmic image into a mathematical model obtained by training using a machine learning algorithm; and A detection step of detecting at least one of the specific boundary and the specific part based on the obtained probability distribution.

Images