Ophthalmic image processing apparatus, ophthalmic image processing program, and ophthalmic image capturing apparatus

Abstract

To provide an ophthalmologic image processing device, an ophthalmologic image processing program, and an ophthalmologic imaging device that can appropriately determine the propriety of an ophthalmologic image as an analysis object.SOLUTION: A control unit of an ophthalmologic image processing device acquires an ophthalmologic image taken by an ophthalmologic imaging device (S1), and inputs the ophthalmologic image to a mathematical model (S2). The mathematical model is trained by a mechanical learning algorithm. When the ophthalmologic image is input to the mathematical model, the mathematical model executes an analysis for at least one of a specific structure and a disorder on an eye to be examined shown in the input ophthalmologic image. The control unit acquires certainty information indicating certainty of the analysis executed by the mathematical model for the input ophthalmologic image (S3). The control unit determines propriety of the ophthalmologic image input to the mathematical model as an analysis object on the basis of the acquired certainty information (S4).SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to an ophthalmic image processing apparatus used for processing an ophthalmic image of an eye to be examined, an ophthalmic image processing program, and an ophthalmic image capturing apparatus that captures an ophthalmic image of an eye to be examined.

Background Art

[0002] In recent years, techniques for acquiring medical data by analyzing an ophthalmic image of an eye to be examined have been proposed. For example, the fundus image processing apparatus described in Patent Document 1 inputs a fundus image into a mathematical model trained by a machine learning algorithm to obtain detection results of arteries and veins existing in at least a part of the fundus image. Also, techniques for obtaining an analysis result regarding the boundary of each layer of tissues shown in an ophthalmic image based on the ophthalmic image have been proposed.

Prior Art Documents

Non-Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when excluding inappropriate ophthalmic images from the analysis targets, it is useful to be able to grasp the appropriateness as an analysis target of ophthalmic images (hereinafter, may be simply referred to as "analysis appropriateness"). Here, ophthalmic images with poor image quality are often inappropriate as analysis targets. Therefore, it is also conceivable to use values indicating the image quality of ophthalmic images (for example, signal-to-noise ratio and contrast, etc.) as the analysis appropriateness. However, various factors other than image quality can also affect the appropriateness of ophthalmic images as analysis targets. For example, even if the overall image quality is good, there are still ophthalmic images that are inappropriate as analysis targets due to the presence of specific diseases, the inclusion of unnecessary objects (such as eyelids, etc.), or poor shooting ranges. Conversely, even if some of the image quality is degraded due to the inclusion of cilia, etc., there are also ophthalmic images that are appropriate as analysis targets because the necessary range for analysis is properly included.

[0005] Also, it is conceivable to determine the analysis appropriateness of ophthalmic images by using a mathematical model trained based on both ophthalmic images suitable for analysis and those not suitable for analysis. However, as described above, various factors affect the analysis appropriateness of ophthalmic images. Therefore, when training a mathematical model for determining the analysis appropriateness, it is necessary to prepare various ophthalmic images with different factors for the decrease in analysis appropriateness as ophthalmic images with low analysis appropriateness, which is not realistic. In the first place, since it is difficult to define the analysis appropriateness, etc., it is difficult to learn the analysis appropriateness itself. As described above, it has been difficult with the conventional technology to appropriately determine the analysis appropriateness of ophthalmic images.

[0006] A typical object of the present disclosure is to provide an ophthalmic image processing apparatus, an ophthalmic image processing program, and an ophthalmic image photographing apparatus that can appropriately determine the appropriateness as an analysis target of ophthalmic images.

Means for Solving the Problem

[0007] An ophthalmic image processing apparatus provided by a typical embodiment in the present disclosure is an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be examined. The control unit of the ophthalmic image processing apparatus includes an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing apparatus, and is trained by a machine learning algorithm, and by performing an analysis on at least either a specific structure or a disease of the eye to be examined shown in the input ophthalmic image For detecting at least one of the specific structure and the disease shown in the ophthalmic image an image input step of inputting the ophthalmic image to a mathematical model that outputs a probability distribution having each of a plurality of classes as a random variable, a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis performed by the mathematical model on the input ophthalmic image based on the probability distribution output by the mathematical model, and a determination step of determining the appropriateness of the ophthalmic image as an analysis target based on the acquired confidence information.

[0008] An ophthalmic image processing program provided by a typical embodiment in the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of a tissue of an eye to be examined. When the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing apparatus, an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing apparatus, and is trained by a machine learning algorithm, and by performing an analysis on at least either a specific structure or a disease of the eye to be examined shown in the input ophthalmic image For detecting at least one of the specific structure and the disease shown in the ophthalmic image an image input step of inputting the ophthalmic image to a mathematical model that outputs a probability distribution having each of a plurality of classes as a random variable, a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis performed by the mathematical model on the input ophthalmic image based on the probability distribution output by the mathematical model, and a determination step of determining the appropriateness of the ophthalmic image as an analysis target based on the acquired confidence information, are executed by the ophthalmic image processing apparatus.

[0009] The ophthalmic image capturing apparatus provided by a typical embodiment in the present disclosure includes an ophthalmic image capturing unit that captures an ophthalmic image which is an image of the tissue of an eye to be examined, and a control unit that controls the operation of the apparatus. The control unit includes an image capturing step of capturing an ophthalmic image by the ophthalmic image capturing unit, and a mathematical model that is trained by a machine learning algorithm and that outputs a probability distribution having each of a plurality of classes as a random variable, and an image input step of inputting the ophthalmic image to the mathematical model, and a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis executed by the mathematical model for the input ophthalmic image based on the probability distribution output by the mathematical model, and a determination step of determining the appropriateness of the ophthalmic image as an analysis target based on the acquired confidence information. For detecting at least one of the specific structure and the disease shown in the ophthalmic image The control unit further includes an image input step of inputting the ophthalmic image to a mathematical model that outputs a probability distribution having each of a plurality of classes as a random variable, a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis executed by the mathematical model for the input ophthalmic image based on the probability distribution output by the mathematical model, and a determination step of determining the appropriateness of the ophthalmic image as an analysis target based on the acquired confidence information.

[0010] According to the ophthalmic image processing apparatus, the ophthalmic image processing program, and the ophthalmic image capturing apparatus according to the present disclosure, the appropriateness of an ophthalmic image as an analysis target is appropriately determined.

Brief Description of Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0012] <Summary> The control unit of the ophthalmic image processing apparatus exemplified in the present disclosure executes an image acquisition step, an image input step, a confidence information acquisition step, and a determination step. In the image acquisition step, the control unit acquires an ophthalmic image captured by the ophthalmic imaging apparatus. In the image input step, the control unit inputs the ophthalmic image into a mathematical model. The mathematical model is trained by a machine learning algorithm. When the ophthalmic image is input into the mathematical model, the mathematical model executes an analysis of at least one of the specific structure and disease of the eye under examination shown in the input ophthalmic image. In the confidence information acquisition step, the control unit acquires confidence information indicating the confidence level of the analysis executed by the mathematical model for the input ophthalmic image. In the determination step, the control unit determines the appropriateness as an analysis target (hereinafter, may be simply referred to as "analysis appropriateness") of the ophthalmic image input into the mathematical model based on the acquired confidence information.

[0013] In a mathematical model trained with multiple ophthalmic images, when an ophthalmic image approximated to the ophthalmic images used for training is input, the confidence level of the analysis of the ophthalmic image tends to increase. On the other hand, when an ophthalmic image not approximated to the ophthalmic images used for training is input to the mathematical model, the confidence level of the analysis of the ophthalmic image tends to decrease. Here, when constructing a mathematical model for analyzing ophthalmic images, ophthalmic images with high analysis appropriateness are likely to be used for training the mathematical model. Therefore, when an ophthalmic image with high analysis appropriateness is input to the mathematical model as an analysis target, the confidence level of the analysis is likely to increase. That is, the correlation between the analysis appropriateness and the confidence level of each ophthalmic image becomes high.

[0014] Based on the above findings, the control unit of the ophthalmic image processing apparatus exemplified in the present disclosure determines the analysis appropriateness of an ophthalmic image based on the confidence information of the analysis performed on the ophthalmic image by a mathematical model. As a result, the analysis appropriateness affected by various factors such as image quality and shooting range is appropriately determined.

[0015] The "confidence level" may be the degree of certainty of the analysis of the ophthalmic image by the mathematical model, or may be the reciprocal of the low degree of certainty (which can also be expressed as uncertainty). Further, for example, when the uncertainty is expressed as x%, the confidence level may be a value expressed as (100 - x)%. That is, not only when using the value of the "confidence level" itself indicating the high degree of certainty of the analysis, but also when using the low degree of certainty (uncertainty) of the analysis, the analysis appropriateness of the ophthalmic image is appropriately determined. Therefore, the term "based on the confidence information" in the present disclosure includes not only the case of using the high degree of certainty but also the case of using the low degree of certainty (uncertainty). Note that the "confidence level" indicates the degree of likelihood of a prediction made using a mathematical model. The confidence level and the accuracy of the analysis result are not necessarily proportional.

[0016] The confidence level may include the entropy (average amount of information) of the probability distribution of the output in the automatic analysis by the mathematical model. Entropy indicates the spread (variation) of the probability distribution. When the confidence level in the automatic analysis reaches the maximum value, the entropy of the probability distribution becomes 0. Also, as the confidence level decreases, the entropy increases. Therefore, by using the entropy of the probability distribution as the confidence level, the analysis appropriateness is appropriately determined. Also, values other than entropy may be adopted as the confidence level. For example, at least any one of the standard deviation, coefficient of variation, variance, etc., which indicate the degree of dispersion of the probability distribution in the automatic analysis, may be used as the confidence level. The KL divergence, which is a measure for depicting the difference between probability distributions, etc., may be used as the confidence level. The maximum value of the probability distribution may be used as the confidence level. Also, when ranking a plurality of structures, etc. by automatic analysis, the difference between the probability of the first rank or the difference between the probability of the first rank and the probabilities of other ranks (for example, the sum of the probabilities of the second rank or a plurality of ranks below the second rank, etc.) may be used as the confidence level. Also, as the confidence level, the variation in the outputs between a plurality of mathematical models in which the data or conditions used for learning are different from each other may be used. In this case, it can be applied even if the output of the mathematical model is not a probability distribution.

[0017] The specific form of the confidence level information acquired in the confidence level information acquisition step can be appropriately selected. For example, based on information indicating the distribution of the confidence level (hereinafter, sometimes referred to as "confidence map") of the analysis performed by the mathematical model for each part (for example, each pixel, etc.) in the image area of the ophthalmic image, the analysis appropriateness may be determined.

[0018] The device that executes the image acquisition step, the image input step, the confidence information acquisition step, and the determination step can be appropriately selected. For example, the control unit of a personal computer (hereinafter referred to as "PC") may execute all of the image acquisition step, the image input step, the confidence information acquisition step, and the determination step. That is, the control unit of the PC may acquire an ophthalmic image from an ophthalmic imaging device and determine the analysis appropriateness of the acquired ophthalmic image. Also, the control unit of the ophthalmic imaging device may execute all of the image acquisition step, the image input step, the confidence information acquisition step, and the determination step. Further, the control units of a plurality of devices (for example, an ophthalmic imaging device and a PC, etc.) may cooperate to execute the image acquisition step, the image input step, the confidence information acquisition step, and the determination step.

[0019] The mathematical model may output a probability distribution for identifying tissues in the ophthalmic image. The confidence information may be obtained based on the probability distribution output by the mathematical model. In this case, by using the mathematical model for identifying tissues, the analysis appropriateness of the ophthalmic image is appropriately determined.

[0020] Note that the specific content of the analysis performed by the mathematical model on the ophthalmic image can also be appropriately selected. For example, from the fundus image of the eye to be examined, the analysis results of the fundus vascular tissue (the analysis result of the artery, the analysis result of the vein, or the analysis results of both the artery and the vein) may be output by the mathematical model. From the fundus image of the eye to be examined, the analysis results of tissues other than the fundus blood vessels (for example, the optic disc, etc.) may be output. Also, from the tomographic image of the tissue (for example, the fundus, etc.) of the eye to be examined, the analysis results of at least either the tissue layer or the layer boundary may be output by the mathematical model. The analysis result of the corneal endothelial cells of the eye to be examined may be output by the mathematical model.

[0021] However, the analysis performed by the mathematical model may not be an analysis of tissues. For example, the automatic analysis result indicating the presence or absence of a certain disease in the eye to be examined may be output by the mathematical model. Also, the probability of the presence of each disease, etc. may be output as the analysis result.

[0022] In addition, a plurality of analyses may be performed using a mathematical model. For example, the control unit may use confidence information of a second analysis (e.g., analysis of fundus blood vessels) executed by the mathematical model to determine the appropriateness of a first analysis (e.g., analysis of the optic nerve head) by the mathematical model. In this case, the mathematical model for executing the first analysis and the mathematical model for executing the second analysis may be the same or different.

[0023] An analysis executed by a mathematical model that inputs an ophthalmic image in the image input step is defined as a first analysis. The control unit may further execute an analysis step of executing a second analysis on the ophthalmic image according to the determination result of the analysis appropriateness after the first analysis. That is, the second analysis may be executed according to the determination result of the analysis appropriateness, which is different from the first analysis executed by the mathematical model before the analysis appropriateness is determined. In this case, after the appropriateness as the analysis target of the second analysis is determined based on the confidence information regarding the first analysis, the second analysis is executed according to the determination result. Therefore, it becomes easier to appropriately execute the second analysis.

[0024] Note that the analysis appropriateness determined in the determination step may be the appropriateness as the target of the second analysis by the mathematical model, or the appropriateness as the target of the second analysis (e.g., reading by a doctor, etc.) performed without using the mathematical model. That is, when the second analysis is executed, the mathematical model may not be used for the second analysis.

[0025] The tissue that is the target of the first analysis and the tissue that is the target of the second analysis may be the same. In this case, the analysis appropriateness in the second analysis is determined by executing the first analysis on the same tissue as the target tissue of the second analysis. Therefore, the analysis appropriateness is determined with higher accuracy compared to the case where the targets of the first analysis and the second analysis are different.

[0026] The content of the first analysis and the content of the second analysis may be different. For example, the first analysis may be an analysis of fundus vascular tissue, and the second analysis may be an analysis of the subject's condition (e.g., at least any one of the degree of arteriosclerosis, blood pressure, and age, etc.). Also, the content of the first analysis and the content of the second analysis may be the same.

[0027] In the determination step, when the confidence information about the ophthalmic image input into the mathematical model does not meet the conditions, the control unit may exclude the ophthalmic image from the analysis target (e.g., the target of the second analysis described above). In this case, the inappropriate ophthalmic image as the analysis target is automatically excluded from the analysis target. Therefore, the analysis of the ophthalmic image is executed more appropriately.

[0028] Note that a specific method for determining whether the confidence information of the ophthalmic image meets the conditions can be appropriately selected. For example, the control unit refers to the confidence map obtained for the ophthalmic image and determines whether the average value or cumulative value of the confidence within the image region is equal to or greater than a threshold value, thereby determining whether the confidence information of the ophthalmic image meets the conditions (i.e., whether the analysis appropriateness is good). In this case, the threshold value is preferably set to a value that can appropriately distinguish whether the analysis (e.g., the second analysis described above) of the ophthalmic image is properly performed.

[0029] In addition, the control unit may determine the appropriateness of the analysis of the ophthalmic image based on the confidence information within a partial region of interest among the entire image region of the ophthalmic image. In this case, by referring only to the confidence information within the region of interest necessary for the analysis among the entire image region, the determination accuracy of the analysis appropriateness is improved. For example, when the mathematical model identifies a specific tissue, the region of interest may be a region detected as a specific tissue (that is, detected with a probability of being a specific tissue higher than the probability of not being a specific tissue). As an example, when the mathematical model detects fundus blood vessels, the region of interest may be a region composed of pixels where "(probability of artery + probability of vein) > probability of background". Regions other than the specific tissue (for example, background, etc.) are more easily analyzed by the mathematical model compared to the specific tissue (that is, in a state with high certainty). Therefore, by calculating the average value of the confidence within the region in a state where regions with a low possibility of being a specific tissue are excluded, the correlation between the analysis appropriateness of the ophthalmic image and the average value of the confidence becomes even higher. Thus, the analysis appropriateness is determined with higher accuracy.

[0030] In addition, the control unit may set the region of interest in an arbitrary region within the image region. The control unit may separately determine the appropriateness of the analysis of the ophthalmic image based on the confidence information of each of the multiple regions of interest in the entire image region. In this case, since the analysis appropriateness for each region is determined separately, a more appropriate analysis result can be obtained. For example, the control unit may perform a second analysis on a region with high analysis appropriateness and omit the second analysis on a region with low analysis appropriateness. Note that at least one of the position, size, and shape of the region of interest may be determined according to an instruction input by the user. However, it goes without saying that the confidence information of the entire image region of the ophthalmic image may also be referred to.

[0031] However, it is also possible to change the usage method of the determination result of the analysis appropriateness. For example, the control unit may acquire a plurality of ophthalmic images in the image acquisition step. In the determination step, the control unit may select one or more ophthalmic images whose confidence information meets the conditions among the plurality of ophthalmic images as the objects of analysis.

[0032] Further, the control unit may notify the user of the analysis appropriateness of the ophthalmic image determined in the determination step. In this case, the user can appropriately make various judgments after grasping the analysis appropriateness of the ophthalmic image.

[0033] Also, when acquiring the confidence information, the control unit may determine whether to adopt the result of the analysis (for example, the result of the first analysis described above) performed by the mathematical model based on the confidence information. In other words, the control unit may determine whether to adopt the result of the analysis performed by the mathematical model according to whether the confidence information about the ophthalmic image input to the mathematical model satisfies the conditions. In this case, the analysis step of performing the second analysis described above can also be omitted.

[0034] The control unit of the ophthalmic image photographing apparatus exemplified in the present disclosure executes an image photographing step, an image input step, a confidence information acquisition step, and a determination step. In this case, similar to the ophthalmic image processing apparatus described above, the analysis appropriateness of the ophthalmic image is appropriately determined in the photographing apparatus.

[0035] Needless to say, various features (for example, the point that the confidence information is obtained based on the probability distribution for identifying tissues) described for the ophthalmic image processing apparatus may also be adopted in the ophthalmic image photographing apparatus.

[0036] The control unit may execute at least one of a notification step and a re-photographing step. In the notification step, the determination result of the analysis appropriateness in the determination step is notified. In the re-photographing step, when the analysis appropriateness determined in the determination step does not satisfy the conditions (or when the confidence information about the ophthalmic image does not satisfy the conditions), re-photographing of the same eye to be examined is performed. In this case, the possibility of photographing an ophthalmic image suitable for analysis is appropriately improved. Note that, as described above, various methods can be selected as the method for determining whether the analysis appropriateness satisfies the conditions (that is, whether the confidence information satisfies the conditions).

[0037] The control unit repeatedly executes the steps of taking a provisional ophthalmic image by the image capturing step, the image input step, the confidence information acquisition step, and the determination step. When the analysis appropriateness of the provisional ophthalmic image determined in the determination step satisfies the conditions (or when the confidence information about the ophthalmic image satisfies the conditions), a formal ophthalmic image may be captured by the ophthalmic image capturing unit. In this case, at the timing when the conditions for capturing an ophthalmic image suitable for analysis are satisfied, the ophthalmic image is appropriately captured by the ophthalmic image capturing unit.

[0038] Note that the provisional ophthalmic image and the formal ophthalmic image may be different types of images. For example, the provisional ophthalmic image may be an observation image continuously captured by an infrared camera (that is, an image captured in a way that the subject's eye is less likely to feel dazzled), and the formal ophthalmic image may be an image captured with visible light (for example, a color fundus image captured by a fundus camera, etc.). Also, the provisional ophthalmic image and the formal ophthalmic image may be the same type of image.

[0039] The control unit may further execute an addition average step of continuously capturing the same part of the same subject's eye in the image capturing step and obtaining an addition average image by performing addition average processing on the plurality of captured ophthalmic images. In the image input step, the confidence information acquisition step, and the determination step, the analysis appropriateness of the addition average image obtained in the addition average step may be determined. The control unit may repeat the capturing by the image capturing step and the addition average processing by the addition average step until the analysis appropriateness of the addition average image determined in the determination step satisfies the conditions (or until the confidence information about the addition average image satisfies the conditions). In this case, the capturing of the ophthalmic image and the addition average processing are repeated until the addition average image is suitable for analysis. Therefore, a suitable addition average image for analysis is appropriately obtained (captured).

[0040] <Embodiment> (Device Configuration) Hereinafter, one of the typical embodiments in the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model with a machine learning algorithm. The constructed mathematical model executes an analysis of at least one of a specific structure and a disease of the eye to be examined based on the input ophthalmic image. The ophthalmic image processing device 21 acquires an analysis result using the mathematical model, and determines the appropriateness (analysis appropriateness) as an analysis target of the ophthalmic image based on the confidence information of the analysis executed by the mathematical model. The ophthalmic image capturing devices 11A and 11B capture an ophthalmic image that is an image of the tissue of the eye to be examined.

[0041] As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although details will be described later, the mathematical model construction device 1 uses the data of the ophthalmic image of the eye to be examined (hereinafter referred to as "training ophthalmic image") acquired from the ophthalmic image capturing device 11A and the data indicating at least one of the structure and the disease of the eye to be examined in which the training ophthalmic image was captured to train the mathematical model. As a result, a mathematical model is constructed. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic image capturing device 11A may function as the mathematical model construction device 1. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic image capturing device 11A) may cooperate to construct a mathematical model.

[0042] In addition, a PC is used for the ophthalmic image processing device 21 of the present embodiment. However, the device that can function as the ophthalmic image processing device 21 is not limited to the PC. For example, the ophthalmic image capturing device 11B or a server or the like may function as the ophthalmic image processing device 21. When the ophthalmic image capturing device 11B functions as the ophthalmic image processing device 21, the ophthalmic image capturing device 11B can determine the appropriateness of the analysis of the captured ophthalmic image while capturing the ophthalmic image. Further, the ophthalmic image capturing device 11B can also perform various processes such as re-capturing based on the determination result of the appropriateness of the analysis (details of these will be described later). Further, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmic image processing device 21. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmic image capturing device 11B) may cooperate to perform various processes.

[0043] In addition, in the present embodiment, the case where a CPU is used as an example of a controller that performs various processes will be exemplified. However, it goes without saying that a controller other than the CPU may be used for at least a part of various devices. For example, by adopting a GPU as the controller, the processing speed may be increased.

[0044] The mathematical model construction device 1 will be described. The mathematical model construction device 1 is arranged, for example, in the ophthalmic image processing device 21 or a manufacturer or the like that provides an ophthalmic image processing program to the user. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller that controls and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 3) described later. Further, the communication I / F 5 connects the mathematical model construction device 1 to other devices (for example, the ophthalmic image capturing device 11A and the ophthalmic image processing device 21, etc.).

[0045] The mathematical model construction device 1 is connected to the operation unit 7 and the display device 8. The operation unit 7 is operated by the user to input various instructions to the mathematical model construction device 1. At least one of, for example, a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 7. Note that, together with the operation unit 7 or instead of the operation unit 7, a microphone or the like for inputting various instructions may be used. The display device 8 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used for the display device 8. Note that the "image" in the present disclosure includes both still images and moving images.

[0046] The mathematical model construction device 1 can acquire data of an ophthalmic image (hereinafter, may be simply referred to as "ophthalmic image") from the ophthalmic image photographing device 11A. The mathematical model construction device 1 may acquire the data of the ophthalmic image from the ophthalmic image photographing device 11A by at least one of, for example, wired communication, wireless communication, a removable storage medium (for example, a USB memory), etc.

[0047] The ophthalmic image processing device 21 will be described. The ophthalmic image processing device 21 is arranged, for example, in a facility (for example, a hospital or a health examination facility, etc.) that performs diagnosis or examination of a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller in charge of control and a storage device 24 that can store programs, data, etc. A ophthalmic image processing program for executing an analysis appropriateness determination process (see FIG. 6) described later is stored in the storage device 24. The ophthalmic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (for example, the ophthalmic image photographing device 11B and the mathematical model construction device 1, etc.).

[0048] The ophthalmic image processing device 21 is connected to the operation unit 27 and the display device 28. Similar to the operation unit 7 and the display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0049] The ophthalmic image processing device 21 can acquire an ophthalmic image from the ophthalmic image capturing device 11B. The ophthalmic image processing device 21 may acquire an ophthalmic image from the ophthalmic image capturing device 11B by, for example, at least any one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), etc. Further, the ophthalmic image processing device 21 may acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

[0050] The ophthalmic image capturing device 11 (11A, 11B) will be described. As an example, in the present embodiment, the case where the ophthalmic image capturing device 11A that provides an ophthalmic image to the mathematical model construction device 1 and the ophthalmic image capturing device 11B that provides an ophthalmic image to the ophthalmic image processing device 21 are used will be described. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a plurality of ophthalmic image capturing devices. Further, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from one common ophthalmic image capturing device.

[0051] The ophthalmic image capturing device 11 (11A, 11B) will be described. Various devices for capturing an image of the tissue of the eye to be examined can be used for the ophthalmic image capturing device 11. As an example, the ophthalmic image capturing device 11 used in the present embodiment is a fundus camera capable of capturing a two-dimensional color frontal image of the fundus using visible light. However, a device other than a fundus camera (e.g., at least any one of an OCT device, a scanning laser ophthalmoscope (SLO), a corneal endothelial cell imaging device, etc.) may be used. The ophthalmic image may be a two-dimensional frontal image of the tissue of the eye to be examined captured from the frontal side of the eye to be examined, or a three-dimensional image of the tissue.

[0052] The ophthalmic imaging device 11 includes a control unit 12 (12A, 12B) that performs various control processes and an ophthalmic imaging unit 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) which is a controller for controlling, and a storage device 14 (14A, 14B) capable of storing programs, data, etc. The ophthalmic imaging control program executed by the CPU 13 may be stored in the storage device 14. As described above, the ophthalmic imaging device 11 can also determine the appropriateness of ophthalmic image analysis. In this case, the ophthalmic imaging control program for the ophthalmic imaging device 11 to execute imaging control processes (see FIGS. 7 to 9) includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The ophthalmic imaging control program may be stored in the storage device 14. The ophthalmic imaging unit 16 includes optical members and the like for imaging an ophthalmic image of the eye to be examined.

[0053] (Mathematical model construction process) With reference to FIGS. 2 and 3, the mathematical model construction process executed by the mathematical model construction device 1 will be described. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

[0054] In the mathematical model construction process, a mathematical model is constructed that executes an analysis of at least one of a specific structure and a disease of the eye to be examined shown in the ophthalmic image. As an example, in the present embodiment, a mathematical model is exemplified that outputs an analysis result (in the present embodiment, an image of blood vessels in the input fundus) of the fundus blood vessels, which is one of the structures of the eye to be examined, by analyzing the input ophthalmic image. In other words, the mathematical model of the present embodiment detects a specific structure (fundus blood vessels) of the eye to be examined shown in the input ophthalmic image. However, as described above, the mathematical model may output an analysis result different from the analysis result of the fundus blood vessels (for example, tissue layers and boundaries, etc.).

[0055] In this embodiment, a convolutional neural network is used as the mathematical model. The mathematical model of this embodiment is trained to output a probability distribution having a class (a pixel of an artery, a pixel of a vein, or another pixel) as a random variable for each pixel constituting the ophthalmic image. By obtaining the class that takes the maximum value for each pixel with respect to the output of the mathematical model, arteries and veins are detected (in this embodiment, an artery blood vessel image and a vein blood vessel image are obtained). Note that blood vessel images of arteries and veins may be created by extracting the probabilities corresponding to the arteries and veins of each pixel, respectively.

[0056] Note that, although details will be described later, when determining the analysis appropriateness of an ophthalmic image, confidence information of the analysis performed by the mathematical model is used. In this embodiment, by calculating the entropy for each pixel with respect to the output of the mathematical model, a confidence map indicating the distribution of certainty or uncertainty of each pixel is obtained. The average value in a specific region of the confidence map is used as the confidence information about the ophthalmic image input to the mathematical model.

[0057] In the mathematical model construction process, the mathematical model is constructed by training the mathematical model with a training data set. The training data set includes input-side data (input training data) and output-side data (output training data). Hereinafter, as an example, a case will be described in which a two-dimensional color front image captured by a fundus camera is input to the mathematical model as an input image, and the mathematical model outputs a blood vessel image.

[0058] FIG. 2 shows an example of input training data and output training data when a two-dimensional color frontal image is used as an input image to cause a mathematical model to output a blood vessel image. In the example shown in FIG. 2, an ophthalmic image 30, which is a two-dimensional color frontal image captured by an ophthalmic image capturing device (a fundus camera in this embodiment) 11A, is used as the input training data. In this embodiment, the image area of the ophthalmic image 30 used as the input training data includes both the optic nerve head 31 and the macula 32 of the eye to be examined. Also, blood vessel images 40A and 40B, which are images showing at least either an artery or a vein in the ophthalmic image 30 used as the input training data, are used as the output training data. The output training data may be generated, for example, according to an instruction input by an operator. In this case, the operator may input an instruction while referring to the fundus blood vessels shown in the ophthalmic image 30 used as the input training data.

[0059] In the example shown in FIG. 2, the blood vessel image 40A of the artery in the ophthalmic image 30 and the blood vessel image 40B of the vein in the ophthalmic image 30 are prepared separately. However, one blood vessel image showing both the artery and the vein may be used as the output training data. Also, when only the blood vessel image of the artery is output to the mathematical model, only the blood vessel image 40A of the artery may be used as the output training data. Similarly, when only the blood vessel image of the vein is output to the mathematical model, only the blood vessel image 40B of the vein may be used as the output training data.

[0060] Referring to FIG. 3, the mathematical model construction process will be described. The CPU 3 acquires the ophthalmic image 30 captured by the ophthalmic image capturing device 11A as the input training data (S1). In this embodiment, the data of the ophthalmic image 30 is acquired by the mathematical model construction device 1 after being generated by the ophthalmic image capturing device 11A. However, the CPU 3 may acquire a signal (for example, a light reception signal by a light receiving element, etc.) that is the basis for generating the ophthalmic image 30 from the ophthalmic image capturing device 11A, and generate the ophthalmic image 30 based on the acquired signal to acquire the data of the ophthalmic image 30.

[0061] Next, the CPU 3 acquires, as output training data, data indicating at least one of the structure and disease of the eye to be examined in which the ophthalmic image 30 was captured (in this embodiment, the fundus blood vessels which are one of the structures) (S2). As described above, the output training data in this embodiment are the blood vessel images 40A and 40B indicating at least one of the arteries and veins in the ophthalmic image 30.

[0062] Next, the CPU 3 executes training of a mathematical model using a training data set by means of a machine learning algorithm (S3). As the machine learning algorithm, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

[0063] A neural network is a method that mimics the behavior of the neural cell network of a living organism. Examples of neural networks include feedforward (forward propagation type) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0064] A random forest is a method of learning based on randomly sampled training data to generate a number of decision trees. When using a random forest, follow the branches of a plurality of decision trees that have been learned in advance as discriminators and take the average (or majority vote) of the results obtained from each decision tree.

[0065] Boosting is a technique for generating a strong discriminator by combining a plurality of weak discriminators. A strong discriminator is constructed by sequentially learning simple and weak discriminators.

[0066] SVM is a method for constructing a two-class pattern classifier using linear input elements. SVM learns the parameters of the linear input elements based on a criterion (hyperplane separation theorem) of finding a hyperplane that maximizes the margin, which is the maximum distance from each data point, for example, from training data.

[0067] A mathematical model refers to a data structure for predicting the relationship between input data (in this embodiment, data of ophthalmic images similar to the training ophthalmic images) and output data (in this embodiment, data of analysis results regarding fundus blood vessels), for example. The mathematical model is constructed by being trained using a training data set. As described above, the training data set is a set of input training data and output training data. For example, through training, the correlation data (for example, weights) between each input and output are updated.

[0068] In this embodiment, a multi-layer neural network is used as the machine learning algorithm. A neural network includes an input layer for inputting data, an output layer for generating data of the analysis result to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used.

[0069] Note that other machine learning algorithms may be used. For example, a generative adversarial network (GAN) that utilizes two competing neural networks may be adopted as the machine learning algorithm.

[0070] Until the construction of the mathematical model is completed (S4: NO), the processes of S1 to S3 are repeated. When the construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. The program and data for realizing the constructed mathematical model are incorporated into the ophthalmic image processing apparatus 21 (in other embodiments, the ophthalmic image capturing apparatus 11B).

[0071] (Correlation between confidence level and analytical adequacy) With reference to FIGS. 4 and 5, the relationship between the confidence level of the analysis of ophthalmic images executed by a mathematical model and the analytical adequacy of ophthalmic images will be described. The confidence level indicates the degree of likelihood of a prediction made using the mathematical model. As described above, the confidence level may be a high degree of certainty of the analysis, or may be the reciprocal of a low degree of certainty (uncertainty), etc. Therefore, for example, the term "high certainty" is synonymous with the term "low uncertainty".

[0072] As described above, the mathematical model used in this embodiment outputs a probability distribution for identifying tissues in an ophthalmic image as an analysis result for the ophthalmic image. By calculating the entropy for each pixel with respect to the output of the mathematical model, a confidence map indicating the distribution of certainty or uncertainty for each pixel can be obtained. That is, the confidence map can be expressed as information indicating the distribution of the confidence level of the analysis performed by the mathematical model for each pixel constituting the ophthalmic image. Below, the relationship between the average value of the confidence level at each pixel within the image region of the ophthalmic image and the analytical adequacy of the ophthalmic image will be considered.

[0073] The analytical adequacy as an object of analysis of an ophthalmic image can be affected not only by the image quality of the ophthalmic image but also by factors other than the image quality. Therefore, it is difficult to obtain high accuracy even if the analytical adequacy of an ophthalmic image is determined based only on specific parameters of the ophthalmic image (for example, only the image quality). Also, as described above, it is unrealistic to construct a mathematical model for determining the analytical adequacy of an ophthalmic image. Therefore, it is desirable that the analytical adequacy of an ophthalmic image can be determined more easily and appropriately.

[0074] Here, the inventor of the present invention focused on the correlation between the confidence level of the analysis of ophthalmic images executed by a mathematical model and the appropriateness of the analysis of ophthalmic images. That is, in a mathematical model, when an ophthalmic image approximated to the ophthalmic images used for training (for example, the ophthalmic image 30 shown in FIG. 2) is input, the confidence level of the analysis of the ophthalmic image tends to increase. On the other hand, when an ophthalmic image not approximated to the ophthalmic images used for training is input to the mathematical model, the confidence level of the analysis of the ophthalmic image tends to decrease. Here, when constructing a mathematical model for analyzing ophthalmic images, ophthalmic images with high analysis appropriateness are likely to be used for training the mathematical model. Therefore, when an ophthalmic image with high analysis appropriateness is input to the mathematical model as an analysis target, the confidence level of the analysis is likely to be high. Thus, the correlation between the analysis appropriateness and the confidence level of each ophthalmic image becomes high.

[0075] FIG. 4 is a graph showing the relationship between confidence level information and frequency when analysis (analysis of fundus blood vessels in the present embodiment) by a mathematical model is performed on a plurality of ophthalmic images. Specifically, the horizontal axis in FIG. 4 indicates uncertainty, which is a type of confidence level information. Specifically, the uncertainty in FIG. 4 is the average value of the uncertainties of a plurality of pixels (in the present embodiment, a plurality of pixels for which “(probability of being an artery + probability of being a vein)> probability of being a background”) that are analyzed to have a higher probability of being fundus blood vessel tissue among the respective pixels constituting the ophthalmic image. The vertical axis in FIG. 4 indicates the number (frequency) of ophthalmic images for each uncertainty. FIG. 5 is a diagram listing the ophthalmic images in which the confidence level (degree of certainty) was within the lower 15th rank in the analysis shown in FIG. 4.

[0076] As shown in FIG. 4, as a result of performing analysis on a plurality of ophthalmic images, a peak in frequency appeared in the range where the uncertainty was about 0.3 to about 0.8. When checking the ophthalmic images included in the peak, all of them were images suitable for analysis. On the other hand, as exemplified in FIG. 5, when checking the ophthalmic images with high analysis uncertainty (that is, the ophthalmic images with low analysis certainty), all of the ophthalmic images were images not suitable for analysis due to various influences such as poor image quality and inclusion of diseases.

[0077] As described above, the correlation between the confidence level of the analysis of ophthalmic images executed by the mathematical model and the appropriateness of the analysis of ophthalmic images is high. Therefore, in the present embodiment, based on the confidence information of the analysis executed by the mathematical model for the ophthalmic image, the appropriateness of the analysis of the ophthalmic image is determined. As a result, the appropriateness of the analysis affected by various elements is appropriately determined. For example, referring to FIG. 4, by using about 1.0 (for example, about 0.9 to about 1.1) as the uncertainty threshold that can appropriately classify whether the analysis is appropriately performed, it is also possible to determine whether the ophthalmic image is suitable for analysis.

[0078] (Analysis Appropriateness Determination Process) With reference to FIG. 6, the analysis appropriateness determination process executed by the ophthalmic image processing apparatus 21 will be described. The analysis appropriateness determination process illustrated in FIG. 6 is executed by the CPU 23 of the ophthalmic image processing apparatus 21 according to the ophthalmic image processing program stored in the storage device 24.

[0079] First, the CPU 23 acquires an ophthalmic image of the eye to be examined taken by the ophthalmic image capturing apparatus (a fundus camera in the present embodiment) 11B (S1). The ophthalmic image acquired in S1 is the same type of image as the ophthalmic image 30 (see FIG. 2) used as input training data when training the mathematical model (that is, an image taken by the same type of ophthalmic image capturing apparatus).

[0080] Next, the CPU 23 inputs the ophthalmic image acquired in S1 to the mathematical model trained by the machine learning algorithm (S2). As described above, the mathematical model executes an analysis of at least either the specific structure or the disease of the eye to be examined shown in the input ophthalmic image. Specifically, the mathematical model of the present embodiment outputs a probability distribution for identifying tissues in the ophthalmic image. In the present embodiment, the mathematical model executes an analysis of the fundus blood vessels of the eye to be examined (the first analysis in the present embodiment).

[0081] The CPU 23 acquires confidence information associated with the analysis (S3). The confidence information is obtained based on the probability distribution output by the mathematical model. As described above, in the present embodiment, the average value in a specific region of the confidence map indicating the distribution of certainty or uncertainty of each pixel is calculated as the confidence information. Next, the CPU 23 determines whether the confidence information acquired in S3 satisfies the condition (S4). As an example, in the present embodiment, the CPU 23 determines whether the average value of the confidence (degree of certainty) in a specific region within the image region of the ophthalmic image is equal to or greater than a threshold value (or determines whether the average value of uncertainty is equal to or less than the threshold value) to determine whether the confidence information satisfies the condition. The threshold value is set to a value that can appropriately classify whether the analysis of the ophthalmic image is properly performed.

[0082] Note that in S4 of the present embodiment, based on the confidence information (in the present embodiment, the average value of the confidence) within a partial target region among the entire image region of the ophthalmic image acquired in S1, the analysis appropriateness of the ophthalmic image is determined. As a result, only the confidence information within the target region necessary for the analysis is referred to, so the determination accuracy of the analysis appropriateness is improved. Specifically, in the present embodiment, a region composed of pixels analyzed by the mathematical model to have a higher probability of being a specific tissue (fundus vascular tissue) (pixels for which "(probability of being an artery + probability of being a vein)>probability of being a background") is set as the target region. As a result, the average value of the confidence within the region is calculated in a state where an area where the analysis is easy (in the present embodiment, the background region where no blood vessels exist) is excluded, so the correlation between the analysis appropriateness and the average value of the confidence becomes even higher. However, the target region may be set in any region within the image region. Also, a partial arbitrary region within the image region and a region analyzed to have a higher probability of being a specific tissue may be set as the target region. In this case, the determination accuracy of the analysis appropriateness is further improved. Also, the confidence information of the entire image region of the ophthalmic image may be referred to.

[0083] When the confidence information about the ophthalmic image obtained in S1 satisfies the condition (S4: YES), the CPU 23 determines that the analysis appropriateness of the ophthalmic image obtained in S1 is good and executes a second analysis on the ophthalmic image (S7). That is, the CPU 23 executes a second analysis on the ophthalmic image according to the determination result of the analysis appropriateness in S4 after the first analysis executed by the mathematical model in S2 (S7). Therefore, the second analysis is appropriately executed for the ophthalmic image with high analysis appropriateness.

[0084] In this embodiment, the target tissue of the first analysis executed by the mathematical model in S2 and the target tissue of the second analysis executed in S7 are the same. In this case, the analysis appropriateness in the second analysis is determined by executing the first analysis on the same tissue as the target tissue of the second analysis. Therefore, the analysis appropriateness of the ophthalmic image is determined with higher accuracy compared to the case where the targets of the first analysis and the second analysis are different.

[0085] As an example, in this embodiment, the content of the first analysis (analysis of fundus vascular tissue) and the second analysis (analysis of at least any one of the degree of arteriosclerosis, blood pressure, and age of the subject) are different. However, when executing the second analysis, the content of the first analysis and the content of the second analysis may be the same.

[0086] When the confidence information does not satisfy the condition (that is, when it is determined that the analysis appropriateness of the ophthalmic image is not good) (S4: NO), the CPU 23 excludes the ophthalmic image obtained in S1 from the analysis target (S8).

[0087] It is also possible to change the specific method for determining the appropriateness of ophthalmic image analysis. For example, the CPU 23 may determine whether the appropriateness of ophthalmic image analysis is good or not in multiple steps (for example, by generating a score indicating the appropriateness of analysis) instead of in two steps. Also, the CPU 23 may separately determine the appropriateness of ophthalmic image analysis based on the confidence information of each of a plurality of regions of interest in the entire image region. The CPU 23 may perform a second analysis on regions with high appropriateness of analysis and omit the second analysis on regions with low appropriateness of analysis.

[0088] Also, as described above, it is possible for the ophthalmic image capturing device 11 to determine the appropriateness of ophthalmic image analysis. In this case, the program executed by the ophthalmic image capturing device 11 includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. Hereinafter, the first to third shooting control processes for the ophthalmic image capturing device 11 to determine the appropriateness of ophthalmic image analysis will be described.

[0089] (First shooting control process) With reference to FIG. 7, the first shooting control process executed by the ophthalmic image capturing device 11 of the first modification example will be described. First, the CPU 13 captures an ophthalmic image of the eye to be examined by the ophthalmic image capturing unit 16 and acquires data (S11). The CPU 13 inputs the ophthalmic image captured in S11 into a mathematical model trained by a machine learning algorithm (S12). Also, the CPU 13 acquires confidence information associated with the analysis (S13).

[0090] The CPU 13 determines the appropriateness of the ophthalmic image captured in S11 based on the confidence information acquired in S13 (S14). As described above, the appropriateness of analysis may be determined in two steps or in multiple steps. Next, the CPU 13 notifies the determination result of the appropriateness of analysis performed in S14 (S15). As a result, the user can appropriately grasp the appropriateness of the captured ophthalmic image analysis.

[0091] Further, the CPU 13 determines whether or not the analysis appropriateness determined in S14 satisfies the condition (or whether or not the confidence information acquired in S13 satisfies the condition) (S16). If the analysis appropriateness does not satisfy the condition (for example, when the average value of the confidence levels within the image region is less than the threshold) (S16: NO), the process returns to S11 and re - imaging of the eye to be examined is performed. As a result, the possibility of capturing an ophthalmic image suitable for analysis is improved. If the analysis appropriateness satisfies the condition (for example, when the average value of the confidence levels is equal to or greater than the threshold) (S16: YES), the process ends as it is.

[0092] (Second imaging control process) Referring to FIG. 8, the second imaging control process executed by the ophthalmic image capturing apparatus 11 of the second modification will be described. Note that the ophthalmic image capturing apparatus 11 that executes the second imaging control process can capture both an observation ophthalmic image, which is a provisional ophthalmic image, and an analysis ophthalmic image, which is a formal analysis target, of the same tissue of the eye to be examined. As an example, the ophthalmic image capturing apparatus 11 of the second modification can continuously capture an observation ophthalmic image of the fundus of the eye to be examined by an infrared camera and capture an analysis ophthalmic image of the fundus by visible light.

[0093] As shown in FIG. 8, the CPU 13 captures an observation ophthalmic image of the eye to be examined by the ophthalmic image capturing unit 16 and acquires data (S21). The CPU 13 inputs the observation ophthalmic image captured in S21 into a mathematical model trained by a machine learning algorithm (S22). Further, the CPU 13 acquires confidence information associated with the analysis (S23).

[0094] Next, the CPU 13 determines whether the analysis adequacy of the observation ophthalmic image captured in S21 meets the conditions (or whether the confidence information obtained in S23 meets the conditions) (S24). If the analysis adequacy of the observation ophthalmic image does not meet the conditions (for example, when the average value of the confidence levels within the image area is less than the threshold) (S24: NO), it is highly likely that the conditions for capturing an ophthalmic image suitable for analysis are not met. Therefore, the process returns to S21, and the processes of S21 to S24 are repeated. When the analysis adequacy of the observation ophthalmic image meets the conditions (S24: YES), the CPU 13 captures an analysis ophthalmic image of the subject eye by the ophthalmic image capturing unit 16 (S25). As a result, the analysis ophthalmic image is appropriately captured at the timing when the conditions for capturing an ophthalmic image suitable for analysis are satisfied.

[0095] (Third shooting control process) With reference to FIG. 9, the third shooting control process executed by the ophthalmic image capturing apparatus 11 of the third modification will be described. Note that the ophthalmic image capturing apparatus 11 that executes the third shooting control process can capture the same part of the same subject eye continuously and obtain an added average image by performing an added average process on a plurality of captured ophthalmic images. Generally, as the number of ophthalmic images used for the added average process increases, the analysis adequacy of the added average image tends to increase. As an example, the ophthalmic image capturing apparatus 11 of the third modification is an OCT apparatus that can capture an image of the tissue of the subject eye (for example, a tomographic image of the fundus tissue, etc.), unlike the fundus camera exemplified in the above embodiment.

[0096] As shown in FIG. 9, the CPU 13 captures an ophthalmic image of a predetermined part of the subject eye by the ophthalmic image capturing unit 16 and acquires data (S31). Next, the CPU 13 obtains an added average image by performing an added average process on a plurality of ophthalmic images captured for the same predetermined part (S32). The CPU 13 inputs the added average image obtained in S32 to a mathematical model trained by a machine learning algorithm (S33). Further, the CPU 13 acquires confidence information associated with the analysis (S34).

[0097] Next, the CPU 13 determines whether the analysis appropriateness of the addition-average image acquired in S32 satisfies the condition (or whether the confidence information acquired in S34 satisfies the condition) (S35). If the analysis appropriateness of the addition-average image does not satisfy the condition (for example, when the average value of the confidence levels within the image area is less than the threshold) (S35: NO), it is highly likely that an addition-average image suitable for analysis has not yet been obtained. Therefore, the process returns to S31, and the processes of S31 to S35 are repeated. When the analysis appropriateness of the addition-average image satisfies the condition (S35: YES), the latest addition-average image is stored as the image for analysis (S36), and the addition-average process ends. As a result, an addition-average image suitable for analysis is appropriately acquired (captured).

[0098] The technologies disclosed in the above embodiments and modified examples are merely examples. Therefore, it is also possible to change the technologies exemplified in the above embodiments and modified examples. For example, it is also possible to execute only a part of the plurality of technologies exemplified in the above embodiments and modified examples. Further, it is also possible to execute a combination of the technologies exemplified in each of the above embodiments and modified examples.

[0099] Note that the process of acquiring an ophthalmic image in S1 of FIG. 6, S11 of FIG. 7, S21 of FIG. 8, and S31 of FIG. 9 is an example of the "image acquisition step". The process of capturing an ophthalmic image in S11 of FIG. 7, S21 of FIG. 8, and S31 of FIG. 9 is an example of the "image capture step". The process of inputting an ophthalmic image into a mathematical model in S2 of FIG. 6, S12 of FIG. 7, S22 of FIG. 8, and S33 of FIG. 9 is an example of the "image input step". The process of acquiring confidence information in S3 of FIG. 6, S13 of FIG. 7, S23 of FIG. 8, and S34 of FIG. 9 is an example of the "confidence information acquisition step". The process of determining the analysis appropriateness in S4 of FIG. 6, S14, S16 of FIG. 7, S24 of FIG. 8, and S35 of FIG. 9 is an example of the "determination step". The process of notifying the determination result in S13 of FIG. 7 is an example of the "notification step". In S16 of FIG. 7, the process of executing re-capture when the analysis appropriateness does not satisfy the condition is an example of the "re-capture step". The process of acquiring an addition-average image in S32 of FIG. 9 is an example of the "addition-average step".

Description of Symbols

[0100] 11A, 11B Ophthalmic imaging device 13A, 13B CPU 14A, 14B Storage device 16A, 16B Ophthalmic imaging unit 21 Ophthalmic image processing device 23 CPU 24 Storage device 30 Ophthalmic image 40A, 40B Vascular image

Claims

1. An ophthalmic image processing apparatus that processes an ophthalmic image which is an image of the tissue of an eye to be examined, wherein a control unit of the ophthalmic image processing apparatus performs an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing apparatus, an image input step of inputting the ophthalmic image to a mathematical model that is trained by a machine learning algorithm and outputs a probability distribution in which each of a plurality of classes for detecting at least any one of a specific structure and a disease of the eye to be examined shown in the input ophthalmic image is a random variable, and performs analysis on at least any one of the specific structure and the disease shown in the ophthalmic image, a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis performed by the mathematical model on the input ophthalmic image, based on the probability distribution output by the mathematical model, and a determination step of determining the appropriateness of the ophthalmic image as an analysis target, based on the acquired confidence information. An ophthalmic image processing apparatus characterized by performing the above steps.

2. The ophthalmic image processing apparatus according to Claim 1, wherein the mathematical model into which the ophthalmic image is input in the image input step outputs a probability distribution for identifying tissues in the ophthalmic image, and the confidence information is obtained based on the probability distribution output by the mathematical model.

3. The ophthalmic image processing apparatus according to Claim 1 or 2, further comprising an analysis step of performing a second analysis on the ophthalmic image according to the determination result in the determination step, after the first analysis performed by the mathematical model into which the ophthalmic image is input in the image input step.

4. The ophthalmic image processing apparatus according to Claim 3, wherein the tissue that is the target of the first analysis and the tissue that is the target of the second analysis are the same.

5. The ophthalmic image processing apparatus according to any one of Claims 1 to 4, wherein the control unit excludes the ophthalmic image from the analysis target when the confidence information about the ophthalmic image input to the mathematical model does not meet the conditions in the determination step.

6. An ophthalmic image processing program executed by an ophthalmic image processing apparatus that processes an ophthalmic image which is an image of the tissue of an eye to be examined, wherein when the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing apparatus, an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing apparatus; an image input step of inputting the ophthalmic image to a mathematical model that is trained by a machine learning algorithm and outputs a probability distribution having each of a plurality of classes for detecting at least one of a specific structure and a disease of the eye to be examined shown in the input ophthalmic image as a random variable, by performing an analysis on at least one of the specific structure and the disease of the eye to be examined shown in the ophthalmic image; a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis performed by the mathematical model on the input ophthalmic image, based on the probability distribution output by the mathematical model; a determination step of determining the appropriateness of the ophthalmic image as an analysis target, based on the acquired confidence information; and causing the ophthalmic image processing apparatus to execute the steps, which is characterized in that it is an ophthalmic image processing program.

7. An ophthalmic image capturing apparatus comprising: an ophthalmic image capturing unit that captures an ophthalmic image which is an image of the tissue of an eye to be examined; and a control unit that controls the operation of the apparatus, wherein the control unit performs an image capturing step of capturing an ophthalmic image by the ophthalmic image capturing unit; an image input step of inputting the ophthalmic image to a mathematical model that is trained by a machine learning algorithm and outputs a probability distribution having each of a plurality of classes for detecting at least one of a specific structure and a disease of the eye to be examined shown in the input ophthalmic image as a random variable, by performing an analysis on at least one of the specific structure and the disease of the eye to be examined shown in the ophthalmic image; a confidence information acquisition step of acquiring confidence information indicating the degree of certainty of the analysis performed by the mathematical model on the input ophthalmic image, based on the probability distribution output by the mathematical model; and a determination step of determining the appropriateness of the ophthalmic image as an analysis target, based on the acquired confidence information. The ophthalmic image capturing apparatus is characterized by performing the steps.

Images