Fundus image processing apparatus and fundus image processing program

Abstract

To provide a fundus image processing device and fundus image processing program which can highly accurately detect a specific portion in a fundus image.SOLUTION: A control unit of a fundus image processing device executes an image acquisition step (S31), a deviation degree acquisition step (S37-S47); and a portion identification step (S49). In the image acquisition step, the control unit acquires a fundus image captured by a fundus imaging device. In the deviation degree acquisition step, the control unit acquires the probability distribution for identifying a first portion of the fundus captured in the fundus image by inputting the fundus image to a mathematical model trained with a machine learning algorithm and acquires the deviation degree of the acquired probability distribution to the probability distribution in a case where the first portion is correctly identified. In the portion identification step, the control unit identifies a second portion different from the first portion of the fundus on the basis of the deviation degree.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present disclosure relates to a fundus image processing apparatus and a fundus image processing program used for processing a fundus image of an eye to be examined.

Background Art

[0002] In recent years, techniques for identifying specific sites in the fundus by analyzing a fundus image of an eye to be examined have been proposed. For example, the ophthalmic imaging apparatus described in Patent Document 1 performs image processing (such as edge detection or Hough transform) on a frontal image of the fundus of the eye to be examined, thereby detecting the position of the optic disc (hereinafter, may be simply referred to as "disc") of the fundus shown in the frontal image and the position of the macula.

Prior Art Documents

Non-Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When identifying a specific site in the fundus by performing image processing or the like on a fundus image, the identification accuracy may decrease due to various influences (for example, the presence or absence of eye diseases). For example, it may be difficult to identify a specific site from a fundus image due to the influence of a disease existing in the fundus. The presence of cataracts or the imaging environment of the image may also affect the identification accuracy of a specific site. Therefore, a technique capable of detecting a specific site shown in a fundus image with high accuracy regardless of the presence or absence of eye diseases is desired.

[0005] A typical object of the present disclosure is to provide a fundus image processing apparatus and a fundus image processing program capable of detecting a specific site shown in a fundus image with high accuracy.

Means for Solving the Problems

[0006] A fundus image processing apparatus provided by a typical embodiment in the present disclosure is a fundus image processing apparatus that processes a fundus image of an eye to be examined. The control unit of the fundus image processing apparatus includes an image acquisition step of acquiring a fundus image captured by a fundus image photographing apparatus, and by inputting the fundus image into a mathematical model trained by a machine learning algorithm, obtaining a probability distribution for identifying a first part of the fundus shown in the fundus image, and obtaining a degree of divergence of the obtained probability distribution with respect to the probability distribution when the first part is accurately identified. A divergence degree acquisition step, and a part identification step of identifying a second part of the fundus different from the first part based on the divergence degree.

[0007] A fundus image processing program provided by a typical embodiment in the present disclosure is a fundus image processing program executed by a fundus image processing apparatus that processes a fundus image of an eye to be examined. By the fundus image processing program being executed by the control unit of the fundus image processing apparatus, an image acquisition step of acquiring a fundus image captured by a fundus image photographing apparatus, and by inputting the fundus image into a mathematical model trained by a machine learning algorithm, obtaining a probability distribution for identifying a first part of the fundus shown in the fundus image, and obtaining a degree of divergence of the obtained probability distribution with respect to the probability distribution when the first part is accurately identified. A divergence degree acquisition step, and a part identification step of identifying a second part of the fundus different from the first part based on the divergence degree, are executed by the fundus image processing apparatus.

[0008] According to the fundus image processing apparatus and the fundus image processing program according to the present disclosure, a specific part shown in the fundus image is detected with high accuracy.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiment for Carrying Out the Invention

[0010] <Summary> (First Aspect) The control unit of the fundus image processing apparatus exemplified in the present disclosure executes an image acquisition step, a divergence degree acquisition step, and a site identification step. In the image acquisition step, the control unit acquires a fundus image captured by a fundus image photographing apparatus. In the divergence degree acquisition step, the control unit inputs the fundus image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying a first site of the fundus shown in the fundus image, and obtains the divergence degree of the obtained probability distribution with respect to the probability distribution when the first site is accurately identified. In the site identification step, the control unit identifies a second site different from the first site in the fundus based on the divergence degree.

[0011] In the fundus, the state of the first site may change between the position where the second site exists and the position where the second site does not exist. For example, between the position where the papilla (second site) exists and the position where the papilla does not exist (e.g., around the papilla, etc.), the state of at least one of the fundus layers and boundaries (first site) is different. Generally, a plurality of layers and boundaries normally exist around the papilla, but specific layers and boundaries are missing at the position of the papilla.

[0012] Here, assume a case where a fundus image in which both the first site and the second site are shown is input into a mathematical model for identifying the first site. In this case, at the position where the first site exists, it is natural that the first site is easily and accurately identified, so the divergence degree tends to be small. On the other hand, if the first site is missing at the position where the second site exists, the divergence degree tends to be large. This tendency is likely to appear regardless of the presence or absence of eye diseases.

[0013] Based on the above findings, when the control unit of the fundus image processing apparatus of the present disclosure inputs a fundus image into a mathematical model for identifying a first site, it identifies a second site based on the divergence of the probability distribution. As a result, regardless of the presence or absence of eye diseases, etc., the identification accuracy of the second site is improved.

[0014] The divergence will be further explained. When the first site is identified with high accuracy by the mathematical model, the obtained probability distribution tends to be skewed. On the other hand, when the identification accuracy of the first site by the mathematical model is low, the obtained probability distribution is less likely to be skewed. Therefore, the divergence between the probability distribution when the first site is accurately identified and the actually obtained probability distribution changes according to the state of the first site. Thus, according to the fundus image processing apparatus of the present disclosure, when the state of the first site changes between the position where the second site exists and the position where the second site does not exist, by using the divergence, the second site can be identified with high accuracy regardless of the presence or absence of diseases, etc.

[0015] Note that the divergence may be output by the mathematical model. Also, the control unit may calculate the divergence based on the probability distribution output by the mathematical model.

[0016] The divergence can also be expressed as the uncertainty of the identification of the first site executed by the mathematical model for the fundus image. Also, even when the reciprocal of the certainty (confidence) of the identification by the mathematical model, etc. is used as the divergence, the same result can be obtained.

[0017] The divergence may include the entropy (average amount of information) of the obtained probability distribution. Entropy represents the degree of uncertainty, randomness, and disorder. In the present disclosure, the entropy of the probability distribution output when the first site is accurately identified is 0. Also, the entropy increases as the identification of the first site becomes more difficult.

[0018] However, values other than entropy may be adopted as the degree of deviation. For example, at least any one of the standard deviation, coefficient of variation, variance, etc. indicating the degree of dispersion of the obtained probability distribution may be used as the degree of deviation. KL divergence or the like, which is a measure for depicting the difference between probability distributions, may be used as the degree of deviation. Also, the maximum value of the obtained probability distribution may be used as the degree of deviation.

[0019] In the step of obtaining the degree of deviation, the degree of deviation may be obtained by using at least any one of a plurality of layers and boundaries in the fundus shown in the fundus image as the first part. That is, the first part may be at least any one of a plurality of layers and layer boundaries in the fundus (hereinafter, may also be referred to as "layer·boundary"). As described above, the state of at least any one of the fundus layers and boundaries (the first part) may be different between the position where the second part exists and the position where the second part does not exist. Therefore, by obtaining the degree of deviation with the layer·boundary as the first part, it becomes easier to appropriately identify the second part based on the degree of deviation.

[0020] However, it is also possible to use a part other than the layer·boundary in the fundus as the first part. For example, there may be a case where the state of the fundus blood vessels is different between the position where the second part exists and the position where the second part does not exist. In this case, the degree of deviation may be obtained with the fundus blood vessels as the first part.

[0021] When the first part is the layer·boundary, the control unit may identify the papilla (optic nerve papilla) in the fundus as the second part based on the degree of deviation in the part identification step. As described above, the state of at least any one of the fundus layers and boundaries is different between the position where the papilla exists and the position where the papilla does not exist. Therefore, by using the layer·boundary as the first part and the papilla as the second part, the papilla can be appropriately detected based on the degree of deviation.

[0022] When the layer / boundary is the first part and the papilla is the second part, in the deviation degree acquisition step, among the plurality of layers and boundaries of the fundus oculi shown in the fundus image, at least any one of the layer and boundary at a position deeper than the nerve fiber layer (NFL) may be used as the first part, and the deviation degree may be acquired. At the position where the papilla exists, while the NFL exists, the layer and boundary at a position deeper than the NFL are missing. That is, at the position where the papilla exists, the deviation degree regarding the identification of the layer and boundary at a position deeper than the NFL becomes larger than at the position where the papilla does not exist. Therefore, by using at least any one of the layer and boundary at a position deeper than the NFL as the first part, the identification accuracy of the papilla is further improved.

[0023] Also, in the deviation degree acquisition step, the NFL and at least any one of the layer and boundary at a position deeper than the NFL may be used together as the first part, and the deviation degree may be acquired. In the part detection step, a part where the deviation degree regarding the identification of the layer / boundary at a position deeper than the NFL is larger than the first threshold value and the deviation degree regarding the identification of the NFL is smaller than the second threshold value may be detected as the papilla. In this case, the position where a plurality of layers / boundaries including the NFL are missing due to the influence of a disease or the like and the position where the papilla exists are appropriately distinguished. Therefore, the identification accuracy of the papilla is further improved.

[0024] In the fundus image acquisition step, the control unit may acquire a three-dimensional tomographic image of the fundus as a fundus image. The control unit may further execute a reference position setting step, a radial pattern setting step, an image extraction step, and a papilla end detection step. In the reference position setting step, the control unit sets a reference position within the region of the papilla identified in the site identification step among the two-dimensional measurement regions where the three-dimensional tomographic image was taken. In the radial pattern setting step, the control unit sets a radial pattern, which is a line pattern that radiates radially around the reference position, for the two-dimensional measurement region. In the image extraction step, the control unit extracts a two-dimensional tomographic image (a two-dimensional tomographic image that intersects each of the plurality of lines of the radial pattern) at each of the plurality of lines of the set radial pattern from the three-dimensional tomographic image. In the papilla end detection step, the control unit detects the position of the end of the papilla shown in the three-dimensional tomographic image based on the plurality of extracted two-dimensional tomographic images.

[0025] In the reference position setting step, when the reference position is correctly set within the region of the papilla, the papilla will necessarily be included in all of the plurality of two-dimensional tomographic images extracted according to the radial pattern in the image extraction step. Therefore, by detecting the position of the end of the papilla based on the plurality of extracted two-dimensional tomographic images, the possibility of erroneously detecting the end of the papilla from a two-dimensional tomographic image in which the papilla is not shown is reduced. Also, compared to the case of processing all of the plurality of two-dimensional tomographic images that make up the three-dimensional tomographic image, an excessive increase in the amount of image processing is suppressed. Thus, using the identification result of the site of the papilla executed based on the degree of deviation, the end of the papilla is also detected with high accuracy.

[0026] When the first site is a layer / boundary, the control unit may identify the fovea in the fundus as a second site based on the degree of deviation in the site identification step. At least one of the states of the layers and boundaries of the fundus differs between the position where the fovea exists and the position where the fovea does not exist. Therefore, by setting the layer / boundary as the first site and the fovea as the second site, the fovea is appropriately detected based on the degree of deviation.

[0027] When the layer / boundary is the first part and the fovea is the second part, in the deviation degree acquisition step, among the plurality of layers and boundaries of the fundus shown in the fundus image, at least one of the layer and boundary on the surface side of the retina rather than the retinal pigment epithelium (RPE) may be used as the first part, and the deviation degree may be acquired. At the position where the fovea exists, while the RPE and Bruch's membrane exist, the layer and boundary on the surface side of the retina rather than the RPE are missing. That is, at the position where the fovea exists, the deviation degree regarding the identification of the layer / boundary on the surface side rather than the RPE is larger than that at the position where the fovea does not exist. Therefore, by using at least one of the layer and boundary on the surface side of the retina rather than the RPE as the first part, the identification accuracy of the fovea is further improved.

[0028] Also, in the deviation degree acquisition step, at least one of the RPE and Bruch's membrane (hereinafter simply referred to as "RPE / Bruch's membrane") and at least one of the layer and boundary on the surface side rather than the RPE may be used as the first part together, and the deviation degree may be acquired. In the part detection step, a part where the deviation degree regarding the identification of the layer / boundary on the surface side rather than the RPE is larger than the first threshold value and the deviation degree regarding the identification of the RPE / Bruch's membrane is smaller than the second threshold value may be detected as the fovea. In this case, the position where a plurality of layers / boundaries including the RPE / Bruch's membrane are missing due to the influence of a disease or the like and the position where the fovea exists are appropriately distinguished. Therefore, the identification accuracy of the fovea is further improved.

[0029] Note that the second part to be identified based on the degree of deviation is not limited to the papilla and the fovea centralis. The second part may be a part other than the papilla and the fovea centralis in the fundus oculi (for example, the macula or fundus blood vessels, etc.). For example, at the position where the fundus blood vessels (the second part) exist, since the measurement light is blocked by the fundus blood vessels, the imaging state of the layer / boundary (the first part) at a position deeper than the fundus blood vessels deteriorates. Therefore, at the position where the fundus blood vessels exist, the degree of deviation regarding the identification of the layer / boundary at a position deeper than the fundus blood vessels is larger than that at the position where the fundus blood vessels do not exist. Thus, the control unit may identify, as the site where the fundus blood vessels exist, a site where the degree of deviation regarding the identification of at least any one of the layer / boundary at a position deeper than the fundus blood vessels is larger than the threshold value. Further, the fundus image processing device may identify the site of the disease existing in the fundus oculi as the second part.

[0030] In the step of obtaining the degree of deviation, the control unit may obtain the two-dimensional distribution of the degree of deviation when looking at the fundus oculi from the front (that is, when looking at the fundus oculi along the optical axis of the imaging light of the fundus image) by inputting the three-dimensional tomographic image of the fundus oculi into the mathematical model. In the step of identifying the part, the position of the second part when looking at the fundus oculi from the front may be identified based on the two-dimensional distribution of the degree of deviation. In this case, compared with the case of identifying the two-dimensional position of the second part from the two-dimensional fundus image, the second part is identified based on more data. Therefore, the identification accuracy of the second part is further improved.

[0031] Note that a specific method for obtaining the two-dimensional distribution of the degree of deviation from the three-dimensional tomographic image can also be appropriately selected. For example, the control unit may input each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image into the mathematical model, and arrange the degrees of deviation obtained for each two-dimensional tomographic image in two dimensions to obtain the two-dimensional distribution of the degree of deviation. Further, the control unit may obtain the two-dimensional distribution of the degree of deviation by inputting the whole three-dimensional tomographic image into the mathematical model at once. Note that the tomographic images (three-dimensional tomographic images and two-dimensional tomographic images) may be taken by various devices such as an OCT device or a shine-proof camera, for example.

[0032] However, the control unit may identify the second part in the fundus by inputting the two-dimensional fundus image into a mathematical model. For example, the control unit may input the two-dimensional frontal image when the fundus is viewed from the front into a mathematical model for identifying the fundus blood vessels as the first part. The control unit may detect the second part (such as the optic disc, etc.) based on the obtained two-dimensional distribution of the deviation degree. The two-dimensional frontal image may be an image captured by a fundus camera, an image captured by a scanning laser ophthalmoscope (SLO), etc. The two-dimensional frontal image may also be an Enface image generated based on the data of the three-dimensional tomographic image captured by an OCT device. Further, the two-dimensional frontal image may be an image (so-called "motion contrast image") created from motion contrast data obtained by processing a plurality of OCT data acquired from the same position at different times.

[0033] The control unit may further execute a frontal image acquisition step and an auxiliary identification result acquisition step. In the frontal image acquisition step, the control unit acquires a two-dimensional frontal image when the fundus where the three-dimensional tomographic image is captured is viewed from the front. In the auxiliary identification result acquisition step, the control unit acquires an auxiliary identification result which is the identification result of the second part executed based on the two-dimensional frontal image. The second part may be identified based on the deviation degree and the auxiliary identification result. In this case, in addition to the deviation degree obtained from the three-dimensional tomographic image, the auxiliary identification result based on the two-dimensional frontal image is also considered, so that the second part can be more appropriately identified.

[0034] A specific method for obtaining the auxiliary identification result can be appropriately selected. For example, the auxiliary identification result may be the result of identifying the second part by performing image processing on the two-dimensional frontal image. In this case, the image processing may be executed by the control unit of the fundus image processing device or by other devices. Further, the control unit may obtain the auxiliary identification result by inputting the two-dimensional frontal image acquired in the frontal image acquisition step into a mathematical model that outputs the identification result of the second part in the two-dimensional frontal image.

[0035] A specific method for identifying the second part based on the auxiliary identification result and the degree of deviation can also be appropriately selected. For example, the control unit may extract a part of the three-dimensional tomographic image obtained in the image acquisition step, where the second part is likely to be included, based on the auxiliary identification result. The control unit may input the extracted three-dimensional tomographic image into a mathematical model to obtain the degree of deviation, and identify the second part based on the obtained degree of deviation. In this case, since the processing amount by the mathematical model is reduced, the second part can be identified more efficiently. Also, the control unit may identify the second part by adding the identification result based on the degree of deviation and the auxiliary identification result after performing arbitrary weighting. Further, the control unit may notify the user of a warning or an error, etc., when the difference between the identification result based on the degree of deviation and the auxiliary identification result does not satisfy the condition.

[0036] The mathematical model may output the distribution of scores indicating the possibility of being the second part, together with the identification result of the first part of the fundus shown in the fundus image. In the part identification step, the second part may be identified based on the degree of deviation and the distribution of scores. In this case, the second part is identified based on the distribution of scores of the second part and the degree of deviation that is less affected by the presence or absence of eye diseases, etc. Therefore, the identification accuracy of the second part is further improved.

[0037] Note that a specific method for identifying the second part based on both the degree of deviation and the distribution of scores can also be appropriately selected. For example, the control unit may identify the second part by adding the identification result based on the degree of deviation and the identification result based on the distribution of scores. In this case, the control unit may add each identification result after performing arbitrary weighting. However, it is also possible for the control unit to identify the second part without using the distribution of scores of the second part.

[0038] (Second aspect)

[0039] The control unit of the fundus image processing apparatus exemplified in the present disclosure executes an image acquisition step, a papilla center setting step, a radial pattern setting step, an image extraction step, and a papilla end detection step. In the image acquisition step, the control unit acquires a three-dimensional tomographic image of the fundus of the eye to be examined, which is taken by irradiating a measurement light onto a two-dimensional measurement region that spreads in a direction intersecting the optical axis of the OCT measurement light. In the reference position setting step, the control unit sets a reference position within the region of the papilla in the two-dimensional measurement region where the three-dimensional tomographic image is taken. In the radial pattern setting step, the control unit sets a radial pattern, which is a line pattern that spreads radially around the reference position, for the two-dimensional measurement region. In the image extraction step, the control unit extracts two-dimensional tomographic images (two-dimensional tomographic images intersecting each of the plurality of lines of the radial pattern) in each of the plurality of lines of the set radial pattern from the three-dimensional tomographic image. In the papilla end detection step, the control unit detects the position of the end of the papilla shown in the three-dimensional tomographic image based on the plurality of extracted two-dimensional tomographic images.

[0040] In the reference position setting step, when the reference position is correctly set within the region of the papilla, the papilla will necessarily be included in all of the plurality of two-dimensional tomographic images extracted according to the radial pattern in the image extraction step. Therefore, by detecting the position of the end of the papilla based on the plurality of extracted two-dimensional tomographic images, the possibility of erroneously detecting the end of the papilla from a two-dimensional tomographic image in which the papilla does not appear is reduced. Also, compared with the case of processing all of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image, an excessive increase in the amount of image processing is suppressed. Thus, the end of the papilla is appropriately detected with high accuracy.

[0041] When the tomographic image captured by the OCT device is used for diagnosis, it is desirable that various information such as not only information about the papilla but also the retinal thickness and the like can be obtained based on the tomographic image. Here, it is also conceivable to set the center of the radial pattern within the region of the papilla and then actually scan the measurement light along the radial pattern onto the fundus to capture a plurality of two-dimensional tomographic images. Even in this case, it seems that the position of the end of the papilla can be detected from the plurality of captured two-dimensional tomographic images. However, it is difficult to obtain various information such as the retinal thickness from the plurality of two-dimensional tomographic images captured according to the radial pattern. On the other hand, in addition to the papilla end detection step, the control unit of the fundus image processing device of the present disclosure may perform an analysis step of the fundus (for example, analysis of the thickness of a specific layer of the retina, etc.) on the three-dimensional tomographic image acquired in the image acquisition step. That is, according to the fundus image processing device of the present disclosure, by using the three-dimensional tomographic image, not only can the position of the end of the papilla be detected with high accuracy, but also the analysis result of the fundus can be obtained.

[0042] Note that the details of the "end of the papilla" to be detected can be appropriately selected. For example, at least any one of the Bruch's Membrane Opening (BMO), the edge of the optic nerve papilla, and the peripapillary atrophy (PPA) may be detected as the end of the papilla.

[0043] Note that various devices can function as the fundus image processing device. For example, the OCT device itself may function as the fundus image processing device in the present disclosure. Also, a device (for example, a personal computer, etc.) capable of exchanging data with the OCT device may function as the fundus image processing device. The control units of a plurality of devices may cooperate to perform the processing.

[0044] The OCT device may be provided with a scanning unit. The scanning unit scans the measurement light irradiated onto the tissue by the irradiation optical system in a two-dimensional direction intersecting the optical axis. The three-dimensional tomographic image may be obtained by scanning the spot of the measurement light in a two-dimensional direction within the measurement region by the scanning unit. In this case, the three-dimensional tomographic image can be appropriately obtained by the OCT device.

[0045] However, it is also possible to change the configuration of the OCT device. For example, the irradiation optical system of the OCT device may simultaneously irradiate the measurement light onto a two-dimensional region on the tissue of the subject. In this case, the light receiving element may be a two-dimensional light receiving element that detects the interference signal in the two-dimensional region on the tissue. That is, the OCT device may acquire OCT data according to the principle of so-called full-field OCT (FF-OCT). Further, the OCT device may simultaneously irradiate the measurement light onto an irradiation line extending in a one-dimensional direction in the tissue and scan the measurement light in a direction intersecting the irradiation line. In this case, the light receiving element may be a one-dimensional light receiving element (for example, a line sensor) or a two-dimensional light receiving element. That is, the OCT device may acquire a tomographic image according to the principle of so-called line-field OCT (LF-OCT).

[0046] The control unit may further execute an alignment step of aligning the image in the direction along the optical axis of the OCT measurement light with respect to the three-dimensional tomographic image or the two-dimensional tomographic image extracted in the extraction step. The control unit may detect the position of the end of the papilla based on the two-dimensional tomographic image in a state where the image alignment is executed. In this case, by executing the image alignment, the misalignment of the end of the annular papilla in the direction along the optical axis of the OCT measurement light (the depth direction of the tissue) is reduced. Therefore, the end of the papilla can be detected with higher accuracy.

[0047] The control unit may further execute a papilla position detection step of automatically detecting the position of the papilla in a two-dimensional region intersecting the optical axis of the OCT measurement light based on the fundus image. The control unit may set a reference position based on the automatically detected position of the papilla. In this case, even if the accuracy of the automatic detection of the position of the papilla is low, if the detected position is within the actual papilla region, the end of the papilla will be appropriately detected in the subsequent papilla end detection step. Therefore, the detection process is performed more smoothly.

[0048] In addition, in the papilla position detection step, the central position of the papilla may be detected. In this case, compared with the case where a position other than the center of the papilla is detected and set as the reference position, the possibility that the reference position falls within the papilla region is even higher.

[0049] A specific method for automatically detecting the position of the papilla based on the fundus image can be appropriately selected. As an example, at the position where the papilla exists, the NFL exists, while the layers and boundaries at positions deeper than the NFL are missing. Therefore, when at least one of the fundus layers and boundaries (hereinafter simply referred to as "layer·boundary") shown in the three-dimensional tomographic image is detected by a mathematical model trained by a machine learning algorithm, the uncertainty of the detection of the layer·boundary at a position deeper than the NFL becomes higher at the position of the papilla. Therefore, the control unit may automatically detect the position (central position) of the papilla based on the uncertainty when detecting the layer·boundary at a position deeper than the NFL by the mathematical model. For example, the control unit may detect a region where the uncertainty is equal to or greater than the threshold value as the papilla region, and detect the center (e.g., the centroid, etc.) of the detected region as the central position of the papilla.

[0050] In addition, the control unit can also automatically detect the position of the papilla based on the two-dimensional front image when the three-dimensional tomographic image is viewed from the front (the direction along the optical axis of the OCT measurement light). For example, the control unit may detect the area of the papilla by performing known image processing on the two-dimensional front image, and detect the center of the detected area as the center position of the papilla. Further, the control unit may automatically detect the position (center position) of the papilla by inputting the two-dimensional front image into a mathematical model that detects and outputs the position of the papilla shown in the two-dimensional front image. Note that the two-dimensional front image may be a front image (so-called "Enface image", etc.) generated based on the three-dimensional tomographic image acquired in the image acquisition step. Further, the two-dimensional front image may be an image captured by a principle different from the imaging principle of the three-dimensional tomographic image (for example, a fundus camera image or an SLO image, etc.).

[0051] However, it is also possible to change the method for setting the reference position. For example, the control unit may set the reference position at a position specified by the user in the two-dimensional measurement area. That is, the user may set the reference position by himself / herself. In this case, based on the user's experience, etc., when the reference position is set within the area of the papilla, the end of the papilla can be appropriately detected in the subsequent papilla end detection step. Note that the control unit may set the reference position at a position specified by the user, for example, when the automatic detection of the position of the papilla described above fails. Further, the user may be allowed to set the reference position without performing the automatic detection of the position of the papilla. In addition, for example, when the position of the papilla detected in the past is stored, etc., the reference position may be set at the stored position of the papilla. In this case, it is also possible to omit the process of automatically detecting the position of the papilla.

[0052] In the nipple end detection step, a mathematical model trained by a machine learning algorithm may be used. The mathematical model may be trained to output a detection result of the end of the nipple shown in the input two-dimensional tomographic image. The control unit may detect the position of the end of the nipple by inputting the plurality of two-dimensional tomographic images extracted in the image extraction step into the mathematical model and obtaining the position of the end of the nipple output by the mathematical model. In this case, the position of the end of the nipple is automatically and appropriately detected from the plurality of two-dimensional tomographic images extracted according to the radial pattern.

[0053] The position of the end of the nipple automatically detected by using the machine learning algorithm may be corrected according to an instruction by the user. For example, the control unit may cause the display device to display the two-dimensional tomographic image input to the mathematical model and the position of the end of the nipple output by the mathematical model. The control unit may correct the position of the end of the nipple according to an instruction from the user who has confirmed the displayed position of the end of the nipple. In this case, even if the accuracy of the automatic detection of the end of the nipple is low, the position is appropriately corrected by the user. Therefore, the end of the nipple is detected with higher accuracy.

[0054] However, it is also possible to change the specific method for detecting the position of the end of the nipple. For example, the control unit may receive an input of an instruction from the user while the two-dimensional tomographic image extracted in the image extraction step is being displayed on the display device. The control unit may detect the position indicated by the user as the position of the end of the nipple. As described above, the two-dimensional tomographic image appropriately extracted according to the radial pattern always includes the nipple. Therefore, the user can appropriately input (instruct) the position of the end of the nipple by looking at the displayed two-dimensional tomographic image. As a result, the end of the nipple is detected with high accuracy. Further, the control unit may automatically detect the position of the end of the nipple by performing known image processing on the plurality of two-dimensional tomographic images extracted in the image extraction step.

[0055] In the papilla end detection step, the control unit may detect the position of the annular end of the papilla by performing a smoothing process on the detection results of a plurality of positions detected based on a plurality of two-dimensional tomographic images. For example, due to the influence of the presence of fundus blood vessels or the like, the position of the end of the papilla in some two-dimensional tomographic images may be erroneously detected. In this case, among the annular ends of the papilla, the erroneously detected position will deviate from the normally detected position. On the contrary, by performing a smoothing process on the detection results of the detected plurality of positions, the influence of some erroneously detected positions is suppressed. Therefore, the position of the annular end of the papilla is detected more appropriately.

[0056] In the papilla center identification step, the control unit may further execute a step of identifying the center position of the papilla based on the position of the end of the papilla detected in the papilla end detection step. In this case, based on the position of the end of the papilla detected with high accuracy, the center position of the papilla is identified. Therefore, the center position of the papilla is identified with high accuracy.

[0057] A specific method for identifying the center position of the papilla based on the detected position of the end of the papilla can be appropriately selected. For example, the control unit may identify the centroid position of the detected annular end of the papilla as the center position of the papilla. Alternatively, the control unit may fit an ellipse to the detected end of the papilla and identify the center position of the fitted ellipse as the center position of the papilla.

[0058] The control unit may re-execute the reference position setting step, the radial pattern setting step, the image extraction step, and the papilla end detection step, using the center position of the papilla identified in the papilla center identification step as the setting position of the reference position in the reference position setting step. The closer the reference position is to the center position of the papilla, the more similar the positions of the ends of the papilla in each of the plurality of two-dimensional tomographic images extracted according to the radial pattern will be, so the detection accuracy of the annular end of the papilla will also increase. Therefore, by re-detecting the position of the end of the papilla using the center position of the papilla identified in the papilla center identification step as the reference position, the detection accuracy is further improved.

[0059] The control unit may further execute an annular extraction step and an output step. In the annular extraction step, the control unit extracts a two-dimensional tomographic image (that is, an image obtained by deforming a tomographic image that intersects a cylindrical shape in an annular line pattern two-dimensionally) in an annular line pattern centered on the center position of the papilla identified in the papilla center identification step from the three-dimensional tomographic image. In the output step, the control unit outputs information regarding the two-dimensional tomographic image extracted in the annular extraction step. In this case, based on the center position of the papilla detected with high accuracy, the state of the tissue in the vicinity of the papilla is appropriately observed.

[0060] When outputting information regarding the two-dimensional tomographic image extracted according to the annular line pattern, a specific information output method can be appropriately selected. For example, the control unit may display the extracted two-dimensional tomographic image on a display device. The control unit may also display, on the display device, a graph indicating the thickness of a specific layer (for example, the thickness of the NFL, or the thickness from the ILM to the NFL, etc.) of the retina shown in the extracted two-dimensional tomographic image. The control unit may display at least one of the two-dimensional tomographic image and the graph of the patient in comparison with the data of a normal eye without the disease.

[0061] At the position where the fundus blood vessels exist, since the end of the papilla is difficult to appear in the tomographic image, the possibility of the end of the papilla being erroneously detected is increased. The control unit may acquire information on the position of the fundus blood vessels in the measurement region where the three-dimensional tomographic image was taken. The control unit may adjust at least one of the overall angle of the radial pattern, the angle of at least any one of the lines included in the radial pattern, the length of the line, the number of lines, etc., so that the overlapping amount of the lines of the radial pattern and the fundus blood vessels is reduced as much as possible. In this case, since the influence of the fundus blood vessels is reduced, the detection accuracy of the end of the papilla is further improved.

[0062] Note that the method for acquiring the information on the position of the fundus blood vessels can be appropriately selected. For example, the control unit may detect the position of the fundus blood vessels by performing known image processing on a two-dimensional frontal image of the fundus (for example, an Enface image, an SLO image, or a fundus camera image, etc.). Further, the control unit may obtain the detection result of the fundus blood vessels output by the mathematical model by inputting a fundus image (a two-dimensional frontal image or a three-dimensional tomographic image, etc.) into the mathematical model trained by a machine learning algorithm. Further, the control unit may obtain the information on the position of the fundus blood vessels by inputting an instruction on the position of the fundus blood vessels given by a user who has checked the fundus image.

[0063] Further, the control unit may adjust at least any one of the overall angle of the radial pattern, the angle of at least any one of the lines included in the radial pattern, the length of the line, the number of lines, etc. according to an instruction input by a user who has checked the fundus image. In this case, the user can appropriately set the radial pattern so that the overlapping amount between the lines of the radial pattern and the fundus blood vessels is reduced as much as possible.

[0064] The control unit can also detect various structures of the fundus based on the detection result of the end of the papilla. For example, when detecting the BMO as the end of the papilla, the control unit may detect the position of the optic nerve papilla depression (Cup) based on the detected BMO. As an example, the control unit may set a straight line that is parallel to the reference straight line passing through the pair of detected BMOs and is spaced a predetermined distance from the reference straight line to the surface side of the retina. The control unit may detect the position where the set straight line intersects the ILM (inner limiting membrane) in the fundus image as the position of the Cup. Further, the control unit may detect the shortest distance between the detected BMO and the ILM in the fundus image as the minimum thickness of the nerve fiber layer (Minimum Rim Width).

[0065] <Embodiment> (Device Configuration) Hereinafter, one of the typical embodiments in the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 101, a fundus image processing device 1, and OCT devices (fundus image capturing devices) 10A and 10B are used. The mathematical model construction device 101 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The constructed mathematical model identifies or detects a specific part shown in the input fundus image based on the input fundus image. The fundus image processing device 1 executes various processes using the result output by the mathematical model. The OCT devices 10A and 10B function as fundus image capturing devices that capture fundus images (in this embodiment, cross-sectional images of the fundus) of the eye to be examined.

[0066] As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 101 in this embodiment. Although details will be described later, the mathematical model construction device 101 uses the data of the fundus image of the eye to be examined (hereinafter referred to as "training fundus image") acquired from the OCT device 10A and the data indicating the first part (in this embodiment, the part of the optic disc) of the eye to be examined where the training fundus image was captured to train the mathematical model. As a result, the mathematical model is constructed. However, the device that can function as the mathematical model construction device 101 is not limited to a PC. For example, the OCT device 10A may function as the mathematical model construction device 101. Further, the control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the OCT device 10A) may cooperate to construct the mathematical model.

[0067] In addition, a PC is used for the fundus image processing apparatus 1 of the present embodiment. However, the device that can function as the fundus image processing apparatus 1 is not limited to a PC. For example, an OCT apparatus 10B or a server or the like may function as the fundus image processing apparatus 1. When the OCT apparatus 10B functions as the fundus image processing apparatus 1, the OCT apparatus 10B can process the captured fundus image while capturing the fundus image. Further, a mobile terminal such as a tablet terminal or a smartphone may function as the fundus image processing apparatus 1. The control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the OCT apparatus 10B) may cooperate to perform various processes.

[0068] In addition, in the present embodiment, the case where a CPU is used as an example of a controller that performs various processes will be exemplified. However, it goes without saying that a controller other than the CPU may be used for at least a part of various devices. For example, by adopting a GPU as the controller, the processing speed may be increased.

[0069] The mathematical model construction apparatus 101 will be described. The mathematical model construction apparatus 101 is arranged, for example, in the fundus image processing apparatus 1 or a manufacturer or the like that provides a fundus image processing program to a user. The mathematical model construction apparatus 101 includes a control unit 102 that performs various control processes and a communication I / F 105. The control unit 102 includes a CPU 103 that is a controller in charge of control and a storage device 104 that can store programs, data, and the like. The storage device 104 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 7) described later. The communication I / F 105 connects the mathematical model construction apparatus 101 to other devices (for example, the OCT apparatus 10A and the fundus image processing apparatus 1, etc.).

[0070] The mathematical model construction device 101 is connected to the operation unit 107 and the display device 108. The operation unit 107 is operated by the user to input various instructions to the mathematical model construction device 101. At least one of, for example, a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 107. Note that, together with the operation unit 107 or instead of the operation unit 107, a microphone or the like for inputting various instructions may be used. The display device 108 displays various images. Various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used for the display device 108. Note that the "image" in the present disclosure includes both still images and moving images.

[0071] The mathematical model construction device 101 can acquire data of a fundus image (hereinafter, may be simply referred to as "fundus image") from the OCT device 10A. The mathematical model construction device 101 may acquire data of the fundus image from the OCT device 10A by, for example, at least one of wired communication, wireless communication, a removable storage medium (for example, a USB memory), etc.

[0072] The fundus image processing device 1 will be described. The fundus image processing device 1 is arranged, for example, in a facility (for example, a hospital or a health examination facility, etc.) that performs diagnosis or examination of a subject. The fundus image processing device 1 includes a control unit 2 that performs various control processes and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller for controlling and a storage device 4 that can store programs, data, etc. The storage device 4 stores a fundus image processing program for executing the fundus image processing (see FIG. 8) and the part identification processing (see FIG. 15) described later. The fundus image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 101. The communication I / F 5 connects the fundus image processing device 1 to other devices (for example, the OCT device 10B and the mathematical model construction device 101, etc.).

[0073] The fundus image processing apparatus 1 is connected to the operation unit 7 and the display device 8. Similar to the operation unit 107 and the display device 108 described above, various devices can be used for the operation unit 7 and the display device 8.

[0074] The fundus image processing apparatus 1 can acquire a fundus image (in this embodiment, a three-dimensional tomographic image of the fundus) from the OCT apparatus 10B. The fundus image processing apparatus 1 may acquire the fundus image from the OCT apparatus 10B by at least any one of, for example, wired communication, wireless communication, a removable storage medium (e.g., a USB memory), etc. Further, the fundus image processing apparatus 1 may acquire a program or the like for realizing the mathematical model constructed by the mathematical model construction apparatus 101 via communication or the like.

[0075] The OCT apparatus 10 (10A, 10B) will be described. As an example, in this embodiment, the case where the OCT apparatus 10A that provides a fundus image to the mathematical model construction apparatus 101 and the OCT apparatus 10B that provides a fundus image to the fundus image processing apparatus 1 are used will be described. However, the number of OCT apparatuses used is not limited to two. For example, the mathematical model construction apparatus 101 and the fundus image processing apparatus 1 may acquire fundus images from a plurality of OCT apparatuses. Further, the mathematical model construction apparatus 101 and the fundus image processing apparatus 1 may acquire fundus images from one common OCT apparatus.

[0076] As shown in FIG. 2, the OCT apparatus 10 includes an OCT unit and a control unit 30. The OCT unit includes an OCT light source 11, a coupler (optical splitter) 12, a measurement optical system 13, a reference optical system 20, a light receiving element 22, and a front observation optical system 23.

[0077] The OCT light source 11 emits light (OCT light) for acquiring OCT data. The coupler 12 splits the OCT light emitted from the OCT light source 11 into measurement light and reference light. Further, the coupler 12 in the present embodiment combines and interferes the measurement light reflected by the subject (the fundus of the eye E in the present embodiment) and the reference light generated by the reference optical system 20. That is, the coupler 12 in the present embodiment serves as a branching optical element that branches the OCT light into measurement light and reference light, and a multiplexing optical element that multiplexes the reflected light of the measurement light and the reference light.

[0078] The measurement optical system 13 guides the measurement light split by the coupler 12 to the subject and returns the measurement light reflected by the subject to the coupler 12. The measurement optical system 13 includes a scanning unit 14, an irradiation optical system 16, and a focus adjustment unit 17. The scanning unit 14 can scan (deflect) the measurement light in a two-dimensional direction intersecting the optical axis of the measurement light by being driven by the drive unit 15. The irradiation optical system 16 is provided on the downstream side of the optical path (that is, the subject side) with respect to the scanning unit 14 and irradiates the tissue of the subject with the measurement light. The focus adjustment unit 17 adjusts the focus of the measurement light by moving an optical member (for example, a lens) included in the irradiation optical system 16 in the direction along the optical axis of the measurement light.

[0079] The reference optical system 20 generates reference light and returns it to the coupler 12. The reference optical system 20 in the present embodiment generates reference light by reflecting the reference light split by the coupler 12 by a reflection optical system (for example, a reference mirror). However, the configuration of the reference optical system 20 can also be changed. For example, the reference optical system 20 may transmit the light incident from the coupler 12 without reflecting it and return it to the coupler 12. The reference optical system 20 includes an optical path length difference adjustment unit 21 that changes the optical path length difference between the measurement light and the reference light. In the present embodiment, the optical path length difference is changed by moving the reference mirror in the optical axis direction. Note that the configuration for changing the optical path length difference may be provided in the optical path of the measurement optical system 13.

[0080] The light-receiving element 22 detects an interference signal by receiving the interference light of the measurement light and the reference light generated by the coupler 12. In the present embodiment, the principle of Fourier domain OCT is adopted. In Fourier domain OCT, the spectral intensity of the interference light (spectral interference signal) is detected by the light-receiving element 22, and a complex OCT signal is obtained by performing a Fourier transform on the spectral intensity data. As an example of Fourier domain OCT, Spectral-domain-OCT (SD-OCT), Swept-source-OCT (SS-OCT), etc. can be adopted. Also, for example, Time-domain-OCT (TD-OCT), etc. can also be adopted.

[0081] In the present embodiment, the spot of the measurement light is scanned within a two-dimensional measurement region by the scanning unit 14, thereby obtaining three-dimensional OCT data (three-dimensional tomographic image). However, it is also possible to change the principle of obtaining the three-dimensional OCT data. For example, the three-dimensional OCT data may be obtained according to the principle of line-field OCT (hereinafter referred to as "LF-OCT"). In LF-OCT, the measurement light is simultaneously irradiated onto an irradiation line extending in a one-dimensional direction in the tissue, and the interference light of the reflected light of the measurement light and the reference light is received by a one-dimensional light-receiving element (for example, a line sensor) or a two-dimensional light-receiving element. In the two-dimensional measurement region, the measurement light is scanned in a direction intersecting the irradiation line, thereby obtaining three-dimensional OCT data. Also, the three-dimensional OCT data may be obtained according to the principle of full-field OCT (hereinafter referred to as "FF-OCT"). In FF-OCT, the measurement light is irradiated onto a two-dimensional measurement region on the tissue, and the interference light of the reflected light of the measurement light and the reference light is received by a two-dimensional light-receiving element. In this case, the OCT apparatus 10 may not include the scanning unit 14.

[0082] The front observation optical system 23 is provided for capturing in real time a two-dimensional front image of the tissue of the subject (the fundus of the eye E in this embodiment). The two-dimensional front image in this embodiment is a two-dimensional image when the tissue is viewed from the direction (front direction) along the optical axis of the measurement light of OCT. In this embodiment, a scanning laser ophthalmoscope (SLO) is employed as the front observation optical system 23. However, as the configuration of the front observation optical system 23, a configuration other than the SLO (for example, an infrared camera that irradiates infrared light uniformly over a two-dimensional imaging range to capture a front image) may be employed.

[0083] The control unit 30 is in charge of various controls of the OCT apparatus 10. The control unit 30 includes a CPU 31, a RAM 32, a ROM 33, and a non-volatile memory (NVM) 34. The CPU 31 is a controller that performs various controls. The RAM 32 temporarily stores various information. The ROM 33 stores programs executed by the CPU 31 and various initial values, etc. The NVM 34 is a non-transitory storage medium that can retain the stored content even when the power supply is cut off. An operation unit 37 and a display device 38 are connected to the control unit 30. For the operation unit 37 and the display device 38, various devices can be used in the same way as the operation unit 107 and the display device 108 described above.

[0084] The method for capturing a fundus image in this embodiment will be described. As shown in FIG. 3, the OCT apparatus 10 in this embodiment sets a plurality of linear scanning lines (scan lines) 41 for scanning spots at equal intervals within a two-dimensional measurement region 40 that spreads in a direction intersecting the optical axis of the OCT measurement light. The OCT apparatus 10 can capture a two-dimensional tomographic image 42 (see FIG. 4) of a cross section intersecting each scanning line 41 by scanning the spot of the measurement light on each scanning line 41. The two-dimensional tomographic image 42 may be an addition-average image generated by performing addition-average processing on a plurality of two-dimensional tomographic images of the same site. Further, the OCT apparatus 10 can acquire (capture) a three-dimensional tomographic image 43 (see FIG. 5) by arranging the plurality of two-dimensional tomographic images 42 captured for the plurality of scanning lines 41 in a direction intersecting perpendicularly to the image region.

[0085] In addition, the OCT apparatus 10 can also acquire (generate) an Enface image 45, which is a two-dimensional front image when the tissue is viewed from the direction (front direction) along the optical axis of the measurement light, based on the captured three-dimensional tomographic image 43. When acquiring the Enface image 45 in real time, it is also possible to omit the front observation optical system 23. The data of the Enface image 45 may be, for example, integrated image data in which luminance values are integrated in the depth direction (Z direction) at each position in the XY direction, integrated values of spectral data at each position in the XY direction, luminance data at each position in the XY direction in a certain depth direction, luminance data at each position in the XY direction in any layer (for example, the retinal surface layer) of the retina, and the like. Further, the Enface image 45 may be obtained from a motion contrast image (for example, an OCT angiography image) obtained by acquiring a plurality of OCT signals at different times from the same position of the tissue of the patient's eye.

[0086] Returning to the description of FIG. 1, the OCT apparatus 10A connected to the mathematical model construction apparatus 101 can at least capture a two-dimensional tomographic image 42 (see FIG. 4) of the fundus of the eye to be examined. In addition, the OCT apparatus 10B connected to the fundus image processing apparatus 1 can capture a three-dimensional tomographic image 43 (see FIG. 5) of the fundus of the eye to be examined in addition to the two-dimensional tomographic image 42 described above.

[0087] (Structure of Fundus Layers and Boundaries) Referring to FIG. 6, the structure of the layers in the fundus of the eye to be examined and the boundaries between the layers adjacent to each other will be described. FIG. 6 schematically shows the structure of the layers and boundaries in the fundus. The upper side of FIG. 6 is the surface side of the retina of the fundus. That is, the depth of the layers and boundaries increases as going downward in FIG. 6. In addition, in FIG. 6, parentheses are attached to the names of the boundaries between adjacent layers.

[0088] The layers of the fundus will be described. In the fundus, from the surface side (the upper side in Fig. 6) in order, there are ILM (internal limiting membrane), NFL (nerve fiber layer), GCL (ganglion cell layer), IPL (inner plexiform layer), INL (inner nuclear layer), OPL (outer plexiform layer), ONL (outer nuclear layer), ELM (external limiting membrane), IS / OS (junction between photoreceptor inner and outer segment), RPE (retinal pigment epithelium), BM (Bruch's membrane), and Choroid.

[0089] Also, as boundaries that are likely to appear in tomographic images, for example, there are NFL / GCL (the boundary between NFL and GCL), IPL / INL (the boundary between IPL and INL), OPL / ONL (the boundary between OPL and ONL), RPE / BM (the boundary between RPE and BM), BM / Choroil (the boundary between BM and Choroid), etc.

[0090] (Mathematical model construction process) With reference to Fig. 7, the mathematical model construction process executed by the mathematical model construction device 101 will be described. The mathematical model construction process is executed by the CPU 103 according to the mathematical model construction program stored in the storage device 104.

[0091] Hereinafter, as an example, a case will be illustrated in which a mathematical model is constructed to output the identification result of at least any one of a plurality of layers and boundaries (a specific layer and boundary which is the first part in this embodiment) shown in a fundus image by analyzing the input two-dimensional tomographic image. However, in the mathematical model construction process, it is also possible to construct a mathematical model that outputs a result different from the identification result of the layer and boundary. For example, a mathematical model (details will be described later) that outputs the detection result of the end of the papilla shown in the input two-dimensional tomographic image is also constructed by the mathematical model construction process.

[0092] In addition, the mathematical model exemplified in this embodiment is trained to output, together with the identification result of the first part (a specific layer and boundary in this embodiment) shown in the input fundus image, the distribution of scores indicating the possibility that each part (each A-scan image) in the two-dimensional tomographic image is the second part (the papilla in this embodiment).

[0093] In the mathematical model construction process, the mathematical model is constructed by training the mathematical model with a training dataset. The training dataset includes input-side data (training data for input) and output-side data (training data for output).

[0094] As shown in FIG. 7, the CPU 103 acquires, as training data for input, the data of the fundus image (a two-dimensional tomographic image in this embodiment) taken by the OCT apparatus 10A (S1). Next, the CPU 103 acquires, as training data for output, the data indicating the first part of the subject eye in which the fundus image acquired in S1 was taken (S2). The training data for output in this embodiment includes the data of the label indicating the position of the specific layer and boundary shown in the fundus image. The label data may be generated, for example, by an operator operating the operation unit 107 while viewing the layer and boundary in the fundus image. In this embodiment, in order to cause the mathematical model to output a score indicating the possibility of being the second part (the papilla in this embodiment), the label data indicating the second part in the fundus image is also included in the training data for output.

[0095] Next, the CPU 103 executes training of a mathematical model using a training dataset by a machine learning algorithm (S3). As the machine learning algorithm, for example, neural networks, random forests, boosting, support vector machines (SVM), etc. are generally known.

[0096] A neural network is a method that mimics the behavior of a biological neural cell network. Examples of neural networks include feedforward (forward propagation type) neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0097] Random forest is a method of learning based on randomly sampled training data to generate a number of decision trees. When using a random forest, follow the branches of a plurality of decision trees that have been learned in advance as discriminators and take the average (or majority vote) of the results obtained from each decision tree.

[0098] Boosting is a technique for generating a strong discriminator by combining a plurality of weak discriminators. A strong discriminator is constructed by sequentially learning simple and weak discriminators.

[0099] SVM is a technique for constructing a two-class pattern discriminator using linear input elements. SVM learns the parameters of the linear input elements based on a criterion (hyperplane separation theorem) of, for example, obtaining a maximum margin hyperplane with the maximum distance from each data point from the training data.

[0100] A mathematical model refers to, for example, a data structure for predicting the relationship between input data (in this embodiment, data of a two-dimensional tomographic image similar to the training data for input) and output data (in this embodiment, data of the identification result of the first part). The mathematical model is constructed by being trained using a training data set. As described above, the training data set is a set of training data for input and training data for output. For example, through training, the correlation data (e.g., weights) between each input and output is updated.

[0101] In this embodiment, a multi-layer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data of the analysis result to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used.

[0102] Note that other machine learning algorithms may be used. For example, a generative adversarial network (GAN) that utilizes two competing neural networks may be adopted as the machine learning algorithm.

[0103] Until the construction of the mathematical model is completed (S4: NO), the processes of S1 to S3 are repeated. When the construction of the mathematical model is completed (S4: YES), the mathematical model construction process ends. In this embodiment, the program and data for realizing the constructed mathematical model are incorporated into the fundus image processing apparatus 1.

[0104] (Fundus Image Processing) With reference to FIGS. 8 to 17, the fundus image processing executed by the fundus image processing apparatus 1 will be described. In the fundus image processing of the present embodiment, a plurality of two-dimensional tomographic images are extracted from the three-dimensional tomographic image according to a radial pattern, and the end portion of the optic disc is detected based on the plurality of extracted two-dimensional tomographic images. As an example, in the present embodiment, the case of detecting the position of the Bruch's membrane opening (BMO) as the position of the end portion of the optic disc is illustrated. Further, the fundus image processing of the present embodiment also includes a site discrimination process (see S3 in FIG. 8 and FIG. 15). In the site discrimination process, a second site (the optic disc in the present embodiment) different from the first site is discriminated based on the degree of divergence of the probability distribution when the mathematical model discriminates the first site (a specific layer / boundary in the present embodiment). The CPU 3 of the fundus image processing apparatus 1 executes the fundus image processing shown in FIG. 8 and the site discrimination process shown in FIG. 15 according to the fundus image processing program stored in the storage device 4.

[0105] As shown in FIG. 8, the CPU 3 acquires a three-dimensional tomographic image of the fundus of the eye to be examined (S1). As described above, the three-dimensional tomographic image 43 (see FIG. 5) is taken by irradiating the OCT measurement light on the two-dimensional measurement region 40 (see FIG. 3). The three-dimensional tomographic image 43 of the present embodiment is configured by arranging a plurality of two-dimensional tomographic images 42 (see FIG. 4).

[0106] The CPU 3 performs image alignment in the direction along the optical axis of the OCT (the Z direction in the present embodiment) on the three-dimensional tomographic image acquired in S1 (S2). In FIG. 9, a part of the two-dimensional tomographic images in the three-dimensional tomographic image before and after the image alignment is compared. The left side of FIG. 9 is the two-dimensional tomographic image before the image alignment, and the right side of FIG. 9 is the two-dimensional tomographic image after the image alignment. As shown in FIG. 9, by performing the image alignment, the positional deviation in the Z direction of the image including the optic disc is reduced. As a result, in the processes of S12 and S13 described later, the end portion of the optic disc is detected with higher accuracy.

[0107] As an example, in S2 of the present embodiment, image alignment in the Z direction is performed among a plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image. Further, for each of the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image, image alignment is performed among a plurality of pixel columns (in the present embodiment, a plurality of A-scan images extending in the Z direction) constituting the two-dimensional tomographic image.

[0108] Note that instead of the process of S2, image alignment in the Z direction may be performed on the plurality of two-dimensional tomographic images extracted in S11 described later. Even in this case, the detection accuracy of the end portion of the papilla is improved. Also, it is possible to omit the image alignment process.

[0109] Next, the CPU 3 executes a site identification process (S3). The site identification process is a process of identifying a specific site in a two-dimensional measurement region 40 (see FIG. 3) where the three-dimensional tomographic image is taken, based on an image of the fundus of the eye to be examined. The site identification process (S3) of the present embodiment is executed as an automatic papilla detection process. The site identification process (S3) of the present embodiment is a preparatory stage process for detecting the position of the end portion of the papilla with high accuracy in the processes described later. Details of the site identification process will be described later with reference to FIGS. 15 to 17.

[0110] When the automatic detection of the papilla is successful (S5: YES), a reference position is set at the detected position of the papilla (in the present embodiment, the center position of the papilla automatically detected in S3) (S6). As shown in FIG. 10, the reference position RP serves as a reference for setting the radial pattern 60 described later.

[0111] On the other hand, when the automatic detection of the optic disc fails (S5: NO), the CPU 3 sets a reference position at the position specified by the user (S7). In the present embodiment, the CPU 3 receives an input of an instruction from the user while displaying a fundus image (for example, a two-dimensional front image or the like) of the eye to be examined on the display device 8. When the user inputs an instruction to specify a position via the operation unit 7, the CPU 3 sets the reference position at the specified position. In some cases, it is also possible to set the reference position by the process of S7 without executing the processes of S3, S5, and S6.

[0112] Next, the CPU 3 sets a radial pattern centered on the reference position for the two-dimensional measurement region 40 (S10). As shown in FIG. 10, in the process of S10, a pattern of lines 61 that radiate radially centering on the reference position RP is set as the radial pattern 60. When the reference position RP is correctly set within the region of the optic disc, all of the plurality of lines 61 that constitute the radial pattern 60 pass through the edge of the optic disc. As an example, in the radial pattern 60 shown in FIG. 10, 16 lines 61 of the same length with one end being the reference position RP extend at the same intervals in a direction away from the reference position RP.

[0113] Next, the CPU 3 extracts a two-dimensional tomographic image 64 (see FIG. 11) at each of the lines 61 of the radial pattern 60 set in S10 from the three-dimensional tomographic image acquired in S1 (S11). That is, the CPU 3 extracts a plurality of two-dimensional tomographic images 64 that intersect each of the lines 61 of the radial pattern 60 from the three-dimensional tomographic image. When the reference position RP is correctly set within the region of the optic disc, all of the two-dimensional tomographic images 64 extracted in S11 will include the edge of the optic disc. The BMO 67 of the optic disc is shown in the two-dimensional tomographic image 64 shown in FIG. 11.

[0114] The CPU 3 acquires the position of the end portion of the papilla (BMO in this embodiment) in each of the plurality of two-dimensional tomographic images 64 extracted in S11 (S12). In this embodiment, the CPU 3 inputs the two-dimensional tomographic image 64 into a mathematical model. The mathematical model is trained by a machine learning algorithm so as to output a detection result of the position of the BMO shown in the input two-dimensional tomographic image. Specifically, as shown in FIG. 11, when the two-dimensional tomographic image 64 is input, the mathematical model of this embodiment outputs a probability map 65 showing the distribution of the probability of the position of the BMO 67 in the region of the input two-dimensional tomographic image 64. In the probability map 65 shown in FIG. 11, the position 68 where the BMO 67 actually exists is white, indicating that the probability of being the BMO 67 is high. The CPU 3 detects the position of the BMO by acquiring the detection result of the position of the BMO output by the mathematical model (the position where the probability map 65 is maximized).

[0115] Furthermore, in this embodiment, the position of the BMO automatically detected by using the machine learning algorithm is corrected according to an instruction from the user. Specifically, as shown in FIG. 12, the CPU 3 displays the position that is the automatic detection result with the highest probability of being the BMO on the two-dimensional tomographic image extracted in S11. When the position displayed based on the automatic detection result is incorrect, the user inputs the correct position of the BMO via the operation unit 7 or the like. The CPU 3 detects the position input by the user as the position of the BMO. Note that the CPU 3 can also detect the position instructed by the user as the position of the BMO without using the machine learning algorithm.

[0116] Next, the CPU 3 detects the position of the annular end portion of the papilla (the annular BMO in this embodiment) (S13) by performing a smoothing process on the detection results of a plurality of positions detected based on the plurality of two-dimensional tomographic images 64. As a result, even if the position of the end portion is erroneously detected for some of the two-dimensional tomographic images 64, the influence of the erroneous detection is suppressed. As an example, in this embodiment, for each of the X, Y, and Z dimensions of the detection results of a plurality of positions detected based on the plurality of two-dimensional tomographic images 64, a one-dimensional Gaussian filter smoothing process is performed. Also, before the position of the BMO is detected, a three-dimensional Gaussian filter smoothing process may be performed on the plurality of probability maps 65. Further, elliptical fitting or the like for the plurality of detection results may be used for smoothing.

[0117] Based on the position of the end portion of the papilla detected in S12 and S13, the CPU 3 specifies the central position of the papilla (S14). As an example, in this embodiment, the CPU 3 specifies the barycentric position of the detected annular BMO in the XY plane as the central position of the papilla in the XY plane.

[0118] The CPU 3 causes the display device 8 to display the detected position of the end portion of the papilla (S20). In this embodiment, as shown in FIG. 13, the detected position 70 of the annular BMO is superimposed and displayed on the two-dimensional front image of the fundus of the eye to be examined. Specifically, the CPU 3 performs spline interpolation on the detected positions of the plurality of BMOs in the XY plane and displays the contour line of the BMO. Therefore, the user can appropriately grasp the two-dimensional position of the BMO.

[0119] Next, as shown in FIG. 13, the CPU 3 sets an annular (a circular ring in this embodiment) line pattern 71 centered on the central position CP of the papilla specified in S14 for the two-dimensional measurement region (S21). The diameter of the line pattern 71 is predetermined, but may be changed according to an instruction from the user.

[0120] The CPU 3 extracts, from the three-dimensional tomographic image acquired in S1, a two-dimensional tomographic image in the annular line pattern 71 set in S21 (that is, an image obtained by deforming two-dimensionally a tomographic image that intersects the annular line pattern 71 in a cylindrical shape) (S22).

[0121] The CPU 3 generates a layer thickness graph indicating the thickness of a specific layer of the retina shown in the two-dimensional tomographic image (for example, the thickness of the NFL or the thickness from the ILM to the NFL) by processing the two-dimensional tomographic image extracted in S22 (S23).

[0122] The CPU 3 causes the display device 8 to display the layer thickness graph generated in S23 in a state where it is compared with the data of a normal eye. FIG. 14 shows an example of a display method of the two-dimensional tomographic images 75R, 75L and the layer thickness graphs 76R, 76L. In the example shown in FIG. 14, for each of the right eye and the left eye of the subject, the two-dimensional tomographic images 75R, 75L extracted in S22 are displayed. Further, the layer thickness graphs 76R, 76L generated in S23 are displayed side by side with the corresponding two-dimensional tomographic images 75R, 75L. Inside the layer thickness graphs 76R, 76L, the range of the data of a normal eye is displayed together with a graph indicating the thickness of a specific layer analyzed based on the two-dimensional tomographic images 75R, 75L. Therefore, the user can appropriately grasp the state of the eye to be examined.

[0123] (Region detection process) With reference to FIGS. 15 to 17, the region detection process executed by the fundus image processing apparatus 1 will be described. The region detection process is executed by the CPU 3 in accordance with the fundus image processing program stored in the storage device 4. In the region detection process, a second region different from the first region is identified based on the degree of divergence of the probability distribution when the mathematical model identifies the first region. As described above, in the present embodiment, when automatically detecting a certain position of the optic disc as the second region, the region identification process is executed.

[0124] Generally, a plurality of layers and boundaries normally exist around the papilla, but specific layers and boundaries are missing at the position of the papilla. Specifically, at the position where the papilla exists, the NFL exists, while the layers and boundaries at positions deeper than the NFL are missing. Based on the above findings, in the site detection process of this embodiment, the papilla is identified based on the degree of divergence of the probability distribution when the mathematical model identifies specific layers and boundaries (layers and boundaries at positions deeper than the NFL).

[0125] As shown in FIG. 15, the CPU 3 acquires a fundus image of the eye to be examined that is the detection target of the second site (the papilla in this embodiment) (S31). In this embodiment, as the fundus image, a three-dimensional tomographic image 43 (see FIG. 5) of the fundus of the eye to be examined is acquired, and the second site is detected based on the three-dimensional tomographic image 43. Therefore, the second site is detected based on more data than when the second site is detected from a two-dimensional fundus image. If the three-dimensional tomographic image 43 has already been acquired in S1 of FIG. 8, the process of S31 may be omitted.

[0126] Next, the CPU 3 acquires a two-dimensional front image when viewing the fundus where the three-dimensional tomographic image 43 acquired in S31 (or S1) was taken from the front (that is, the direction along the OCT measurement light) (S32). As an example, in S32 of this embodiment, an Enface image 45 (see FIG. 5) generated based on the data of the three-dimensional tomographic image 43 acquired in S31 is acquired as the two-dimensional front image. However, the two-dimensional front image may be an image taken by a principle different from the imaging principle of the three-dimensional tomographic image 43 (for example, a two-dimensional front image taken by the front observation optical system 23, etc.).

[0127] The CPU 3 acquires an auxiliary identification result of the second site (the papilla in this embodiment) based on the two-dimensional front image acquired in S32 (S33). The method for auxiliary identification of the second site for the two-dimensional front image can be appropriately selected. In this embodiment, the CPU 3 identifies the papilla by performing known image processing on the two-dimensional front image.

[0128] Based on the auxiliary identification result obtained in S33, the CPU 3 extracts a portion highly likely to contain the second part (the papilla in this embodiment) from the entire three-dimensional tomographic image 43 obtained in S31 (or S1) (S34). As a result, the amount of subsequent processing is reduced, so that the second part can be detected more appropriately.

[0129] The CPU 3 extracts the T-th (the initial value of T is "1") two-dimensional tomographic image from among the plurality of two-dimensional tomographic images constituting the three-dimensional tomographic image extracted in S34 (S36). FIG. 16 shows an example of the extracted two-dimensional tomographic image 42. In the two-dimensional tomographic image 42, a plurality of layers and boundaries in the fundus of the eye to be examined are shown. Also, a plurality of one-dimensional regions A1 to AN are set in the two-dimensional tomographic image 42. In this embodiment, the one-dimensional regions A1 to AN set in the two-dimensional tomographic image 42 extend along an axis intersecting a specific layer or boundary. Specifically, the one-dimensional regions A1 to AN of this embodiment coincide with the regions of each of the plurality (N) of A-scans constituting the two-dimensional tomographic image 42 photographed by the OCT apparatus 10.

[0130] By inputting the T-th two-dimensional tomographic image into the mathematical model, the CPU 3 obtains, as the probability distribution for identifying the first part (a specific layer or boundary), the probability distribution of the coordinates where the M-th (the initial value of M is "1") layer or boundary exists in each of the plurality of one-dimensional regions A1 to AN (S37). The CPU 3 obtains the degree of divergence of the probability distribution regarding the M-th layer or boundary (S38). The degree of divergence is the difference between the probability distribution obtained in S37 and the probability distribution when the first tissue is accurately identified. In the one-dimensional region where the first tissue exists, the degree of divergence tends to be small. On the other hand, in the one-dimensional region where the first tissue does not exist, the degree of divergence tends to be large. This tendency is likely to appear regardless of the presence or absence of eye diseases.

[0131] In this embodiment, as the degree of divergence, the entropy of the probability distribution P is calculated. The entropy is given by the following (Equation 1). The entropy H(P) takes a value of 0 ≤ H(P) ≤ log(number of events), and the more biased the probability distribution P is, the smaller the value becomes. That is, the smaller the entropy H(P) is, the higher the discrimination accuracy of the first tissue tends to be. The entropy of the probability distribution when the first tissue is accurately discriminated is 0. H(P)=-Σplog(p)···(Equation 1)

[0132] Next, the CPU 3 determines whether the degree of divergence of all the layers and boundaries to be identified in the T-th two-dimensional tomographic image has been acquired (S40). If the degree of divergence of some of the layers and boundaries has not been acquired (S40: NO), "1" is added to the order M of the layers and boundaries (S41), and the process returns to S37, and the degree of divergence of the next layer and boundary is acquired (S37, S38). When the degree of divergence of all the layers and boundaries has been acquired (S40: YES), the CPU 3 stores the degree of divergence of the T-th two-dimensional tomographic image in the storage device 4 (S42).

[0133] Next, the CPU 3 determines whether the degree of divergence of all the two-dimensional tomographic images constituting the three-dimensional tomographic image has been acquired (S44). If the degree of divergence of some of the two-dimensional tomographic images has not yet been acquired (S44: NO), "1" is added to the order T of the two-dimensional tomographic images (S45), and the process returns to S36, and the degree of divergence of the next two-dimensional tomographic image is acquired (S36 to S42).

[0134] When the deviation degrees of all two-dimensional tomographic images are acquired (S44: YES), the CPU 3 acquires a two-dimensional distribution of the magnitudes of the deviation degrees when the fundus is viewed from the front (hereinafter, may be simply referred to as "deviation degree distribution") (S47). In the present embodiment, as shown in FIG. 17, the CPU 3 acquires the deviation degree distribution of a specific layer / boundary among a plurality of layers / boundaries in the fundus. Specifically, at the position where the papilla exists, the NFL exists, while the layers and boundaries at positions deeper than the NFL are missing. Therefore, at the position where the papilla exists, the deviation degree regarding the identification of the layers and boundaries at positions deeper than the NFL is larger than that at the position where the papilla does not exist. Thus, in S47 of the present embodiment, in order to identify the papilla with high accuracy, the deviation degree distribution of the layers / boundaries at positions deeper than the NFL (specifically, a plurality of layers / boundaries including the IPL / INL and the BM) is acquired. In the deviation degree distribution shown in FIG. 17, the parts with large deviation degrees are represented in bright colors.

[0135] The CPU 3 acquires a distribution of scores indicating the possibility that each part (each A-scan image) is the second part (hereinafter, referred to as "score distribution of the second part") (S48). As described above, the score distribution of the second part is output by the mathematical model together with the identification result of the first part.

[0136] Next, the CPU 3 generates an identification result of the second part based on the deviation degree when the mathematical model identifies the first part (S49). In the present embodiment, as shown in FIG. 17, the CPU 3 integrates (adds together) the deviation degree distribution of the layers / boundaries at positions deeper than the NFL and the score distribution of the second part. The CPU 3 generates an identification result of the second part by performing binarization processing on the integrated distribution. Note that arbitrary weighting may be performed when integrating the deviation degree distribution and the score distribution.

[0137] (Modification example) The technology disclosed in the above embodiment is merely an example. Therefore, it is also possible to modify the technology exemplified in the above embodiment. For example, the CPU 3 may detect structures other than the optic disc in the fundus based on the detection result of the end of the optic disc detected by fundus image processing (see FIG. 8). In the example shown in FIG. 18, the CPU 3 detects the position of the optic nerve cup (Cup) 87 based on the position of the BMO 85 detected by fundus image processing. Specifically, the CPU 3 sets a straight line L2 that is parallel to the reference straight line L1 passing through the pair of detected BMOs 85 and is spaced a predetermined distance from the reference straight line L1 toward the surface side of the retina. The CPU 3 detects the position where the set straight line L2 intersects the ILM (inner limiting membrane) 89 in the fundus image as the position of the Cup 87. Further, the CPU 3 detects the shortest distance between the position of the BMO 85 detected by fundus image processing and the ILM 89 in the fundus image as the minimum thickness of the nerve fiber layer (Minimum Rim Width). According to fundus image processing, the position of the end of the optic disc is detected with high accuracy. Therefore, by detecting structures other than the optic disc based on the detected position of the end of the optic disc, structures other than the optic disc can also be detected with high accuracy.

[0138] In S3 of the fundus image processing (see FIG. 8) of the above embodiment, the site identification process shown in FIG. 15 is used to automatically detect the optic disc. However, it is also possible to modify the process in S3 of FIG. 8. For example, the CPU 3 may automatically detect the position of the optic disc based on a two-dimensional frontal image of the fundus of the eye to be examined (that is, a two-dimensional image when viewed from the direction along the optical axis of the OCT measurement light). In this case, the CPU 3 may detect the position of the optic disc by performing known image processing on the two-dimensional frontal image. The CPU 3 may also detect the position of the optic disc by inputting the two-dimensional frontal image into a mathematical model that detects and outputs the position of the optic disc. Various images such as the aforementioned Enface image 45, fundus camera image, or SLO image can be used as the two-dimensional frontal image.

[0139] In S10 of FIG. 8, the specific method for setting the radial pattern 60 centered on the reference position RP can also be changed as appropriate. For example, the CPU 3 may acquire information on the position of the fundus blood vessels in the measurement region 40 where the three-dimensional tomographic image is taken. The CPU 3 adjusts at least any one of the overall angle of the radial pattern 60, the angle of at least any one of the lines 61 included in the radial pattern 60, the length of the line 61, the number of the lines 61, etc., so that the overlapping amount between the line 61 of the radial pattern 60 and the fundus blood vessels is reduced as much as possible. In this case, it is appropriately suppressed that the detection accuracy of the end of the papilla is reduced due to the presence of the fundus blood vessels. Further, the CPU 3 may adjust at least any one of the overall angle of the radial pattern 60, the angle of at least any one of the lines 61 included in the radial pattern 60, the length of the line 61, the number of the lines 61, etc., according to an instruction input by the user who has confirmed the fundus image. In this case, the detection accuracy of the end of the papilla is further improved.

[0140] In the fundus image processing of the above embodiment (see FIG. 8), as the reference position RP is closer to the actual center position of the papilla, the positions of the ends of the papilla in each of the plurality of two-dimensional tomographic images 64 extracted according to the radial pattern 60 are approximated, so that the detection accuracy of the annular end of the papilla is also increased. The reference position RP set in S6 and S7 may be away from the actual center position of the papilla. Therefore, when the CPU 3 has executed the detection process of the end of the papilla shown in S3 to S14 only once, after executing the process of S14, the CPU 3 may reset the reference position RP to the center position specified in S14 and execute the processes of S10 to S14 again. The center position of the papilla specified in S14 is likely to be more accurate than the center position detected by the process such as S3. Therefore, by redetecting the end of the papilla with the center position of the papilla specified in S14 as the reference position RP, the detection accuracy is further improved. Note that the number of times of repeating the processes of S10 to S14 can be set as appropriate. For example, when the center positions of the papilla specified a plurality of times in S14 converge within a certain range, the CPU 3 may execute the processes after S21.

[0141] In the above-described embodiment, the site identification process shown in FIG. 15 is executed as part of the fundus image process shown in FIG. 8. However, it is also possible to execute the site identification process shown in FIG. 15 independently. In this case, it is also possible to detect sites other than the optic disc in the fundus image. Generally, a plurality of layers and boundaries normally exist around the fovea, but specific layers and boundaries are missing at the position of the fovea. Specifically, at the position where the fovea exists, while the RPE and Bruch's membrane exist, the layers and boundaries on the surface side of the retina relative to the RPE are missing. Based on the above findings, the fundus image processing apparatus 1 may identify the fovea (second site) based on the degree of divergence of the probability distribution when the mathematical model identifies the layers and boundaries (first site) on the surface side of the retina relative to the RPE. In this case, in S37 to S47 of FIG. 15, as the degree of divergence related to the analysis of the first site, at least one degree of divergence distribution of the layers and boundaries on the surface side relative to the RPE is obtained. In S49, the fovea is identified as the second site. As a result, the fovea is identified with high accuracy.

[0142] Also, at the position where the fundus blood vessels (second site) exist, since the measurement light is blocked by the fundus blood vessels, the imaging state of the layers and boundaries (first site) at a position deeper than the fundus blood vessels tends to deteriorate. Therefore, at the position where the fundus blood vessels exist, the degree of divergence related to the identification of the layers and boundaries at a position deeper than the fundus blood vessels is larger than at the position where the fundus blood vessels do not exist. Based on the above findings, in S47, as the degree of divergence related to the analysis of the first site, at least one degree of divergence distribution of the layers and boundaries at a position deeper than the fundus blood vessels may be obtained. In S49, a site where the degree of divergence is larger than the threshold value may be identified as the site of the fundus blood vessels (second site).

[0143] It is also possible to execute only a part of the plurality of technologies exemplified in the above embodiment. For example, in S33 and S34 (see FIG. 15) of the above embodiment, the auxiliary identification result of the second part executed based on the two-dimensional front image is used. However, the second part may be identified without using the auxiliary identification result. Also, in S48 and S49 (see FIG. 15) of the above embodiment, the score distribution of the second part is used. However, the second part may be identified without using the score distribution of the second part.

[0144] Note that the process of acquiring the three-dimensional tomographic image in S1 of FIG. 8 is an example of the "image acquisition step". The process of setting the reference position in S6 and S7 of FIG. 8 is an example of the "reference position setting step". The process of setting the radial pattern in S10 of FIG. 8 is an example of the "radial pattern setting step". The process of extracting the two-dimensional tomographic image in S11 of FIG. 8 is an example of the "image extraction step". The process of detecting the position of the end of the nipple in S12 and S13 of FIG. 8 is an example of the "nipple end detection step". The process of performing image alignment in S2 of FIG. 8 is an example of the "alignment step". The process of automatically detecting the position of the nipple in S3 of FIG. 8 is an example of the "nipple position detection step". The process of specifying the center position of the nipple in S14 of FIG. 8 is an example of the "nipple center specification step". The process of extracting the two-dimensional tomographic image in S22 of FIG. 8 is an example of the "annular extraction step". The process of outputting information regarding the two-dimensional tomographic image in S24 of FIG. 8 is an example of the "output step".

[0145] The process of acquiring the fundus image in S31 of FIG. 15 is an example of the "image acquisition step". The process of acquiring the degree of deviation in S37 to S47 of FIG. 15 is an example of the "degree of deviation acquisition step". The process of identifying the second part in S49 of FIG. 15 is an example of the "part identification step". The process of acquiring the two-dimensional front image in S32 of FIG. 15 is an example of the "front image acquisition step". The process of acquiring the auxiliary identification result in S33 of FIG. 15 is an example of the "auxiliary identification result acquisition step".

Explanation of Reference Numerals

[0146] 1 Fundus image processing device 3 CPU 4 Memory device 10(10A, 10B) OCT device 40 Measurement area 41 Scanning line 42 Two-dimensional tomographic image 43 Three-dimensional tomographic image 60 Radial pattern 65 Probability map 76R, 76L Thickness graph

Claims

1. An ophthalmic fundus image processing apparatus for processing an ophthalmic fundus image, wherein a control unit of the ophthalmic fundus image processing apparatus performs an image acquisition step of acquiring an ophthalmic fundus image captured by an ophthalmic fundus imaging device, a divergence degree acquisition step of inputting the ophthalmic fundus image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying a first part of the fundus shown in the ophthalmic fundus image, and obtaining a divergence degree of the obtained probability distribution with respect to the probability distribution when the first part is accurately identified, and a part identification step of identifying a second part of the fundus different from the first part based on the divergence degree. An ophthalmic fundus image processing apparatus characterized by performing the above steps.

2. The ophthalmic fundus image processing apparatus according to Claim 1, wherein in the divergence degree acquisition step, the divergence degree is obtained by using at least any one of a plurality of layers and boundaries in the fundus shown in the ophthalmic fundus image as the first part.

3. The ophthalmic fundus image processing apparatus according to Claim 2, wherein in the part identification step, the optic disc in the fundus is identified as the second part.

4. The ophthalmic fundus image processing apparatus according to Claim 3, wherein in the image acquisition step, a three-dimensional tomographic image of the fundus is acquired as the ophthalmic fundus image, and a reference position setting step of setting a reference position within the region of the optic disc identified in the part identification step among two-dimensional measurement regions where the three-dimensional tomographic image is captured, a radial pattern setting step of setting a radial pattern, which is a line pattern radially spreading centering on the reference position, for the two-dimensional measurement region, an image extraction step of extracting two-dimensional tomographic images in each of a plurality of lines of the set radial pattern from the three-dimensional tomographic image, and a papilla end detection step of detecting a position of an end of the papilla shown in the three-dimensional tomographic image based on the extracted plurality of two-dimensional tomographic images. The ophthalmic fundus image processing apparatus further characterized by performing the above steps.

5. The ophthalmic fundus image processing apparatus according to Claim 2, wherein in the part identification step, the fovea centralis in the fundus is identified as the second part.

6. The ophthalmic fundus image processing apparatus according to any one of Claims 1 to 5, In the image acquisition step, a three-dimensional tomographic image of the fundus is acquired as the fundus image, In the deviation degree acquisition step, by inputting the three-dimensional tomographic image into the mathematical model, a two-dimensional distribution of the deviation degree when the fundus is viewed from the front is acquired, In the part identification step, the position of the second part of the fundus when the fundus is viewed from the front is identified based on the two-dimensional distribution of the deviation degree. A fundus image processing apparatus characterized by this.

7. A fundus image processing program executed by a fundus image processing apparatus that processes a fundus image of an eye to be examined, By the fundus image processing program being executed by the control unit of the fundus image processing apparatus, An image acquisition step of acquiring a fundus image photographed by a fundus image photographing apparatus, By inputting the fundus image into a mathematical model trained by a machine learning algorithm, a probability distribution for identifying a first part of the fundus shown in the fundus image is acquired, and the deviation degree of the acquired probability distribution with respect to the probability distribution when the first part is accurately identified is acquired. A deviation degree acquisition step, A part identification step of identifying a second part different from the first part of the fundus based on the deviation degree, A fundus image processing program characterized by causing the fundus image processing apparatus to execute the above.

Images