Ophthalmic information processing apparatus, ophthalmic apparatus, method of operating an ophthalmic information processing apparatus, and program

Abstract

To provide a new technique for performing appropriate treatment for a disease of a subject eye at an early stage.SOLUTION: An ophthalmologic information processing device includes an acquisition unit, multiple estimators, and a classifier. The acquisition unit acquires multiple images of a subject eye with mutually-different cross-sectional directions. The multiple estimators use respective learned models obtained through machine learning for each type of multiple images to output a feature amount or certainty information representing certainty that the subject eye is an eye with age-related macular degeneration for each type of images. The classifier uses a classification model obtained through machine learning to output estimation information for estimating whether or not the subject eye is the eye with the age-related macular degeneration on the basis of the multiple feature amounts or the multiple pieces of certainty information output from the multiple estimators.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an ophthalmic information processing apparatus, an ophthalmic apparatus, an operating method of the ophthalmic information processing apparatus, and a program.

Background Art

[0002] In recent years, due to the rapid progress of machine learning techniques represented by deep learning, the practical application of artificial intelligence technology has advanced in various fields. In particular, in the medical field, the detection accuracy of disease sites or tissue patterns in diagnostic images has been improved by deep learning, enabling accurate and highly precise medical diagnoses to be made quickly.

[0003] For example, Patent Document 1 discloses a method of performing machine learning using characteristic points in a plurality of past time-series fundus images as teacher data, and diagnosing glaucoma from a plurality of newly acquired time-series fundus images using the obtained learned model. For example, Patent Document 2 discloses a method of classifying the pathological conditions of glaucoma according to the shape of the optic disc using a neural network.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, fundus diseases such as glaucoma, age-related macular degeneration, or diabetic retinopathy are diseases in which symptoms progress if appropriate treatment is not given even if they are detected early by screening or the like. In this case, appropriate treatment can be given to the disease by appropriately grasping the pathological condition of the disease.

[0006] Thus, it is required to accurately identify the condition of a detected fundus disease while detecting the fundus disease with high precision and at an early stage. This applies not only to fundus diseases but also to all diseases of the eye to be examined.

[0007] The present invention has been made in view of such circumstances, and an object thereof is to provide a new technique for early treatment of a disease of an eye to be examined.

Means for Solving the Problems

[0008] A first aspect of some embodiments includes an acquisition unit that acquires a plurality of images of an eye to be examined having different cross-sectional directions from each other, and uses a plurality of learned models obtained by machine learning for each type of the plurality of images to output estimation information for estimating whether the eye to be examined is an eye with a disease from the plurality of images. It is an ophthalmic information processing device including a disease estimation unit.

[0009] In a second aspect of some embodiments, in the first aspect, the disease estimation unit uses each of the plurality of learned models for each type of the plurality of images to output a feature amount or confidence information representing the confidence that the eye to be examined is an eye with a disease. A plurality of estimators, and a classifier that outputs the estimation information from the plurality of feature amounts or the plurality of confidence information output from the plurality of estimators using a classification model obtained by machine learning.

[0010] In a third aspect of some embodiments, in the first aspect or the second aspect, the plurality of images include a tomographic image or a frontal image of the fundus of the eye to be examined, and the disease is a fundus disease.

[0011] In a fourth aspect of some embodiments, in the third aspect, the plurality of images include a B-scan image in a cross-sectional direction that intersects a line connecting the optic nerve head and the fovea centralis.

[0012] In a fifth aspect of some embodiments, in the third or fourth aspect, the plurality of images include a B-scan image in a first cross-sectional direction passing through the optic nerve head, a B-scan image in a second cross-sectional direction passing through the optic nerve head and intersecting the first cross-sectional direction, and a B-scan image passing through the fovea centralis.

[0013] In a sixth aspect of some embodiments, in any one of the third to fifth aspects, the frontal image includes at least one of an en-face image obtained by projecting a deep layer region from a layer region corresponding to the inner limiting membrane, and a projection image.

[0014] In a seventh aspect of some embodiments, in any one of the third to sixth aspects, the fundus disease includes at least one of glaucoma, age-related macular degeneration, and diabetic retinopathy.

[0015] An eighth aspect of some embodiments includes a pathological state estimation unit that outputs classification information for estimating a classification result of the pathological state of the disease from the plurality of images, using a learned classification model obtained by supervised machine learning using feature amounts obtained by the plurality of learned models, in any one of the first to seventh aspects.

[0016] In a ninth aspect of some embodiments, in the eighth aspect, the disease includes glaucoma, and the classification information is information for estimating classification results of a plurality of types of optic nerve head shapes.

[0017] A tenth aspect of some embodiments includes a first learning unit that generates the learned classification model by supervised machine learning using feature amounts obtained by the plurality of learned models, in the eighth or ninth aspect.

[0018] An eleventh aspect of some embodiments includes a second learning unit that generates the plurality of learned models by supervised machine learning for each type of the plurality of images, in any one of the first to tenth aspects.

[0019] In a twelfth aspect of some embodiments, in any of the first to eleventh aspects, an image generation unit that generates at least one of the plurality of images based on the three-dimensional OCT data of the eye to be examined is included.

[0020] A thirteenth aspect of some embodiments is an ophthalmic apparatus including an OCT unit that performs optical coherence tomography on the eye to be examined, an image generation unit that generates at least one of the plurality of images based on the three-dimensional data obtained by the OCT unit, and an ophthalmic information processing apparatus according to any of the first to eleventh aspects.

[0021] A fourteenth aspect of some embodiments is an ophthalmic apparatus including an OCT unit that performs optical coherence tomography on the eye to be examined and the ophthalmic information processing apparatus according to the twelfth aspect.

[0022] A fifteenth aspect of some embodiments is an operating method of an ophthalmic information processing apparatus, including an acquisition step of acquiring a plurality of images of an eye to be examined having different cross-sectional directions from each other, and a disease estimation step of outputting estimation information for estimating whether the eye to be examined is an eye with a disease from the plurality of images using a plurality of learned models obtained by machine learning for each type of the plurality of images.

[0023] In a sixteenth aspect of some embodiments, in the fifteenth aspect, the disease estimation step includes a plurality of estimation steps of outputting feature amounts or confidence information representing a confidence level that the eye to be examined is an eye with a disease using each of the plurality of learned models for each type of the plurality of images, and a classification step of outputting the estimation information from the plurality of feature amounts or the plurality of confidence information output in the plurality of estimation steps using a classification model obtained by machine learning.

[0024] In a seventeenth aspect of some embodiments, in the fifteenth or sixteenth aspect, the plurality of images include a tomographic image or a frontal image of the fundus of the eye to be examined, and the disease is a fundus disease.

[0025] In the 18th aspect of some embodiments, in the 17th aspect, the plurality of images include B-scan images in a cross-sectional direction that intersect a line connecting the optic nerve head and the fovea centralis.

[0026] In the 19th aspect of some embodiments, in the 17th or 18th aspect, the plurality of images include a B-scan image in a first cross-sectional direction passing through the optic nerve head, a B-scan image in a second cross-sectional direction passing through the optic nerve head and intersecting the first cross-sectional direction, and a B-scan image passing through the fovea centralis.

[0027] In the 20th aspect of some embodiments, in any one of the 17th to 19th aspects, the frontal image includes at least one of an en-face image obtained by projecting a deep layer region from a layer region corresponding to the inner limiting membrane and a projection image.

[0028] In the 21st aspect of some embodiments, in any one of the 17th to 20th aspects, the fundus disease includes at least one of glaucoma, age-related macular degeneration, and diabetic retinopathy.

[0029] The 22nd aspect of some embodiments includes a pathological state estimation step of outputting classification information for estimating a classification result of the pathological state of the disease from the plurality of images, using a learned classification model obtained by supervised machine learning using the feature amounts obtained by the plurality of learned models, in any one of the 15th to 21st aspects.

[0030] In the 23rd aspect of some embodiments, in the 22nd aspect, the disease includes glaucoma, and the classification information is information for estimating classification results of a plurality of types of optic nerve head shapes.

[0031] The 24th aspect of some embodiments includes a first learning step of generating the learned classification model by supervised machine learning using the feature amounts obtained by the plurality of learned models, in the 22nd or 23rd aspect.

[0032] The 25th aspect of some embodiments includes a second learning step of generating the plurality of trained models by supervised machine learning for each type of the plurality of images in any one of the 15th to 24th aspects.

[0033] The 26th aspect of some embodiments includes an image generation step of generating at least one of the plurality of images based on the three-dimensional OCT data of the eye to be examined in any one of the 15th to 25th aspects.

[0034] The 27th aspect of some embodiments is a program for causing a computer to execute each step of the operation method of the ophthalmic information processing apparatus according to any one of the 15th to 26th aspects.

[0035] It should be noted that the configurations according to the above-described plurality of aspects can be arbitrarily combined.

Advantages of the Invention

[0036] According to some embodiments of the present invention, it is possible to provide a new technique for early performing appropriate treatment for a disease of an eye to be examined.

Brief Description of the Drawings

[0037]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0038] Examples of an ophthalmic information processing apparatus, an ophthalmic apparatus, an ophthalmic information processing method, and a program according to some embodiments of the present invention will be described in detail with reference to the drawings. Note that matters described in the documents cited in this specification and any arbitrary known techniques can be incorporated into the embodiments.

[0039] The ophthalmic information processing apparatus according to an embodiment can perform predetermined analysis processing and predetermined display processing on three-dimensional OCT data of an eye to be examined acquired by an ophthalmic apparatus using optical coherence tomography (OCT). The ophthalmic apparatus according to some embodiments has not only the function of an OCT apparatus for executing OCT, but also at least one function of a fundus camera, a scanning laser ophthalmoscope, a slit lamp microscope, and a surgical microscope. Further, the ophthalmic apparatus according to some embodiments has a function of measuring optical characteristics of the eye to be examined. Examples of the ophthalmic apparatus having a function of measuring optical characteristics of the eye to be examined include a refractometer, a keratometer, a tonometer, a wavefront analyzer, a specular microscope, and a perimeter. The ophthalmic apparatus according to some embodiments has a function of a laser treatment apparatus used for laser treatment.

[0040] The ophthalmic information processing device generates a plurality of images (tomographic images or frontal images) with different cross-sectional directions from the acquired three-dimensional OCT data of the eye to be examined, and outputs estimation information for estimating whether the eye to be examined is an eye with a disease from the generated plurality of images. Further, the ophthalmic information processing device outputs classification information for estimating the classification result of the pathological condition of the disease from the generated plurality of images (or their feature amounts).

[0041] Hereinafter, the case where the ophthalmic information processing device, the ophthalmic device, the ophthalmic information processing method, and the program according to the embodiment are applied to fundus diseases will be described. However, the application fields of the ophthalmic information processing device, the ophthalmic device, the ophthalmic information processing method, and the program according to the embodiment are not limited to fundus diseases, and can be applied to all diseases of the eye to be examined.

[0042] Fundus diseases include glaucoma, age-related macular degeneration, diabetic retinopathy, and the like.

[0043] When the fundus disease is glaucoma, for example, in order to classify the pathological condition according to the classification by Nicolela et al. (Nicolela classification) disclosed in Patent Document 2, classification information for estimating the classification result based on the shape of the optic nerve head is output.

[0044] When the fundus disease is age-related macular degeneration, for example, in order to classify whether it is atrophic or exudative, classification information for estimating the classification result based on the morphology of the retinal pigment epithelium layer (for example, the morphology of choroidal neovascularization) is output.

[0045] When the fundus disease is diabetic retinopathy, for example, in order to classify whether it is simple diabetic retinopathy, pre-proliferative diabetic retinopathy, or proliferative diabetic retinopathy, classification information for estimating the classification result based on the morphology and distribution of capillary hemangioma, hemorrhage, leukoplakia, neovascularization, and hemorrhage area is output.

[0046] In this way, by being able to provide not only the estimated information regarding the fundus disease but also the classification information regarding the pathological classification of the fundus disease to a doctor or the like, it becomes possible to determine an appropriate treatment method for the fundus disease at an early stage. Thereby, it becomes possible to increase the possibility of curing the fundus disease.

[0047] For example, in the case of a case where it is determined that the fundus disease is glaucoma, among drug therapy, laser treatment, surgery, etc., an appropriate treatment method according to the pathological condition of the disease can be selected.

[0048] For example, in the case of a case where it is determined that the fundus disease is age-related macular degeneration, for exudative age-related macular degeneration, among drug dosage, photodynamic therapy, laser coagulation, surgery, etc., an appropriate treatment method according to the pathological condition of the disease can be selected.

[0049] For example, in the case of a case where it is determined that the fundus disease is diabetic retinopathy, among retinal photocoagulation and vitreous surgery, etc., an appropriate treatment method according to the pathological condition of the disease can be selected.

[0050] The ophthalmic information processing apparatus according to the embodiment uses a learned model obtained by machine learning for each of a plurality of images generated from 3D OCT data to estimate whether the eye to be examined is an eye with a fundus disease and to perform a pathological classification of the fundus disease.

[0051] The ophthalmic system according to the embodiment includes an ophthalmic information processing apparatus. The ophthalmic information processing method according to the embodiment is executed by the ophthalmic information processing apparatus. The program according to the embodiment causes a computer to execute each step of the ophthalmic information processing method.

[0052] Hereinafter, glaucoma will be described as an example of the fundus disease, but the following embodiments can also be applied to other fundus diseases.

[0053] [Ophthalmic System] FIG. 1 shows a block diagram of a configuration example of an ophthalmic system according to an embodiment. The ophthalmic system 1 according to the embodiment includes an ophthalmic device 10, an ophthalmic information processing device (ophthalmic image processing device, ophthalmic analysis device) 100, an operation device 180, and a display device 190.

[0054] The ophthalmic device 10 performs OCT on the eye to be examined and collects three-dimensional OCT data of the eye to be examined. In this embodiment, the ophthalmic device 10 collects OCT data of the fundus by performing an OCT scan on the fundus of the eye to be examined. The ophthalmic device 10 can acquire an image of the fundus from the acquired OCT data of the fundus. The image of the fundus includes a tomographic image and a frontal image of the fundus. Examples of the tomographic image of the fundus include a B-scan image. Examples of the frontal image of the fundus include a C-scan image, an en-face image, a shadowgram, or a projection image. The ophthalmic device 10 transmits the acquired OCT data of the eye to be examined or the data of the acquired image to the ophthalmic information processing device 100.

[0055] In some embodiments, the ophthalmic device 10 and the ophthalmic information processing device 100 are connected via a data communication network. The ophthalmic information processing device 100 according to some embodiments receives the above data from one of a plurality of ophthalmic devices 10 selectively connected via a data communication network.

[0056] The ophthalmic information processing apparatus 100 generates first estimation information for estimating whether the eye to be examined is a glaucoma eye or not for each of a plurality of images generated from three-dimensional OCT data, using an individual estimation model (trained model) obtained by machine learning. The ophthalmic information processing apparatus 100 generates second estimation information for estimating whether the eye to be examined is a glaucoma eye or not from the plurality of first estimation information generated for each of the plurality of images, using a classification model (trained model) obtained by machine learning. Further, the ophthalmic information processing apparatus 100 outputs classification information for estimating a classification result of a pathological condition from the above plurality of images (or their feature amounts), using a pathological condition estimation model (trained model) obtained by machine learning. The pathological condition estimation model is obtained by performing machine learning using the feature amounts of the above plurality of individual estimation models (or a plurality of individual estimation models and a classification model). Thereby, it becomes possible to acquire highly accurate classification information using a pathological condition estimation model obtained by machine learning using less training data as compared with the case of performing machine learning from a natural image or the like to generate a pathological condition estimation model.

[0057] Hereinafter, the case of generating five types of images from three-dimensional OCT data will be described. The five types of images are a horizontal B-scan image passing through the center (or the vicinity thereof) of the optic nerve head, a vertical B-scan image passing through the center (or the vicinity thereof) of the optic nerve head, a vertical B-scan image passing through the fovea (or the vicinity thereof), a projection image, and an en-face image. However, the configuration according to the embodiment is not limited to the type or the number of types of images generated from three-dimensional OCT data. For example, a plurality of images including a B-scan image in a cross-sectional direction intersecting a line connecting the optic nerve head and the fovea may be generated from the three-dimensional OCT data, and the above estimation information or classification information may be acquired based on the generated plurality of images.

[0058] In some embodiments, the ophthalmic information processing apparatus 100 constructs the above-mentioned learned model and outputs estimated information or classification information using the constructed learned model. In some embodiments, the ophthalmic information processing apparatus 100 outputs estimated information or classification information using a learned model constructed externally. In some embodiments, the ophthalmic information processing apparatus 100 constructs the above-mentioned learned model and outputs the constructed learned model to an external apparatus.

[0059] The operation device 180 and the display device 190 provide functions for exchanging information between the ophthalmic information processing apparatus 100 and its user, such as displaying information, inputting information, and inputting operation instructions, as a user interface unit. The operation device 180 includes operation devices such as levers, buttons, keys, and pointing devices. The operation device 180 according to some embodiments includes a microphone for inputting information by sound. The display device 190 includes a display device such as a flat panel display. In some embodiments, the functions of the operation device 180 and the display device 190 are realized by a device in which a device having an input function such as a touch panel display and a device having a display function are integrated. In some embodiments, the operation device 180 and the display device 190 include a graphical user interface (GUI) for inputting and outputting information.

[0060] [Ophthalmic Apparatus] FIG. 2 shows a block diagram of a configuration example of the ophthalmic apparatus 10 according to an embodiment.

[0061] The ophthalmic device 10 is provided with an optical system for acquiring OCT data of the eye to be examined. The ophthalmic device 10 has a function of executing swept-source OCT, but the embodiment is not limited thereto. For example, the type of OCT is not limited to swept-source OCT, and it may be spectral-domain OCT or the like. Swept-source OCT divides light from a wavelength-sweeping type (wavelength-scanning type) light source into measurement light and reference light, makes the return light of the measurement light that has passed through the object to be measured interfere with the reference light to generate interference light, detects this interference light with a balanced photodiode or the like, and performs Fourier transform or the like on the detection data collected according to the wavelength sweep and the scan of the measurement light to form an image. Spectral-domain OCT divides light from a low-coherence light source into measurement light and reference light, makes the return light of the measurement light that has passed through the object to be measured interfere with the reference light to generate interference light, detects the spectral distribution of this interference light with a spectroscope, and performs Fourier transform or the like on the detected spectral distribution to form an image.

[0062] The ophthalmic device 10 includes a control unit 11, an OCT unit 12, an OCT data processing unit 13, and a communication unit 14.

[0063] The control unit 11 controls each part of the ophthalmic device 10. In particular, the control unit 11 controls the OCT unit 12, the OCT data processing unit 13, and the communication unit 14.

[0064] The OCT unit 12 collects three-dimensional OCT data of the eye to be examined by scanning the eye to be examined using OCT. The OCT unit 12 includes an interference optical system 12A and a scan optical system 12B.

[0065] The interference optical system 12A splits the light from a light source (wavelength-sweeping light source) into measurement light and reference light, causes the return light of the measurement light that has passed through the eye to be interfered with the reference light that has passed through the reference optical path to generate interference light, and detects the generated interference light. The interference optical system 12A includes at least a fiber coupler and a light receiver such as a balanced photodiode. The fiber coupler splits the light from the light source into measurement light and reference light, and causes the return light of the measurement light that has passed through the eye to be interfered with the reference light that has passed through the reference optical path to generate interference light. The light receiver detects the interference light generated by the fiber coupler. The interference optical system 12A may include a light source.

[0066] The scanning optical system 12B receives control from the control unit 11 and changes the incident position of the measurement light on the fundus of the eye to be examined by deflecting the measurement light generated by the interference optical system 12A. The scanning optical system 12B includes, for example, an optical scanner disposed at a position optically substantially conjugate to the pupil of the eye to be examined. The optical scanner includes, for example, a first galvanometer mirror that deflects the measurement light in the horizontal direction, a second galvanometer mirror that deflects the measurement light in the vertical direction, and a mechanism that drives these independently. For example, the second galvanometer mirror is configured to further deflect the measurement light deflected by the first galvanometer mirror. Thereby, the measurement light can be scanned in an arbitrary direction on the fundus plane.

[0067] The detection result (detection signal) of the interference light by the interference optical system 12A is an interference signal indicating the spectrum of the interference light.

[0068] The OCT data processing unit 13 receives control from the control unit 11 and forms three-dimensional data (image data) of the fundus based on the data of the eye to be examined collected by the OCT unit 12.

[0069] For example, the OCT data processing unit 13 forms a reflection intensity profile for each A-line, and arranges the formed plurality of reflection intensity profiles in the B-scan direction (the direction intersecting the A-scan direction (for example, the z direction), for example, the x direction) and the direction intersecting the A-scan direction and the B-scan direction (for example, the y direction) to form three-dimensional OCT data (scan data).

[0070] Also, for example, the OCT data processing unit 13 can form an A-scan image of the eye E to be examined by imaging the reflection intensity profile in the A-line. The OCT data processing unit 13 can form a three-dimensional image by arranging a plurality of A-scan images formed for each A-line in the B-scan direction, the A-scan direction, and the direction intersecting the B-scan direction.

[0071] The processes executed by the OCT data processing unit 13 include processes such as noise removal (noise reduction), filtering, and FFT (Fast Fourier Transform). The image data thus obtained is a data set including a group of image data formed by imaging the reflection intensity profiles in a plurality of A-lines (the paths of respective measurement lights in the eye E to be examined). In order to improve the image quality, a plurality of data sets collected by repeating scans in the same pattern a plurality of times can be superimposed (added and averaged).

[0072] In some embodiments, the OCT data processing unit 13 performs various data processing (image processing) and analysis processing on the image. For example, the OCT data processing unit 13 executes correction processing such as brightness correction and variance correction of the image. The OCT data processing unit 13 can form volume data (voxel data) of the eye E to be examined by executing known image processing such as interpolation processing for interpolating pixels between tomographic images. When displaying an image based on the volume data, the OCT data processing unit 13 performs rendering processing on this volume data to form a pseudo three-dimensional image when viewed from a specific viewing direction.

[0073] Each of the control unit 11 and the OCT data processing unit 13 includes a processor. The processor includes circuits such as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and a programmable logic device (e.g., an SPLD (Simple Programmable Logic Device), a CPLD (Complex Programmable Logic Device), an FPGA (Field Programmable Gate Array)). The function of the OCT data processing unit 13 is realized by a data processing processor. In some embodiments, the functions of both the control unit 11 and the OCT data processing unit 13 are realized by one processor.

[0074] The processor realizes the functions according to the embodiments by, for example, reading and executing a program stored in a memory circuit or a storage device. At least a part of the memory circuit or the storage device may be included in the processor. Also, at least a part of the memory circuit or the storage device may be provided outside the processor.

[0075] The storage device and the like store various data. Examples of the data stored in the storage device and the like include data (measurement data, imaging data, etc.) acquired by the OCT unit 12 and information regarding the subject and the eye to be examined. The storage device and the like may store various computer programs and data for operating each part of the ophthalmic device 10.

[0076] The communication unit 14 receives control from the control unit 11 and executes communication interface processing for transmitting or receiving information to and from the ophthalmic information processing device 100.

[0077] The ophthalmic device 10 according to some embodiments transmits data of the eye to be examined formed by the OCT data processing unit 13 to the ophthalmic information processing device 100.

[0078] The ophthalmic apparatus 10 according to some embodiments is provided with a fundus camera, a scanning laser ophthalmoscope, and a slit lamp microscope for acquiring an image of the fundus of an eye to be examined. In some embodiments, the fundus image acquired by the fundus camera is a fluorescein angiography image or a fundus autofluorescence examination image.

[0079] [Ophthalmic Information Processing Apparatus] FIGS. 3 to 6 show block diagrams of configuration examples of the ophthalmic information processing apparatus 100 according to the embodiments. FIG. 3 represents a functional block diagram of the ophthalmic information processing apparatus 100. FIG. 4 represents a functional block diagram of the analysis unit 200 in FIG. 3. FIG. 5 represents a functional block diagram of the estimation model construction unit 210 in FIG. 3. FIG. 6 represents a functional block diagram of the estimation unit 220 in FIG. 3.

[0080] The ophthalmic information processing apparatus 100 includes a control unit 110, a data processing unit 130, and a communication unit 140. In some embodiments, the ophthalmic information processing apparatus 100 includes an image forming unit having the same function as the OCT data processing unit 13 of the ophthalmic apparatus 10.

[0081] The control unit 110 controls each part of the ophthalmic information processing apparatus 100. In particular, the control unit 110 controls the data processing unit 130 and the communication unit 140. The control unit 110 includes a main control unit 111 and a storage unit 112.

[0082] The control unit 110 controls each part of the ophthalmic system 1 based on an operation instruction signal corresponding to the operation content of the user with respect to the operation device 180.

[0083] Each of the control unit 110 and the data processing unit 130 includes a processor. The function of the data processing unit 130 is realized by a data processing processor. In some embodiments, the functions of the control unit 110 and the data processing unit 130 are realized by one processor.

[0084] The storage unit 112 stores various types of data. The data stored in the storage unit 112 includes data acquired by the ophthalmic device 10 (measurement data, imaging data, etc.), data processing results by the data processing unit 130, information regarding the subject and the eye to be examined, and the like. The storage unit 112 may store various computer programs and data for operating each part of the ophthalmic information processing device 100.

[0085] The communication unit 140 receives control from the control unit 110 and executes communication interface processing for transmitting or receiving information to / from the communication unit 14 of the ophthalmic information processing device 100.

[0086] The data processing unit 130 can generate an image in an arbitrary cross-sectional direction specified by performing various renderings on the three-dimensional OCT data from the ophthalmic device 10. In this embodiment, the data processing unit 130 generates a plurality of images with different cross-sectional directions. The plurality of images includes B-scan images, C-scan images, en-face images, projection images, shadowgrams, and the like. An image of an arbitrary cross-section such as a B-scan image or a C-scan image is formed by selecting pixels (pixels, voxels) on the specified cross-section from the three-dimensional OCT data (image data). The en-face image is formed by flattening a part of the three-dimensional OCT data (image data). The projection image is formed by projecting the three-dimensional OCT data in a predetermined direction (z direction, depth direction, A-scan direction). The shadowgram is formed by projecting a part of the three-dimensional OCT data (for example, partial data corresponding to a specific layer) in a predetermined direction.

[0087] In some embodiments, by the user operating the operating device 180, at least one of the cross-sectional position and the cross-sectional direction is specified. The user can specify at least one of the cross-sectional position and the cross-sectional direction using the operating device 180 while referring to a frontal image of the fundus generated from the three-dimensional OCT data or a fundus image acquired using a fundus camera (not shown).

[0088] In some embodiments, the data processing unit 130 identifies at least one of a cross-sectional position and a cross-sectional direction by analyzing OCT data. In this case, the data processing unit 130 can identify a feature region by known analysis processing and identify at least one of the cross-sectional position and the cross-sectional direction so as to pass through the identified feature region. Alternatively, the data processing unit 130 can identify a feature region by known analysis processing and identify at least one of the cross-sectional position and the cross-sectional direction so as to avoid the identified feature region.

[0089] The data processing unit 130 performs predetermined data processing on the formed image of the eye to be examined. Similar to the OCT data processing unit 13, the data processing unit 130 can perform various data processing (image processing) and analysis processing on the formed image. For example, the data processing unit 130 executes correction processing such as brightness correction and variance correction of the image. The data processing unit 130 can form volume data (voxel data) of the eye to be examined by executing known image processing such as interpolation processing for interpolating pixels between tomographic images. When displaying an image based on the volume data, the data processing unit 130 performs rendering processing on this volume data to form a pseudo three-dimensional image when viewed from a specific viewing direction.

[0090] Note that instead of the data processing unit 130, the OCT data processing unit 13 in the ophthalmic apparatus 10 may generate the above-described image group. In this case, the ophthalmic information processing apparatus 100 acquires the above-described image group from the ophthalmic apparatus 10.

[0091] In addition, the data processing unit 130 generates a disease estimation model for estimating whether the eye to be examined is a glaucoma eye from a plurality of images generated from three-dimensional OCT data. Further, the data processing unit 130 generates a pathological condition estimation model for estimating a classification result of a pathological condition from a plurality of images or feature amounts of each of the plurality of images.

[0092] The data processing unit 130 outputs estimation information for estimating whether the eye to be examined is a glaucoma eye or not from a plurality of images of the eye to be examined, and classification information for estimating the classification result of the pathological condition of glaucoma, using the above-described disease estimation model and pathological condition estimation model.

[0093] Such a data processing unit 130 includes an analysis unit 200, an estimation model construction unit 210, and an estimation unit 220.

[0094] The analysis unit 200 performs predetermined analysis processing on the fundus image data (or the fundus image data acquired by the ophthalmic device 10). The analysis processing includes, for example, a process of generating a tomographic image in a desired cross-sectional direction at a desired cross-sectional position. The analysis processing according to some embodiments includes a process of specifying a predetermined site such as the optic nerve head and the fovea centralis.

[0095] As shown in FIG. 4, the analysis unit 200 includes an image generation unit 201. The image generation unit 201 generates a plurality of images having different cross-sectional directions from the three-dimensional OCT data acquired by the ophthalmic device 10. Examples of the plurality of images include a B-scan image, a C-scan image, an en-face image, a projection image, and a shadowgram.

[0096] The image generation unit 201 can generate at least one image in which the cross-sectional position and the cross-sectional direction are specified by the user from the three-dimensional OCT data. Further, the image generation unit 201 can generate at least one image in which the cross-sectional position and the cross-sectional direction are specified so as to pass through a predetermined site specified in the analysis processing in the analysis unit 200. Furthermore, the image generation unit 201 can generate at least one image in which the cross-sectional position and the cross-sectional direction are specified so as to avoid a predetermined site specified in the analysis processing in the analysis unit 200.

[0097] In this embodiment, the image generation unit 201 generates five types of images with different cross-sectional directions from the three-dimensional OCT data of the eye to be examined. The five types of images include a horizontal B-scan image passing through the center (or its vicinity) of the optic nerve head, a vertical B-scan image passing through the center (or its vicinity) of the optic nerve head, a vertical B-scan image passing through the fovea (or its vicinity), a projection image, and an en-face image. The projection image is an integrated image of all layers in the depth direction. The en-face image is an integrated image of a predetermined depth (e.g., 50 micrometers) in the depth direction from the uppermost layer of the retina (e.g., the inner limiting membrane). That is, the en-face image is an image obtained by projecting a deep layer region from a layer region corresponding to the inner limiting membrane, for example.

[0098] As described above, the estimation model construction unit 210 constructs a disease estimation model (the above-described estimation model and classification model) for estimating whether the eye to be examined is a glaucoma eye and a pathological condition estimation model for estimating the classification result of the pathological condition.

[0099] As shown in FIG. 5, the estimation model construction unit 210 includes a disease estimation learning unit 211 and a pathological condition estimation learning unit 212. As shown in FIG. 6, the estimation unit 220 includes a disease estimation unit 221 and a pathological condition estimation unit 222.

[0100] FIG. 7 shows a schematic diagram for explaining the operation of the ophthalmic information processing apparatus 100 according to the embodiment. FIG. 7 schematically shows the relationship between the estimation model construction unit 210 and the estimation unit 220. In FIG. 7, the relationship between the disease estimation unit 221 and the pathological condition estimation unit 222 and the disease estimation learning unit 211 and the pathological condition estimation learning unit 212 is illustrated. In FIG. 7, the same parts as those in FIG. 5 or FIG. 6 are denoted by the same reference numerals, and the description thereof is omitted as appropriate.

[0101] The disease estimation learning unit 211 generates a plurality of individual estimation models for estimating whether the eye to be examined is a glaucoma eye or not for each of a plurality of images generated from the three-dimensional OCT data. The disease estimation learning unit 211 generates a plurality of individual estimation models by performing supervised machine learning for each of the plurality of images. In some embodiments, the disease estimation learning unit 211 generates a plurality of individual estimation models by transfer learning.

[0102] When the five types of images are generated from the three-dimensional OCT data as described above, the disease estimation learning unit 211 generates an individual estimation model for each type of image.

[0103] For example, for the horizontal B-scan image IMG1 passing through the center (or its vicinity) of the optic nerve head, the disease estimation learning unit 211 uses the B-scan images IMG1 of a plurality of other eyes to be examined excluding the eye to be examined as the training data TR1, and performs supervised machine learning with the label indicating the judgment result of whether it is glaucoma or not by a doctor or the like for each B-scan image as the teacher data SD1 to generate an individual estimation model for the B-scan image IMG1. The function of the individual estimation model for the B-scan image IMG1 is realized by the first estimator 301 shown in FIG. 7.

[0104] For example, for the vertical B-scan image IMG2 passing through the center (or its vicinity) of the optic nerve head, the disease estimation learning unit 211 uses the B-scan images IMG2 of a plurality of other eyes to be examined excluding the eye to be examined as the training data TR1, and performs supervised machine learning with the label indicating the judgment result of whether it is glaucoma or not by a doctor or the like for each B-scan image as the teacher data SD1 to generate an individual estimation model for the B-scan image IMG2. The function of the individual estimation model for the B-scan image IMG2 is realized by the second estimator 302 shown in FIG. 7.

[0105] For example, for the vertical B-scan image IMG3 passing through the fovea (or its vicinity), the disease estimation learning unit 211 uses the B-scan images IMG3 of a plurality of other subject eyes excluding the subject eye to be estimated as the training data TR1, and performs supervised machine learning with the label indicating the determination result of whether or not glaucoma by a doctor or the like for each B-scan image as the teacher data SD1 to generate an individual estimation model for the B-scan image IMG3. The function of the individual estimation model for the B-scan image IMG3 is realized by the third estimator 303 shown in FIG. 7.

[0106] For example, for the projection image IMG4, the disease estimation learning unit 211 uses the projection images IMG4 of a plurality of other subject eyes excluding the subject eye to be estimated as the training data TR1, and performs supervised machine learning with the label indicating the determination result of whether or not glaucoma by a doctor or the like for each projection image as the teacher data SD1 to generate an individual estimation model for the projection image IMG4. The function of the individual estimation model for the projection image IMG4 is realized by the fourth estimator 304 shown in FIG. 7.

[0107] For example, for the en-face image IMG5, the disease estimation learning unit 211 uses the en-face images IMG5 of a plurality of other subject eyes excluding the subject eye to be estimated as the training data TR1, and performs supervised machine learning with the label indicating the determination result of whether or not glaucoma by a doctor or the like for each en-face image as the teacher data SD1 to generate an individual estimation model for the en-face image IMG5. The function of the individual estimation model for the en-face image IMG5 is realized by the fifth estimator 305 shown in FIG. 7.

[0108] In addition, the disease estimation learning unit 211 generates a classification model for estimating whether or not the subject eye is a glaucoma eye from the plurality of estimation information output from the plurality of generated individual estimation models. The disease estimation learning unit 211 generates a classification model by performing unsupervised machine learning or supervised machine learning.

[0109] For example, the disease estimation learning unit 211 generates a classification model by performing unsupervised machine learning or supervised machine learning using a group of images IMG1 to IMG5 of a plurality of other eyes to be examined excluding the eye to be examined as training data TR1. When performing supervised machine learning, a label indicating whether the training eye to be examined is a glaucoma eye is used as teacher data SR1. The function of this classification model is realized by the classifier 306 shown in FIG. 7.

[0110] A disease estimation model is constituted by a plurality of individual estimation models and a classification model for the images IMG1 to IMG5. That is, the estimation unit 220 outputs estimation information for estimating whether the eye to be examined is a glaucoma eye from a plurality of images of the eye to be examined (a plurality of images generated from the OCT data of the eye to be examined) using the disease estimation model generated by the disease estimation learning unit 211. In some embodiments, the estimation information includes information indicating whether the eye to be examined is a glaucoma eye. In some embodiments, the estimation information includes information (confidence information) indicating the confidence level that the eye to be examined is a glaucoma eye (for example, the probability estimated to be glaucoma).

[0111] Each of the above-mentioned plurality of individual estimation models may have a similar configuration.

[0112] FIG. 8 shows a block diagram of a configuration example of the first estimator 301 according to the embodiment. Each of the second estimator 302 to the fifth estimator 305 may have a configuration similar to that of FIG. 8.

[0113] The function of the first estimator 301 is realized, for example, by a Convolutional Neural Network (CNN). That is, according to the instructions from the individual estimation model (trained model) stored in the computer's memory, for the pixel values of the image IMG1 input to the convolutional layer 311 of the feature extractor 310 which is the input layer, operations are performed based on the learned weighting coefficients and response functions in the convolutional neural network, and it operates to output a determination result from the classifier 320 which is the output layer. The first estimator 301 having such a configuration can extract local correlation patterns while gradually reducing the resolution of the image, and output a determination result based on the extracted correlation patterns.

[0114] The first estimator 301 includes a feature extractor 310 and a classifier 320. The feature extractor 310 repeatedly extracts features and downsamples (filters) for each predetermined image region of the input image IMG1 to extract the features of the determination image. The classifier 320 outputs estimation information (for example, confidence) for estimating whether the eye to be examined is a glaucoma eye based on the features extracted by the feature extractor 310.

[0115] The feature extractor 310 includes a plurality of units in which units including a Convolution Layer and a Pooling Layer are connected in multiple stages. In each unit, the input of the pooling layer is connected to the output of the convolutional layer. The pixel values of the corresponding pixels in the image IMG1 are input to the input of the convolutional layer in the first stage. The input of the convolutional layer in the subsequent stage is connected to the output of the pooling layer in the previous stage.

[0116] In FIG. 8, the feature extractor 310 includes two units connected in two stages. That is, the feature extractor 310 has a unit including a convolutional layer 311 and a pooling layer 312, and a unit including a convolutional layer 313 and a pooling layer 314 is connected to the subsequent stage. The output of the pooling layer 312 is connected to the input of the convolutional layer 313.

[0117] The classifier 320 includes fully connected layers 321 and 322, and the output of the fully connected layer 321 is connected to the input of the fully connected layer 322.

[0118] In the feature extractor 310 and the classifier 320, learned weighting coefficients are assigned between neurons of two connected layers. Each neuron performs an operation on the operation result considering the weighting coefficients from one or more input neurons using a response function, and outputs the obtained operation result to the neurons in the next stage.

[0119] The weighting coefficients are updated by performing known machine learning using, as training data, two or more images IMG1 of the eye to be examined (horizontal B-scan images passing through the center (or vicinity) of the optic disc) acquired in the past, and, as teacher data, the labels assigned by a doctor or the like for each image. The existing weighting coefficients are updated by machine learning using two or more images acquired in the past as training data. In some embodiments, the weighting coefficients are updated by transfer learning.

[0120] The first estimator 301 may have a known layer structure such as VGG16, VGG19, InceptionV3, ResNet18, ResNet50, etc. The classifier 320 may have a known configuration such as Random Forest or Support Vector Machine (SVM).

[0121] The classifier 306 in the disease estimation unit 221 is generated, for example, by performing known machine learning using two or more acquired fundus images IMG1 to IMG5 as training data and the labels assigned by a doctor or the like for each image as teacher data. The classifier 306 outputs information OUT indicating whether the fundus to be examined is a glaucoma eye or confidence information OUT indicating the confidence level that the fundus to be examined is a glaucoma eye based on a plurality of pieces of estimation information from the first estimator 301 to the fifth estimator 305. In some embodiments, the classifier 306 outputs information OUT indicating whether the fundus to be examined is a glaucoma eye or confidence information OUT indicating the confidence level that the fundus to be examined is a glaucoma eye based on a plurality of feature amounts extracted from predetermined layers of each of the first estimator 301 to the fifth estimator 305. The classifier 306 may have a known configuration such as a Random Forest or a support vector machine, similar to the classifier 320 shown in FIG. 8.

[0122] The pathological state estimation learning unit 212 generates a pathological state estimation model for estimating the classification result of glaucoma from a plurality of images generated by the image generation unit 201. The pathological state estimation learning unit 212 generates a pathological state estimation model by executing supervised machine learning. In some embodiments, when it is determined that the fundus to be examined is a glaucoma eye from the estimation information output from the classification model, the pathological state estimation learning unit 212 generates classification information for estimating the classification result of the pathological state of glaucoma.

[0123] For example, five types of images IMG1 to IMG5 generated by the image generation unit 201 are used as training data TR2, and supervised machine learning is executed using the labels indicating the determination results of whether or not it is glaucoma by a doctor or the like as teacher data SD2 to generate the pathological state classification estimation model 401.

[0124] At this time, the pathological condition estimation learning unit 212 generates a pathological condition classification estimation model 401 through machine learning using the feature amounts (weight coefficients, etc.) of the individual estimation models for the images IMG1 to IMG5 obtained by the disease estimation learning unit 211. In some embodiments, the pathological condition estimation learning unit 212 generates the pathological condition classification estimation model 401 through machine learning using the feature amounts (weight coefficients, etc.) of the individual estimation models and the classification model for the images IMG1 to IMG5.

[0125] The pathological condition estimation unit 222 outputs classification information OUTa to OUTd for classifying the pathological condition of glaucoma from five types of images IMG1 to IMG5 of the eye to be examined using the pathological condition estimation learning model generated by the pathological condition estimation learning unit 212. In some embodiments, the classification information includes information representing any one of the types to be classified among a plurality of types of the pathological condition of glaucoma. In some embodiments, the classification information includes information representing the confidence level (for example, the probability estimated to be each type) of each of the plurality of types of the pathological condition of glaucoma. For example, the classification information OUTa is confidence level information representing the confidence level that the glaucoma that develops in the eye to be examined is local ischemic type (FI). The classification information OUTb is confidence level information representing the confidence level that the glaucoma that develops in the eye to be examined is myopic type (MY). The classification information OUTc is confidence level information representing the confidence level that the glaucoma that develops in the eye to be examined is age-related sclerosis type (SS). The classification information OUTd is confidence level information representing the confidence level that the glaucoma that develops in the eye to be examined is overall enlarged type (GE).

[0126] In this embodiment, the pathological condition estimation unit 222 outputs classification information according to the Nicolella classification.

[0127] In the Nicolella classification, four types (focal ischemia type, myopic type, senile sclerotic type, and generalized enlargement type) are defined according to the shape of the optic nerve papilla. The focal ischemia type (FI) is a type in which a notch is present in a part of the rim and a local defect in the optic nerve fiber layer is observed. The focal ischemia type is more common in women and often accompanied by migraine and seizures. The myopic type (MY) is a type in which the optic nerve papilla is inclined and a crescent-shaped peripapillary atrophy (PPA) with a depression on the temporal side is observed. The myopic type is more common in the younger generation and often accompanied by myopia. The senile sclerotic type (SS) is a type in which the optic nerve papilla is circular, the depression is shallow, and a halo is observed around the papilla. The senile sclerotic type is more common in the elderly and often accompanied by cardiovascular disorders. The generalized enlargement type (GE) is a type that exhibits a large and deep circular depression. The generalized enlargement type is more common in the test eyes with high intraocular pressure.

[0128] When the test eye is a glaucoma eye, the pathological state estimation unit 222 can output classification information representing the type classified as the pathological state of glaucoma that has developed in the test eye among the plurality of types in the Nicolella classification. In addition, the pathological state estimation unit 222 can output classification information representing the confidence level of each of the plurality of types in the Nicolella classification.

[0129] The ophthalmic device 10, or the communication unit 140 that performs reception processing of OCT data from the ophthalmic device 10, is an example of the "acquisition unit" according to the embodiment. As the five types of images, a horizontal B-scan image passing through the center (or the vicinity thereof) of the optic disc, a vertical B-scan image passing through the center (or the vicinity thereof) of the optic disc, a vertical B-scan image passing through the fovea (or the vicinity thereof), a projection image, and an en-face image are examples of the "plurality of images" according to the embodiment. The individual estimation model for the B-scan image IMG1, the individual estimation model for the B-scan image IMG2, the individual estimation model for the B-scan image IMG3, the individual estimation model for the projection image IMG4, and the individual estimation model for the en-face image IMG5 are examples of the "plurality of learned models" according to the embodiment. The B-scan image is an example of the "tomographic image" according to the embodiment. The projection image or the en-face image is an example of the "front image" according to the embodiment. The disease estimation learning unit 211 is an example of the "second learning unit" according to the embodiment. The pathological state estimation learning unit 212 is an example of the "first learning unit" according to the embodiment. The horizontal direction is an example of the "first cross-sectional direction" according to the embodiment. The vertical direction is an example of the "second cross-sectional direction" according to the embodiment.

[0130] [Operation Example] An operation example of the ophthalmic information processing device 100 according to some embodiments will be described.

[0131] FIGS. 9 and 10 show flowcharts of operation examples of the ophthalmic information processing device 100 according to the embodiment. FIG. 9 represents a flowchart of an example of generation processing of a learned model for performing disease estimation and pathological state estimation. FIG. 10 represents a flowchart of an example of disease estimation processing and pathological state estimation processing using the learned model generated by the generation processing shown in FIG. 9. A computer program for realizing the processing shown in FIGS. 9 and 10 is stored in the storage unit 112. The control unit 110 (main control unit 111) can execute the processing shown in FIGS. 9 and 10 by operating according to this computer program.

[0132] In FIG. 9, assume that in the ophthalmic apparatus 10, OCT has been previously performed on the fundus of a plurality of eyes to be examined, and a plurality of three-dimensional OCT data have been acquired.

[0133] (S1: Create a training data group from three-dimensional OCT data) First, the main control unit 111 controls the communication unit 140 to acquire a plurality of three-dimensional OCT data for a plurality of eyes to be examined acquired in the ophthalmic apparatus 10.

[0134] Subsequently, the main control unit 111 controls the image generation unit 201 to generate the above five types of images as a training data group from each of the plurality of three-dimensional OCT data. That is, the image generation unit 201 generates a horizontal B-scan image passing through the center (or the vicinity thereof) of the optic nerve head, a vertical B-scan image passing through the center (or the vicinity thereof) of the optic nerve head, a vertical B-scan image passing through the fovea (or the vicinity thereof), a projection image, and an en-face image from the three-dimensional OCT data. A doctor or the like reads the generated images and attaches a label indicating whether the eye to be examined is a glaucoma eye or not. The main control unit 111 associates the read images with the labels and stores them in the storage unit 112.

[0135] (S2: Generate a disease estimation model) Next, the main control unit 111 controls the disease estimation learning unit 211 to generate an individual estimation model (disease estimation model) for the image for one of the above five types of images. The disease estimation learning unit 211 generates an individual estimation model by performing supervised machine learning as described above.

[0136] (S3: Next? Subsequently, the main control unit 111 determines whether to perform the generation process of the individual estimation model for the next image. For example, the main control unit 111 determines whether to perform the generation process of the individual estimation model for the next image by counting the previously determined number of image types or the number of image types generated in step S1.

[0137] When it is determined that the generation process of the individual estimation model is to be performed for the next image (step S3: Y), the operation of the ophthalmic information processing apparatus 100 proceeds to step S2. When it is not determined that the generation process of the individual estimation model is to be performed for the next image (step S3: N), the operation of the ophthalmic information processing apparatus 100 proceeds to step S4.

[0138] (S4: Generate a disease state estimation model) In step S3, when it is not determined that the generation process of the individual estimation model is to be performed for the next image (step S3: N), the main control unit 111 controls the disease state estimation learning unit 212 to generate a disease state classification estimation model 401 for estimating the classification result of the glaucoma disease state based on the above five types of images. The disease state estimation learning unit 212 generates the disease state classification estimation model 401 as described above.

[0139] Thus, the operation of the ophthalmic information processing apparatus 100 ends (end).

[0140] As shown in FIG. 10, the ophthalmic information processing apparatus 100 uses the learned model generated according to the flow shown in FIG. 9 to generate estimation information for estimating whether the eye to be examined is a glaucoma eye and classification information for estimating the classification result of the glaucoma disease state.

[0141] In FIG. 10, in the ophthalmic apparatus 10, it is assumed that OCT is executed on the fundus of the eye to be examined which is the estimation target in advance, and 3D OCT data is acquired.

[0142] (S11: Create an image group from 3D OCT data) First, the main control unit 111 controls the communication unit 140 to acquire the 3D OCT data of the eye to be examined which is the estimation target acquired in the ophthalmic apparatus 10.

[0143] Subsequently, the main control unit 111 controls the image generation unit 201 to generate the above five types of images as an image group from the three-dimensional OCT data. That is, the image generation unit 201 generates a horizontal B-scan image passing through the center (or its vicinity) of the optic nerve head, a vertical B-scan image passing through the center (or its vicinity) of the optic nerve head, a vertical B-scan image passing through the fovea (or its vicinity), a projection image, and an en-face image from the three-dimensional OCT data.

[0144] (S12: Disease estimation, pathological condition estimation) Next, the main control unit 111 controls the disease estimation unit 221 to output estimation information for estimating whether the eye to be examined is a glaucoma eye by using the disease estimation model (individual estimation model for each image) generated by repeating steps S2 to S3 in FIG. 9 for the five types of image groups generated in step S11.

[0145] In addition, the main control unit 111 controls the pathological condition estimation unit 222 to output classification information for estimating the classification result of the pathological condition of glaucoma by using the pathological condition estimation model generated in step S4 in FIG. 9 for the five types of image groups generated in step S11.

[0146] In some embodiments, the main control unit 111 operates the disease estimation unit 221 and the pathological condition estimation unit 222 in parallel to output the estimation information and the classification information simultaneously.

[0147] Thus, the operation of the ophthalmic information processing apparatus 100 ends (end).

[0148] Here, in order to evaluate the determination accuracy of glaucoma by the ophthalmic information processing apparatus 100 according to the embodiment, a comparison is made with a comparative example of the embodiment. The ophthalmic information processing apparatus according to the comparative example of the embodiment includes a five-input convolutional neural network that uses the same five types of images as the input data in the embodiment, and is configured to determine whether the eye to be examined is a glaucoma eye from the five types of images of the fundus of the eye to be examined by performing known supervised machine learning.

[0149] Figure 11 shows an example of the area under the receiver operating characteristic curve (AUROC) obtained by the ophthalmic information processing apparatus according to the comparative example of the embodiment.

[0150] In Figure 11, the results of evaluation by five-fold cross-validation using 143 cases of OCT data of normal eyes and 672 cases of OCT data of glaucoma eyes are shown as AUROC. In Figure 11, the AUC (Area Under Curve) is 0.950 ± 0.023.

[0151] Figure 12 shows an example of the AUROC obtained by the ophthalmic information processing apparatus 100 according to the embodiment.

[0152] In Figure 12, the results of evaluation by five-fold cross-validation using the same OCT data as in the comparative example (normal eyes: 143 cases, glaucoma eyes: 672 cases) are shown as AUROC. In Figure 12, the AUC is 0.983 ± 0.009.

[0153] As described above, according to the configuration according to the embodiment, it is possible to improve the determination accuracy compared to the case of simply determining whether it is a glaucoma eye using a convolutional neural network.

[0154] <Modification Example> The configuration according to the embodiment is not limited to the above configuration.

[0155] Some ophthalmic apparatuses according to the embodiments include at least one of the functions of the ophthalmic information processing apparatus 100, the operation apparatus 180, and the display apparatus 190 in addition to the functions of the ophthalmic apparatus 10.

[0156] Hereinafter, an ophthalmic apparatus according to a modification example of some embodiments will be described focusing on the differences from the ophthalmic apparatus according to the above embodiments.

[0157] FIG. 13 shows a block diagram of a configuration example of an ophthalmic apparatus 10a according to a modification of the embodiment. In FIG. 13, the same parts as those in FIG. 2 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0158] The configuration of the ophthalmic apparatus 10a according to this modification is different from the configuration of the ophthalmic apparatus 10 according to the above embodiment in that the ophthalmic apparatus 10a has the functions of the ophthalmic information processing apparatus 100, the operation apparatus 180, and the display apparatus 190. The ophthalmic apparatus 10a includes a control unit 11a, an OCT unit 12, an OCT data processing unit 13, an ophthalmic information processing unit 15a, an operation unit 16a, and a display unit 17a.

[0159] The control unit 11a controls each part of the ophthalmic apparatus 10a. In particular, the control unit 11a controls the OCT unit 12, the OCT data processing unit 13, the ophthalmic information processing unit 15a, the operation unit 16a, and the display unit 17a.

[0160] The ophthalmic information processing unit 15a has the same configuration as the ophthalmic information processing apparatus 100 and has the same functions as the ophthalmic information processing apparatus 100. The operation unit 16a has the same configuration as the operation apparatus 180 and has the same functions as the operation apparatus 180. The display unit 17a has the same configuration as the display apparatus 190 and has the same functions as the display apparatus 190.

[0161] According to this modification, with a compact configuration, it is possible to highly accurately obtain estimation information for estimating whether the eye to be examined is a glaucoma eye (diseased eye) and classification information for estimating the classification result of the pathological condition of glaucoma by machine learning using less training data. Thereby, it is possible to provide a new technique for early performing appropriate treatment for the disease of the eye to be examined.

[0162] <Effect> Hereinafter, an ophthalmic information processing apparatus, an ophthalmic apparatus, an ophthalmic information processing method, and a program according to some embodiments will be described.

[0163] An ophthalmic information processing apparatus (ophthalmic information processing apparatus 100, ophthalmic information processing unit 15a) according to some embodiments includes an acquisition unit (ophthalmic apparatus 10 or communication unit 140 that performs reception processing of OCT data from ophthalmic apparatus 10) and a disease estimation unit (221). The acquisition unit acquires a plurality of images of the eye to be examined having different cross-sectional directions from each other. The disease estimation unit outputs estimation information for estimating whether the eye to be examined is an eye with a disease from the plurality of images using a plurality of learned models obtained by machine learning for each type of the plurality of images.

[0164] According to such a configuration, for each of the images in a plurality of different cross-sectional directions, estimation information for estimating whether the eye to be examined is an eye with a disease is acquired using the learned model obtained by machine learning, so that a disease can be discovered with high accuracy and at an early stage. Thereby, appropriate treatment for the disease of the eye to be examined can be performed at an early stage.

[0165] In some embodiments, the disease estimation unit includes a plurality of estimators (first estimator 301 to fifth estimator 305) that output feature amounts or confidence information representing the confidence that the eye to be examined is an eye with a disease for each type of the plurality of images using each of the plurality of learned models, and a classifier (306) that outputs estimation information from the plurality of feature amounts or the plurality of confidence information output from the plurality of estimators using a classification model obtained by machine learning.

[0166] According to such a configuration, feature amounts or confidence information are output for each type of the plurality of images using the learned model obtained by machine learning, and estimation information is output from the plurality of feature amounts or the plurality of confidence information using the classification model obtained by machine learning, so that a disease can be discovered with high accuracy and at an early stage. Thereby, appropriate treatment for the disease of the eye to be examined can be performed at an early stage.

[0167] In some embodiments, the plurality of images include a tomographic image or a frontal image of the fundus of the eye to be examined, and the disease is a fundus disease.

[0168] According to such a configuration, fundus diseases can be detected with high precision and at an early stage. Thereby, appropriate treatment for the fundus diseases of the eye to be examined can be administered at an early stage.

[0169] In some embodiments, the plurality of images include B-scan images in a cross-sectional direction that intersect a line connecting the optic nerve head and the fovea.

[0170] According to such a configuration, since estimation information is obtained using a plurality of images including B-scan images in a cross-sectional direction that intersect a line connecting the optic nerve head and the fovea, diseases that affect the shape of the cross-section intersecting the line connecting the optic nerve head and the fovea can be detected with high precision and at an early stage.

[0171] In some embodiments, the plurality of images include a B-scan image in a first cross-sectional direction (horizontal direction) passing through the optic nerve head, a B-scan image in a second cross-sectional direction (vertical direction) passing through the optic nerve head and intersecting the first cross-sectional direction, and a B-scan image passing through the fovea.

[0172] According to such a configuration, diseases that affect the shape of the cross-section of the optic nerve head or the fovea can be detected with high precision and at an early stage.

[0173] In some embodiments, the frontal image includes at least one of an en-face image obtained by projecting a deep layer region from a layer region corresponding to the inner limiting membrane and a projection image.

[0174] According to such a configuration, diseases that affect the frontal shape of the eye to be examined can be detected with high precision and at an early stage.

[0175] In some embodiments, the fundus diseases include at least one of glaucoma, age-related macular degeneration, and diabetic retinopathy.

[0176] According to such a configuration, glaucoma and diseases can be detected with high precision and at an early stage. As a result, appropriate treatment for the disease of the eye to be examined can be administered at an early stage.

[0177] Some embodiments include a pathological state estimation unit (222) that outputs classification information for estimating classification results of the pathological states of diseases from a plurality of images, using a classification learned model (pathological state classification estimation model 401) obtained by supervised machine learning using feature amounts obtained by a plurality of learned models.

[0178] According to such a configuration, it becomes possible to estimate the classification results of the pathological states of diseases with high precision. As a result, it becomes possible to administer appropriate treatment for the disease of the eye to be examined at an early stage. In addition, compared with the case of using a learned model obtained by performing machine learning from natural images or the like, it becomes possible to acquire highly accurate classification information by machine learning using less training data.

[0179] In some embodiments, the disease includes glaucoma, and the classification information is information for estimating classification results of a plurality of types of optic disc shapes.

[0180] According to such a configuration, it becomes possible to estimate the classification results of the pathological states of glaucoma with high precision. As a result, it becomes possible to administer appropriate treatment for the glaucoma that has developed in the eye to be examined at an early stage.

[0181] Some embodiments include a first learning unit (pathological state estimation learning unit 212) that generates a classification learned model by supervised machine learning using feature amounts obtained by a plurality of learned models.

[0182] According to such a configuration, it becomes possible to provide an ophthalmic information processing apparatus capable of generating a classification learned model for estimating, with high precision, classification results of the pathological states of diseases that have developed in the eye to be examined, by machine learning using less training data compared with the case of using a learned model obtained by performing machine learning from natural images or the like.

[0183] Some embodiments include a second learning unit (disease estimation learning unit 211) that generates a plurality of learned models by supervised machine learning for each type of a plurality of images.

[0184] According to such a configuration, it becomes possible to provide an ophthalmic information processing apparatus capable of generating a learned model for accurately and early detecting a disease of an eye to be examined.

[0185] Some embodiments include an image generation unit (201) that generates at least one of a plurality of images based on three-dimensional OCT data of an eye to be examined.

[0186] According to such a configuration, it becomes possible to provide an ophthalmic information processing apparatus capable of accurately and early detecting a disease of an eye to be examined by acquiring three-dimensional OCT data of the eye to be examined.

[0187] An ophthalmic apparatus (10, 10a) according to some embodiments includes an OCT unit (12) that performs optical coherence tomography on an eye to be examined, an image generation unit (201) that generates at least one of a plurality of images based on three-dimensional data acquired by the OCT unit, and the ophthalmic information processing apparatus (100, ophthalmic information processing unit 15a) according to any one of the above items.

[0188] According to such a configuration, it becomes possible to provide an ophthalmic apparatus capable of performing optical coherence tomography on an eye to be examined and accurately and early detecting a disease of the eye to be examined from the obtained OCT data.

[0189] An ophthalmic apparatus (10, 10a) according to some embodiments includes an OCT unit (12) that performs optical coherence tomography on an eye to be examined and the ophthalmic information processing apparatus (100, ophthalmic information processing unit 15a) described above.

[0190] According to such a configuration, it becomes possible to provide an ophthalmic apparatus that performs optical coherence tomography on an eye to be examined and can accurately and early detect a disease of the eye to be examined from the obtained OCT data.

[0191] An ophthalmic information processing method according to some embodiments includes an acquisition step of acquiring a plurality of images of an eye to be examined in cross-sectional directions different from each other, and a disease estimation step of outputting estimation information for estimating whether the eye to be examined is an eye with a disease from the plurality of images using a plurality of learned models obtained by machine learning for each type of the plurality of images.

[0192] According to such a method, for each of the images in a plurality of different cross-sectional directions, estimation information for estimating whether the eye to be examined is an eye with a disease is obtained using a learned model obtained by machine learning, so that a disease can be detected accurately and early. Thereby, it becomes possible to perform appropriate treatment on the disease of the eye to be examined at an early stage.

[0193] In some embodiments, it includes a plurality of estimation steps of outputting feature amounts or confidence information representing the confidence that the eye to be examined is an eye with a disease for each type of the plurality of images using each of the plurality of learned models, and a classification step of outputting estimation information from the plurality of feature amounts or the plurality of confidence information output in the plurality of estimation steps using a classification model obtained by machine learning.

[0194] According to such a method, feature amounts or confidence information are output for each type of the plurality of images using a learned model obtained by machine learning, and estimation information is output from the plurality of feature amounts or the plurality of confidence information using a classification model obtained by machine learning, so that a disease can be detected accurately and early. Thereby, it becomes possible to perform appropriate treatment on the disease of the eye to be examined at an early stage.

[0195] In some embodiments, the plurality of images include tomographic images or frontal images of the fundus of the eye to be examined, and the disease is a fundus disease.

[0196] According to such a method, fundus diseases can be detected with high precision and at an early stage. Thereby, appropriate treatment for the fundus diseases of the eye to be examined can be administered at an early stage.

[0197] In some embodiments, the plurality of images include cross-sectional B-scan images in a cross-sectional direction that intersects a line connecting the optic nerve head and the fovea.

[0198] According to such a method, since estimation information is obtained using a plurality of images including cross-sectional B-scan images in a cross-sectional direction that intersects a line connecting the optic nerve head and the fovea, diseases that affect the shape of the cross-section that intersects the line connecting the optic nerve head and the fovea can be detected with high precision and at an early stage.

[0199] In some embodiments, the plurality of images include a B-scan image in a first cross-sectional direction passing through the optic nerve head, a B-scan image in a second cross-sectional direction passing through the optic nerve head and intersecting the first cross-sectional direction, and a B-scan image passing through the fovea.

[0200] According to such a method, diseases that affect the shape of the cross-section of the optic nerve head or the fovea can be detected with high precision and at an early stage.

[0201] In some embodiments, the frontal image includes at least one of an en-face image obtained by projecting a deep layer region from a layer region corresponding to the inner limiting membrane, and a projection image.

[0202] According to such a method, diseases that affect the frontal shape of the eye to be examined can be detected with high precision and at an early stage.

[0203] In some embodiments, the fundus diseases include at least one of glaucoma, age-related macular degeneration, and diabetic retinopathy.

[0204] According to such a method, glaucoma and diseases can be detected with high precision and at an early stage. As a result, appropriate treatment for the disease of the eye to be examined can be administered at an early stage.

[0205] Some embodiments include a pathological state estimation step of outputting classification information for estimating the classification result of the pathological state of a disease from a plurality of images, using a learned classification model (pathological state classification estimation model 401) obtained by supervised machine learning using feature amounts obtained by a plurality of learned models.

[0206] According to such a method, the classification result of the pathological state of a disease can be estimated with high precision. As a result, it becomes possible to administer appropriate treatment for the disease of the eye to be examined at an early stage. In addition, compared with the case of using a learned model obtained by performing machine learning from a natural image or the like, it becomes possible to acquire high-precision classification information by machine learning using less training data.

[0207] In some embodiments, the disease includes glaucoma, and the classification information is information for estimating the classification results of a plurality of types of optic disc shapes.

[0208] According to such a method, the classification result of the pathological state of glaucoma can be estimated with high precision. As a result, it becomes possible to administer appropriate treatment for the glaucoma that has developed in the eye to be examined at an early stage.

[0209] Some embodiments include a first learning step of generating a learned classification model by supervised machine learning using feature amounts obtained by a plurality of learned models.

[0210] According to such a method, it becomes possible to provide an ophthalmic information processing method capable of generating a learned classification model for estimating the classification result of the pathological state of a disease that has developed in an eye to be examined with high precision by machine learning using less training data, compared with the case of using a learned model obtained by performing machine learning from a natural image or the like.

[0211] Some embodiments include a second learning step of generating a plurality of trained models by supervised machine learning for each type of a plurality of images.

[0212] According to such a method, it becomes possible to provide an ophthalmic information processing method capable of generating a trained model for accurately and early detecting a disease of an eye to be examined.

[0213] Some embodiments include an image generation step of generating at least one of a plurality of images based on three-dimensional OCT data of an eye to be examined.

[0214] According to such a method, by acquiring three-dimensional OCT data of an eye to be examined, it becomes possible to provide an ophthalmic information processing method capable of accurately and early detecting a disease of the eye to be examined.

[0215] A program according to some embodiments causes a computer to execute each step of the ophthalmic information processing method described in any of the above.

[0216] According to such a program, for each of a plurality of images in different cross-sectional directions, estimation information for estimating whether the eye to be examined is an eye with a disease is acquired using a trained model obtained by machine learning. Therefore, a disease can be detected accurately and early. As a result, appropriate treatment for the disease of the eye to be examined can be performed early.

[0217] A program for realizing the ophthalmic information processing method according to some embodiments can be stored in any non-transitory computer-readable recording medium. The recording medium may be an electronic medium using magnetism, light, magneto-optics, semiconductors, etc. Typically, the recording medium is a magnetic tape, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, a solid state drive, etc.

[0218] It is also possible to transmit and receive a computer program through a network such as the Internet or a LAN.

[0219] The aspects described above are merely examples for carrying out the present invention. Those who attempt to implement the present invention can make any modifications (omissions, substitutions, additions, etc.) within the scope of the gist of the present invention.

Explanation of Reference Numerals

[0220] 1 Ophthalmic system 10, 10a Ophthalmic device 11, 11a, 110 Control unit 12 OCT unit 12A Interference optical system 12B Scan optical system 14, 140 Communication unit 15a Ophthalmic information processing unit 16a Operation unit 17a Display unit 100 Ophthalmic information processing apparatus 111 Main control unit 112 Storage unit 130 Data processing unit 180 Operating device 190 Display device 200 Analysis unit 201 Image generation unit 210 Estimation model construction unit 211 Disease estimation learning unit 212 Pathological state estimation learning unit 220 Estimation unit 221 Disease estimation unit 222 Pathological state estimation unit

Claims

1. An acquisition unit that acquires a plurality of images of an eye to be examined with different cross-sectional directions; For each type of the plurality of images, using each of a plurality of learned models obtained by machine learning, a plurality of estimators that output, for each type of image, feature quantities or confidence information representing the confidence that the eye to be examined has age-related macular degeneration; A classifier that uses a classification model obtained by machine learning to output estimation information for estimating whether or not the eye to be examined has age-related macular degeneration from the plurality of feature quantities or the plurality of confidence information output from the plurality of estimators; An ophthalmic information processing apparatus comprising the above.

2. A pathological state estimation unit that outputs classification information for estimating a classification result based on the form of the retinal pigment epithelium layer from the plurality of images, using a classification learned model obtained by supervised machine learning using the feature quantities obtained by the plurality of learned models; The ophthalmic information processing apparatus according to claim 1, characterized by the above.

3. An acquisition unit that acquires a plurality of images of an eye to be examined with different cross-sectional directions; For each type of the plurality of images, using each of a plurality of learned models obtained by machine learning, a plurality of estimators that output, for each type of image, feature quantities or confidence information representing the confidence that the eye to be examined has diabetic retinopathy; A classifier that uses a classification model obtained by machine learning to output estimation information for estimating whether or not the eye to be examined has diabetic retinopathy from the plurality of feature quantities or the plurality of confidence information output from the plurality of estimators; An ophthalmic information processing apparatus comprising the above.

4. A pathological state estimation unit that outputs classification information for estimating a classification result based on at least one of the forms or distributions of capillary hemangioma, hemorrhage, leukoplakia, neovascularization, and hemorrhage regions from the plurality of images, using a classification learned model obtained by supervised machine learning using the feature quantities obtained by the plurality of learned models; The ophthalmic information processing apparatus according to claim 3, characterized by the above.

5. Including a first learning unit that generates the classification learned model by supervised machine learning using the feature quantities obtained by the plurality of learned models; The ophthalmic information processing apparatus according to claim 2 or claim 4, characterized by the above.

6. The plurality of images include a tomographic image or a frontal image of the fundus of the eye to be examined; The ophthalmic information processing apparatus according to any one of claims 1 to 5, characterized by the above.

7. Including a second learning unit that generates the plurality of learned models by supervised machine learning for each type of the plurality of images The ophthalmic information processing apparatus according to any one of claims 1 to 6, characterized in that.

8. Including an image generation unit that generates at least one of the plurality of images based on the three-dimensional OCT data of the eye to be examined The ophthalmic information processing apparatus according to any one of claims 1 to 7, characterized in that.

9. An OCT unit that performs optical coherence tomography on the eye to be examined; An image generation unit that generates at least one of the plurality of images based on the three-dimensional data acquired by the OCT unit; An ophthalmic information processing apparatus according to any one of claims 1 to 7; An ophthalmic apparatus including.

10. An OCT unit that performs optical coherence tomography on the eye to be examined; The ophthalmic information processing apparatus according to claim 8; An ophthalmic apparatus including.

11. An acquisition step in which a processor acquires a plurality of images of an eye to be examined having different cross-sectional directions from each other; A plurality of estimation steps in which the processor outputs, for each type of the plurality of images, feature amounts or confidence information representing the confidence that the eye to be examined is an eye with age-related macular degeneration, using each of the plurality of learned models obtained by machine learning for each type of the plurality of images; A classification step in which the processor outputs estimation information for estimating whether or not the eye to be examined is an eye with age-related macular degeneration from the plurality of feature amounts or the plurality of confidence information output in the plurality of estimation steps, using a classification model obtained by machine learning; An operating method of an ophthalmic information processing apparatus including.

12. Including a pathological state estimation step in which the processor outputs classification information for estimating a classification result based on the morphology of the retinal pigment epithelium layer from the plurality of images, using a classification learned model obtained by supervised machine learning using the feature amounts obtained by the plurality of learned models The operating method of the ophthalmic information apparatus according to claim 11, characterized in that.

13. An acquisition step in which a processor acquires a plurality of images of an eye to be examined having different cross-sectional directions from each other; A plurality of estimation steps in which the processor outputs, for each type of the plurality of images, feature amounts or confidence information representing the confidence that the eye to be examined is an eye with diabetic retinopathy, using each of the plurality of learned models obtained by machine learning for each type of the plurality of images; A classification step in which the processor uses a classification model obtained by machine learning to output estimation information for estimating whether the eye to be examined is an eye with diabetic retinopathy from a plurality of feature amounts or a plurality of confidence information outputs in the plurality of estimation steps. An operating method of an ophthalmic information processing apparatus, including the above.

14. The processor includes a pathological state estimation step of outputting classification information for estimating a classification result based on at least one form or distribution of capillary hemangioma, hemorrhage, leukoplakia, neovascularization, and hemorrhage region from the plurality of images by using a classification learned model obtained by supervised machine learning using the feature amounts obtained by the plurality of learned models. The operating method of the ophthalmic information processing apparatus according to claim 13, characterized in that it is as described above.

15. The processor includes a first learning step of generating the classification learned model by supervised machine learning using the feature amounts obtained by the plurality of learned models. The operating method of the ophthalmic information processing apparatus according to claim 12 or claim 14, characterized in that it is as described above.

16. The processor includes a second learning step of generating the plurality of learned models by supervised machine learning for each type of the plurality of images. The operating method of the ophthalmic information processing apparatus according to claim 15, characterized in that it is as described above.

17. The plurality of images include a tomographic image or a frontal image of the fundus of the eye to be examined. The operating method of the ophthalmic information processing apparatus according to any one of claims 11 to 16, characterized in that it is as described above.

18. The processor includes an image generation step of generating at least one of the plurality of images based on the three-dimensional OCT data of the eye to be examined. The operating method of the ophthalmic information processing apparatus according to any one of claims 11 to 17, characterized in that it is as described above.

19. A program characterized in that each step of the operating method of the ophthalmic information processing apparatus according to any one of claims 11 to 18 is executed on a computer.

Images