Diagnostic support device, diagnostic support system, and program

Abstract

The purpose of the present invention is to produce information for assisting an ophthalmic diagnosis and to make the information available in a region having a shortage of devices, physicians and the like.　A close-up imaging device 20, which can generate slit light SL from source light emitted from a light source 92 built in a mobile communication terminal device 10 can also concentrate light including reflected light RL of the slit light SL in a subject's eye E onto an imaging camera lens 91 in the mobile communication terminal device 10, is installed in the mobile communication terminal device 10, and a moving image of a tissue of interest in the subject's eye E is taken with a camera module built in the mobile communication terminal device 10. A diagnosable frame image included in the moving image thus taken is extracted in a diagnosis assisting server device 30, and then at least one item selected from the health condition of the subject's eye E and a distance, an angle and an area in a part of the tissue in the subject's eye E is estimated and, simultaneously, diagnosis assisting information is also produced on the basis of the extracted diagnosable frame image, and the diagnosis assisting information is delivered to a corresponding mobile communication terminal device 10.

Description

Technical Field

[0001] The present invention relates to a diagnostic support device that supports medical diagnosis by processing medical images, and particularly to processing diagnostic images in the ophthalmic field of humans and animals, and based on the diagnostic images, supporting the diagnosis of diseases in the eyes of patients (or diseased animals) to be examined, or performing at least one measurement of distance, angle, and area in a part of the tissue of the eye to be examined.

Background Art

[0002] In recent years, with the remarkable development of machine learning methods such as deep learning and the rapid performance improvement of AI (artificial intelligence), the application of AI (artificial intelligence) to various fields has been expected, and application methods of AI to the medical field have also been proposed (for example, Patent Document 1).

[0003] In the ophthalmic system described in this Patent Document 1, a plurality of ophthalmic imaging devices installed in ophthalmic facilities such as hospitals and an information processing system (server device) are connected directly or via a network, and the information processing system performs machine learning based on the ophthalmic diagnostic images taken by the ophthalmic imaging devices to acquire knowledge for image diagnosis. And in this ophthalmic system, when an image for examining the eye to be examined is transmitted from the ophthalmic imaging device after the acquisition of knowledge by the information processing system, based on the image and the acquired knowledge, suspected disease names, the presence or absence of specific diseases, severity, the necessity of examination, the type of examination, the necessity of surgery, the type of surgery, etc. are inferred, and diagnostic support information including the inference results is automatically generated and distributed so as to be available for use in the ophthalmic examination device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the system described in Patent Document 1 above, it is configured to capture diagnostic images using an ophthalmic imaging device installed in a facility such as a hospital. This type of ophthalmic imaging device is expensive and difficult to handle, and it is difficult to capture images that can be used for diagnosis without experts such as doctors or vision therapists who have specialized knowledge (hereinafter referred to as "doctors, etc."). For this reason, in regions where there is a shortage of equipment and doctors, etc., such as developing countries and remote areas, it is difficult to obtain images that can be used for diagnosis, and it becomes difficult to generate diagnostic support information. In addition, information parameter values including at least one of the distance, angle, and area in a part of the tissue of the eye to be examined cannot be measured without doctors, etc. using advanced ophthalmic devices, and in some regions, they cannot be measured at all.

[0006] The present invention has been made in view of the circumstances described above, and an object thereof is to provide a diagnostic support device or the like that can surely generate and make available diagnostic support information regarding the eye to be examined even in regions where there is a shortage of equipment and doctors, etc.

Means for Solving the Problems

[0007] (1) To solve the above-described problems, the diagnostic support apparatus of the present invention is mounted on a mobile communication terminal device including a light source and a camera lens for imaging, and irradiates an observation target tissue of an eye to be examined with any one of slit light, blue light, and linearly polarized light generated based on the light source light emitted from the light source as observation light, or passes the light source light as it is and irradiates the observation target tissue with the light source light as the observation light; an observation light irradiation member, and (b) a convex lens member that condenses light including reflected light of the observation light in the observation target tissue onto the camera lens for imaging. An image obtained by photographing the eye to be examined with the mobile communication terminal device in a state where a close-up photographing device including at least the above is mounted, and (i) the health state of the eye to be examined and (ii) the distance, angle, and area in a part of the tissue of the eye to be examined. An acquisition unit that acquires a photographed image including at least one first frame image that can be used to estimate at least one of the information parameter values including one or more of them, a first storage unit that stores in advance first knowledge for extracting the first frame image from the acquired photographed image, and a first extraction unit that extracts the first frame image included in the acquired photographed image based on the first knowledge, and based on the photographed image, (i) the health state of the eye to be examined and (ii) a second storage unit that stores second knowledge for estimating at least one of the information parameter values, and based on the extracted first frame image and the second knowledge, (i) the health state of the eye to be examined reflected in the acquired photographed image and (ii) an estimation unit that estimates at least one of the information parameter values, a generation unit that generates diagnostic support information including at least one of the estimated health state of the eye to be examined and the information parameter value, and a distribution unit that distributes the generated diagnostic support information to an external device.

[0008] With this configuration, the light source light emitted from the light source of the mobile communication terminal device is irradiated as observation light on the observation target tissue as it is by the close-up photographing device, or is converted into any one of slit light, blue light, and linearly polarized light and irradiated on the observation target tissue as observation light. Then, while the user irradiates the observation light on the observation target tissue of the eye to be examined (for example, anterior eye tissues including eyelids, ocular surface, cornea, conjunctiva, lens, anterior chamber, iris, and fundus tissue), when the camera module mounted on the mobile communication terminal device photographs the eye to be examined, the light including the reflected light of the observation light in the eye to be examined is condensed by the close-up photographing device onto the photographing camera lens of the mobile communication terminal device, and a moving image in which the observation target tissue of the eye to be examined is reflected can be photographed.

[0009] Since the camera module of the mobile communication terminal device can be operated by the same method as that of an existing smartphone or the like, by using the mobile communication terminal device equipped with the close-up photographing device (hereinafter, also referred to as "smart eye camera"), even a user other than a doctor or the like who is not used to handling an existing ophthalmic photographing device (for example, in addition to paramedical staff, laypersons such as the family and friends of the patient or the patient himself / herself) can easily photograph a moving image including a first frame image (hereinafter, also referred to as "diagnosable frame image") that can be used for the diagnosis of the eye to be examined.

[0010] Here, in order to use a captured image for ophthalmic diagnosis, it is necessary to capture an image that satisfies all of the following three conditions: (Condition 1) the observation light is irradiated onto the tissue to be observed in the eye to be examined; (Condition 2) at least a part of the tissue to be observed is captured together with the reflected light of the irradiated observation light in the tissue to be observed; and (Condition 3) the tissue to be observed is in focus. On the other hand, when a user captures the eye to be examined using a smart eye camera, for example, during the period of capturing a moving image of the eye to be examined for several seconds to several tens of seconds, if a state satisfying the above three conditions can be created even for the time of one frame (for example, 0.033 seconds when capturing at a frame rate of 30 fps), it is possible to capture a moving image including at least one diagnosable frame image that can be used for ophthalmic diagnosis. In particular, when using a smart eye camera, the autofocus mechanism mounted on the camera module of the mobile communication terminal device can automatically focus on the subject to be captured. Therefore, if the user captures the eye to be examined while creating a state in which the observation light is irradiated onto the tissue to be observed in the eye to be examined and the reflected light is captured, it is possible to very easily capture a moving image including a diagnosable frame image that satisfies all of the above three conditions. Note that the eye to be examined does not necessarily have to be a human eye, and may be an eye of an animal (such as a dog, a cat, or a mouse). In addition, when estimating the information parameter value regarding the eye to be examined, a predetermined additional condition is required, which will be described in detail later.

[0011] On the other hand, when the user is not used to taking diagnostic images, there is a high possibility that a moving image including a large number of miscellaneous frame images that do not satisfy all of the above three conditions will be taken, and as it is, it cannot be used to estimate the health state of the eye to be examined. Therefore, the diagnostic support device of the present invention automatically extracts a diagnosable frame image that satisfies the above three conditions from among a plurality of frame images constituting the moving image based on the first knowledge, and based on the extracted diagnosable frame image, at least one of the health state and the information parameter value of the eye to be examined is estimated, and diagnostic support information including at least one of the estimated health state and the information parameter value is generated. With this configuration, the diagnostic support device of the present invention can accurately estimate at least one of the health state and the information parameter value of the eye to be examined based on the captured image even when a user who is not used to taking pictures, such as a layperson, captures the eye to be examined. Therefore, according to the diagnostic support device of the present invention, diagnostic support information can be surely generated and used without being affected by the user, region, etc. Also, since the proximity imaging device to be mounted on the mobile communication terminal device only needs to have at least an observation light irradiation member and a convex lens, it can be manufactured at a very low cost, and diagnostic support information including at least one of the health state and the information parameter value of the eye to be examined can be surely generated and used even in a region where an expensive ophthalmic imaging device cannot be prepared.

[0012] (2) Further, in the above configuration, the first extraction means may adopt a configuration in which, based on the frame image included in the captured image and the first knowledge, the probability that each of the frame images included in the captured image corresponds to the first frame image is calculated, and based on the calculated probability, the first frame image is extracted.

[0013] When extracting images using machine learning, although it is impossible to eliminate the possibility that unintended images may be extracted during extraction, by extracting diagnosable frame images based on the probability of corresponding to diagnosable frame images, it is possible to improve the accuracy and recall rate during the extraction of diagnosable frame images. Note that the criteria for extracting diagnosable frame images based on the calculated probability are arbitrary. For example, it may be possible to extract the top predetermined number of frames with a high corresponding probability (e.g., the best frame with the highest probability, the top 5 frames with a high probability, or the top several % of frame images with a high probability, etc.) as diagnosable frame images.

[0014] (3) Also, in the configuration according to claim 2, the first extraction means extracts a plurality of the frame images with a high calculated probability as the first frame images, and the estimation means, based on the plurality of the extracted first frame images and the second knowledge, for each of the first frame images, estimates at least one of (i) the health state of the eye to be examined and (ii) the information parameter value, and while weighting the estimated health state and information parameter value by the calculated probability, adopts a configuration for estimating at least one of the most likely health state and information parameter value of the eye to be examined.

[0015] With this configuration, the diagnostic support device of the present invention can estimate the most likely health state and information parameter value of the eye to be examined while weighting each frame image included in the moving image by the probability of corresponding to the diagnosable frame image. Generally, the estimation result of the health state, etc. based on the diagnosable frame image with a high probability of corresponding to the diagnosable frame image can obtain a higher correct answer rate than the estimation result of the health state, etc. based on the diagnosable frame image with a lower probability. Therefore, by weighting the estimation result based on each diagnosable frame image by the probability of corresponding to the diagnosable frame image, while increasing the weight of the estimation result with a high correct answer rate, it is possible to estimate the most likely health state and information parameter value of the eye to be examined and improve the reliability of the diagnostic support information.

[0016] Generally, when estimating the health condition of an eye to be examined based on each diagnosable frame image by a classifier, the classifier can estimate the correct answer with a predetermined accuracy and recall rate. However, when estimating the health condition based on a single diagnosable frame image, it becomes difficult to achieve a correct answer rate exceeding the accuracy or recall rate of a single classifier. On the other hand, when the above configuration is adopted, multiple estimation results based on multiple diagnosable frame images can be obtained, and while weighting these estimation results, the final estimation result can be determined. As a result, the diagnostic support device of the present invention can estimate the most likely health condition and information parameter values of the eye to be examined with a higher accuracy and recall rate than estimation by a single classifier, and can improve the reliability of the diagnostic support information. Note that the specific method of the above weighting process is arbitrary. For example, ensemble machine learning may be performed on multiple estimation results obtained based on each diagnosable frame image.

[0017] (4) Further, in the configuration according to any one of claims 1 to 3, the acquisition means acquires the captured image taken while irradiating at least one of the eyelid and the anterior ocular tissue of the eye to be examined with slit light generated based on the light source light as the observation light, and in the first storage means, as the first knowledge, (i) the state of diseases occurring in the eyelid and the anterior ocular tissue of the eye to be examined, and (ii) knowledge for extracting the first frame image that can be used for estimating at least one of the information parameter values related to the eyelid and the anterior ocular tissue is stored in advance. In the second storage means, as the second knowledge, (i) the state of diseases occurring in the eyelid and the anterior ocular tissue of the eye to be examined, and (ii) knowledge for estimating at least one of the information parameter values related to the eyelid and the anterior ocular tissue is stored. The estimation means may adopt a configuration for estimating (i) the state of diseases in at least one of the eyelid and the anterior ocular tissue of the eye to be examined and (ii) at least one of the information parameter values related to the eyelid and the anterior ocular tissue based on the extracted first frame image and the second knowledge.

[0018] With this configuration, the diagnostic support device of the present invention can estimate the state of a disease that has developed in at least one of the eyelids and anterior eye segment of the eye to be examined, as well as information parameter values related to these tissues, and generate and utilize diagnostic support information including the estimation result. When slit light is used as the observation light, in addition to observing the state of diseases in each of the tissues of the eyelid, ocular surface, cornea, conjunctiva, anterior chamber, and lens as the anterior eye segment tissues of the eye to be examined, it is also possible to observe diseases that have developed in some fundus tissues, or estimate the information parameter values of the fundus tissues.

[0019] (5) Further, in the configuration according to any one of claims 1 to 3, in a state where an injury occurring in at least one of the cornea and conjunctiva of the eye to be examined is imaged with a contrast agent, the acquisition means irradiates at least one of the cornea and conjunctiva of the eye to be examined with blue light generated based on the light source light as the observation light and acquires the captured image, and in the first storage means, knowledge for extracting the first frame image that can be used for diagnosing the state of an injury occurring in at least one of the cornea and conjunctiva of the eye to be examined is stored in advance as the first knowledge, and in the second storage means, knowledge for estimating the state of a disease in at least one of the cornea and conjunctiva from the state of an injury occurring in at least one of the cornea and conjunctiva of the eye to be examined is stored as the second knowledge, and the estimation means may adopt a configuration in which the state of a disease in at least one of the cornea and conjunctiva of the eye to be examined is estimated based on the extracted first frame image and the second knowledge.

[0020] With this configuration, the diagnostic support device of the present invention can estimate the state of a disease in at least one of the cornea and conjunctiva of the eye to be examined from the state of an injury occurring in at least one of the cornea and conjunctiva, and generate and utilize diagnostic support information including the estimation result.

[0021] (6) Further, in the configuration according to any one of claims 1 to 3, the acquisition means acquires the captured image obtained by irradiating the fundus tissue of the eye to be examined with linearly polarized light generated based on the light source light as the observation light while capturing an image, and in the first storage means, as the first knowledge, knowledge for extracting the first frame image that can be used for estimating at least one of (i) the state of the disease developing in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue is stored in advance. In the second storage means, as the second knowledge, knowledge for estimating at least one of (i) the state of the disease in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue is stored. The estimation means may be configured to estimate at least one of (i) the state of the disease in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue based on the extracted first frame image and the second knowledge.

[0022] With this configuration, the diagnostic support device of the present invention can estimate at least one of the state of the disease in the fundus tissue of the eye to be examined and the information parameter value in the fundus tissue, and generate and use diagnostic support information including the estimation result.

[0023] (7) Further, in the configuration according to any one of claims 1 to 3, the acquisition means acquires the captured image obtained by photographing while irradiating the observation target tissue of the subject eye with the light source light as the observation light as it is, and in the first storage means, as the first knowledge, (i) the state of a disease occurring in at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye and (ii) knowledge for extracting the first frame image that can be used for estimating at least one of the information parameter values related to at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye is stored in advance. In the second storage means, as the second knowledge, (i) the state of a disease in at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye and (ii) knowledge for estimating at least one of the information parameter values related to at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye is stored. The estimation means may be configured to estimate (i) the state of a disease in at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye and (ii) at least one of the information parameter values related to at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye based on the extracted first frame image and the second knowledge.

[0024] With this configuration, the diagnostic support device of the present invention can irradiate the observation target tissue with the light source light as the observation light as it is, estimate the state of a disease in at least one of the eyelid, ocular surface, cornea, and conjunctiva of the subject eye, and at least one of the information parameter values related to these tissues, and generate and use diagnostic support information including the estimation result.

[0025] (8) Further, in the configuration according to any one of claims 1 to 7, based on the first frame image extracted by the first extraction means, at least one of machine learning and data mining is executed to obtain the second knowledge, and a learning means for storing the obtained second knowledge in the second storage means is further provided, and the estimation means may adopt a configuration for estimating at least one of the disease state and the information parameter value in the eye to be examined based on the second knowledge stored in the second storage means by the learning means and the extracted first frame image.

[0026] With this configuration, the diagnostic support device of the present invention can obtain the second knowledge while executing at least one of machine learning and data mining based on the diagnostic frame image extracted by the first extraction means, so that the labor for providing the second knowledge to the diagnostic support device can be reduced. Note that the specific method for obtaining the second knowledge is arbitrary. For example, the diagnostic frame image extracted by the first extraction means is presented to a plurality of doctors serving as annotators, the diagnostic results of the doctors based on the diagnostic frame image are obtained, and the diagnostic frame image tagged with the diagnostic results of the doctors is used as teaching data and input into a convolutional neural network (also referred to as a convolutional neural network) for supervised learning to obtain the second knowledge. Also, various methods such as unsupervised learning, semi-supervised learning, transductive learning, and multi-task learning may be used.

[0027] (9) Further, in the configuration according to claim 8, the learning means acquires diagnostic result information indicating the diagnostic results of doctors based on the first frame image extracted by the first extraction means, and while using the diagnostic result information and the corresponding first frame image as teaching data, executes at least one of machine learning and data mining to obtain the second knowledge and stores it in the second storage means.

[0028] With this configuration, only the frame images extracted as diagnosable frame images from among the plurality of frame images included in the captured image can be presented to the doctor serving as the annotator. As a result, the diagnostic support apparatus of the present invention can create teaching data while reducing the work load of the doctor when acquiring the second knowledge, and can acquire useful second knowledge.

[0029] (10) Further, in the configuration according to claim 9, in the first storage means, as the first knowledge, a plurality of knowledges for extracting the first frame images that can be used for diagnosing the disease for each disease that can occur in each tissue constituting the eye to be examined are stored, and the first extraction means extracts, for each disease, the first frame images that can be used for estimating the state of the disease based on the first knowledge, and the learning means acquires the diagnostic result information regarding the corresponding disease based on the first frame images corresponding to the extracted diseases, and while using the diagnostic result information and the corresponding first frame images as teaching data, at least one of machine learning and data mining is executed to acquire the second knowledge necessary for diagnosing the disease for each disease, and the acquired second knowledge is stored in the second storage means in association with the corresponding disease. Such a configuration may be adopted.

[0030] With this configuration, the diagnostic support apparatus of the present invention can individually extract diagnosable frame images corresponding to each of a plurality of diseases that can develop in the eye to be examined from one captured image. Then, teacher data is created based on the diagnosis results of a doctor based on the extracted diagnosable frame images, and at least one of machine learning and data mining is executed based on the teacher data, so that second knowledge corresponding to each disease is acquired for each disease, and the acquired second knowledge can be stored in the second storage means in association with the corresponding disease. As a result, the diagnostic support apparatus of the present invention can acquire second knowledge for diagnosing a plurality of diseases from one captured image, and can acquire second knowledge corresponding to a plurality of diseases at once based on a small number of sample images. Further, with this configuration, the diagnostic support apparatus of the present invention can extract diagnosable frame images for annotation from the captured image in advance for each disease and present them to the doctor, so that the work burden on the doctor when performing annotation can be reduced.

[0031] (11) Further, in the configuration according to any one of claims 1 to 10, in the first storage means, as the first knowledge, a plurality of knowledges for extracting the first frame images that can be used for estimating the state of the disease for each disease that can develop in each tissue constituting the eye to be examined are stored, and in the second storage means, as the second knowledge, a plurality of knowledges for estimating the state of the disease for each disease are stored. The first extraction means extracts the first frame images that can be used for diagnosing the disease for each disease based on the first knowledge, and the estimation means estimates the state of each disease in the eye to be examined based on the first frame image extracted for each disease and the second knowledge of the corresponding disease. The generation means may adopt a configuration in which information including the estimated state of each disease is generated as the diagnostic support information.

[0032] With this configuration, the diagnostic support apparatus of the present invention can extract, for each disease, a diagnosable frame image corresponding to each of a plurality of diseases that can develop in the eye to be examined from one captured image. Then, the diagnostic support apparatus of the present invention can estimate, for each disease, the state of a plurality of diseases in the eye to be examined based on the extracted diagnosable frame image, and generate diagnostic support information including the estimated state for each disease.

[0033] As a result, even when a plurality of diseases have developed in the eye to be examined, the diagnostic support apparatus of the present invention can estimate all the disease states developed in the eye to be examined at once based on one captured image, and generate and use diagnostic support information including the estimation result, thereby significantly improving the convenience for the user.

[0034] (12) Further, in the configuration according to any one of claims 1 to 11, labeling means for labeling the corresponding tissue name for the in-focus tissue of the eye to be examined in each frame image included in the captured image, and, in each of the labeled frame images, second extraction means for extracting, as a second frame image, the frame image having the largest area of the pixel region in which the in-focus tissue is reflected, and the generation means generates the diagnostic support information including the extracted second frame image. A configuration may be adopted.

[0035] With this configuration, the diagnostic support device of the present invention can label the in-focus tissues in the frame images included in the captured image, making it possible to identify the tissues captured in each frame image. Further, it can extract the frame image in which the area of the pixel region where the in-focus tissues in the captured image are captured is the largest, and generate and utilize diagnostic support information including the extracted frame image. Usually, the frame image in which the diseased tissue is captured the largest is often the image that can most intuitively convey the disease state. Therefore, with the above configuration, it is possible to generate and utilize diagnostic support information including the best-shot frame image that can most intuitively convey the disease state to the patient as a GUI (Graphical User Interface), dramatically improving the patient's UX (User Experience).

[0036] (13) Further, in the configuration according to any one of claims 1 to 12, the diagnostic support device may further include three-dimensional image construction means for stacking each frame image included in the acquired captured moving image according to the focal length to construct a three-dimensional image of the eye to be examined. The second storage means stores, as the second knowledge, knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the three-dimensional image. The estimation means stores, as the second knowledge, knowledge for estimating the health state of the eye to be examined based on the three-dimensional image. The estimation means may adopt a configuration for estimating at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the generated three-dimensional image and the second knowledge.

[0037] Since the moving image captured by the mobile communication terminal device includes a plurality of frame images with different focal lengths, by stacking the frame images in the order corresponding to the focal length of each frame image, a three-dimensional image of the eye to be examined can be constructed, and the three-dimensional image can be used for estimating the health state, or a three-dimensional image can be presented to the user. Also, since the three-dimensional image contains more information than the planar image, the above configuration can improve the estimation accuracy of the health state and information parameter values of the eye to be examined, and improve the reliability of the diagnostic support information.

Effect of the Invention

[0038] According to the present invention, it is possible to surely generate and use diagnostic support information for the eye to be examined even in an area lacking equipment, doctors, etc.

Brief Description of the Drawings

[0039]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

[0040] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments are embodiments when the diagnostic support device, diagnostic support system, and program according to the present invention are applied to a system for realizing a function of estimating the health state of an eye to be examined based on a moving image of the eye to be examined taken and generating diagnostic support information including the estimation result and providing it for use by a user. However, the embodiments described below do not unduly limit the content of the present invention described in the claims, and not all of the configurations described in the present embodiment are essential constituent elements of the present invention.

[0041] [A] First Embodiment [A1] Configuration and Outline of Diagnostic Support System First, with reference to FIG. 1, the configuration and outline of the diagnostic support system 1 in the first embodiment of the present invention will be described. Note that FIG. 1 is a system configuration diagram showing an example of the configuration of the diagnostic support system 1 of this embodiment. Further, in FIG. 1, in order to prevent the drawing from becoming complicated, only some users are displayed, and only some mobile communication terminal devices 10 (that is, smart eye cameras) with the proximity shooting device 20 for configuring the diagnostic support system 1 attached thereto, and only some annotation terminal devices 40 are shown. That is, there are more users in the diagnostic support system 1 than shown, and there are many mobile communication terminal devices 10 and annotation terminal devices 40 with the proximity shooting device 20 attached thereto.

[0042] As shown in FIG. 1, the diagnostic support system 1 of the present embodiment includes (a) a plurality of mobile communication terminal devices 10-1 to n (hereinafter referred to as "mobile communication terminal devices 10") that are carried by each user and function as smart eye cameras with proximity shooting devices 20 attached thereto, respectively, (b) a diagnostic support server device 30 that is communicatively connected to each mobile communication terminal device 10 via a network N and executes processing based on moving image data transmitted (uploaded) from each mobile communication terminal device 10, and (c) a plurality of annotation terminal devices 40-1 to n (hereinafter referred to as "annotation terminal devices 40") used by an annotator such as an ophthalmologist. In the diagnostic support server device 30, (1) the health state (presence or absence of a disease and its severity, etc.) of the eye to be examined E is estimated based on the moving image data uploaded from the mobile communication terminal device 10, (2) diagnostic support information is generated based on the estimation result, and (3) the generated diagnostic support information is distributed to the corresponding mobile communication terminal device 10 to support the diagnosis of the eye to be examined E. Note that the format and content of the diagnostic support information are arbitrary. For example, information such as a suspected disease name, disease severity, treatment method, and necessity of surgery may be described in a format such as XML (Extensible Markup Language), and these information may be displayed on the mobile communication terminal device 10. Also, the health state of the eye to be examined E estimated by the diagnostic support server device 30 is arbitrary. For example, the presence or absence of a specific disease and the states of various diseases that may develop in various tissues of the eye to be examined E (e.g., eyelids, ocular surface, cornea, conjunctiva, iris, anterior chamber, lens, fundus tissue, etc.) can be estimated. However, in the present embodiment, for the sake of concretizing the explanation, the explanation will be made on the assumption that the state of nuclear sclerosis (cataract) in the eye to be examined E (i.e., the presence or absence of cataract onset and its severity) is estimated, and the method of estimating the states of other diseases will be described in detail in the section on modified examples and the second embodiment. Note that the diagnostic support system 1 of the present embodiment may be configured to estimate only the presence or absence of cataract onset without estimating the severity of cataract in the eye to be examined E.

[0043] Here, for an image for cataract diagnosis, it is necessary for a doctor or the like with specialized knowledge to take the image using an expensive ophthalmic slit lamp microscope installed in an ophthalmology department. Therefore, in regions where there is a shortage of equipment, doctors, etc., such as in developing countries, it is impossible to take a diagnostic image, and it is difficult for a conventional ophthalmic system to estimate the health condition of the eye E to be examined and generate diagnostic support information itself.

[0044] Therefore, in the diagnostic support system 1 of the present embodiment, based on the light source light emitted from a light source 92 (described later) built in the mobile communication terminal device 10, a slit light forming member 61 that generates slit light SL as observation light, and a convex lens member 93 that condenses light including the reflected light RL of the slit light SL in the lens tissue of the eye E to be examined onto a photographing camera lens 91 (described later) of the mobile communication terminal device 10 are provided. At least a proximity photographing device 20 having such a configuration is attached to the mobile communication terminal device 10, and a method of photographing a moving image of the eye E to be examined using a camera module built in the mobile communication terminal device 10 is adopted. Then, based on the moving image photographed by the mobile communication terminal device 10, the diagnostic support server device 30 estimates the state of cataract in the eye E to be examined and generates diagnostic support information based on the estimation result. The configuration and principle of the proximity photographing device 20 will be described in detail later. In addition, in many cases, the mobile communication terminal device 10 is provided with a plurality of camera modules (for example, two such as an in-camera module provided on the display unit side and an out-camera module provided on the back side). However, usually, the out-camera module tends to have a higher resolution and is equipped with a functionally superior optical system. Therefore, in the present embodiment, a configuration is adopted in which the out-camera module is used as the proximity photographing device 20 to improve the resolution during the moving image photographing of the eye E to be examined and improve the accuracy during the state estimation of cataract. That is, when referring to the light source 92 and the photographing camera lens 91 in the present embodiment, the description will be made assuming that they are provided on the out-camera module side of the mobile communication terminal device 10. However, the present invention is not limited to using the out-camera module, and a configuration using the in-camera module is also possible.

[0045] The camera module mounted on the mobile communication terminal device 10 can be easily operated even by a user who does not have specialized knowledge. Further, by attaching the close-up photographing device 20 to the mobile communication terminal device 10, while generating slit light SL based on the light source light emitted from the light source 92 of the mobile communication terminal device 10, the slit light SL is used as observation light to irradiate the lens tissue of the eye E to be examined, and the light including the reflected light RL of the slit light SL in the lens tissue is condensed onto the photographing camera lens 91 (described later) of the camera module, and a moving image including one or more diagnosable frame images can be easily photographed.

[0046] On the other hand, when a user other than a doctor or the like photographs an image of the eye E to be examined, since the user is not used to photographing the eye E to be examined, for example, (a) a frame image in which the observation target tissue (that is, the lens tissue) is out of focus, (b) a frame image in which the lens tissue is not captured, (c) an image in which the eye E to be examined itself is not captured at all, etc., there is a high possibility that a moving image including a lot of frame images that do not satisfy the conditions as diagnosable frame images for cataract diagnosis (that is, frame images that cannot be used for diagnosis) is photographed. Further, even when a doctor or the like photographs an image of the eye E to be examined, there is a possibility that a moving image including frame images that cannot be used for cataract diagnosis is photographed for several seconds after the start of moving image photographing. For this reason, it is difficult to estimate the state of cataract using the moving image photographed using the smart eye camera as it is. Therefore, in the present embodiment, the diagnostic support server device 30 extracts diagnosable frame images that can be used for diagnosing the state of cataract in the eye E to be examined from all the frame images constituting the moving image data uploaded from the mobile communication terminal device 10, and adopts a method of estimating the state of cataract in the eye E to be examined based on the diagnosable frame images.

[0047] Here, in order to diagnose the severity of cataract, it is necessary to use an image that satisfies the following three conditions: (Condition 1) the crystalline lens tissue of the eye E to be examined is irradiated with slit light SL as observation light; (Condition 2) at least a part of the crystalline lens tissue is captured together with the reflected light RL of the irradiated slit light SL in the crystalline lens tissue; and (Condition 3) the crystalline lens tissue is in focus. For this reason, the diagnostic support server device 30 is configured to extract, as diagnostic possible frame images, frame images that satisfy these three conditions from all the frame images constituting the moving image captured by the mobile communication terminal device 10. Then, the diagnostic support server device 30 estimates the state of cataract in the eye E to be examined based on the extracted diagnostic possible frame images, and generates diagnostic support information. Note that the diagnostic support server device 30 of the present embodiment constitutes, for example, the "diagnostic support device" of the present invention.

[0048] With this configuration, the diagnostic support system 1 of the present embodiment can capture a moving image including diagnostic possible frame images for cataract diagnosis of the eye E to be examined even when the user does not have specialized medical knowledge, and appropriately estimate the state of cataract based on the captured moving image. As a result, the diagnostic support system 1 of the present embodiment can surely generate and use diagnostic support information for the eye E to be examined even in areas lacking medical staff such as doctors. In addition, since the proximity photographing device 20 attached to the mobile communication terminal device 10 only needs to have at least a slit light forming member 61 and a convex lens member 93, it can be manufactured at a very low cost. As a result, the diagnostic support system 1 of the present embodiment can capture a moving image including diagnostic possible frame images at low cost without using an expensive ophthalmic slit lamp microscope, and generate diagnostic support information.

[0049] [A2] Schematic Configuration of Diagnostic Support System 1 The mobile communication terminal device 10 is a portable communication terminal device carried by a user, such as a smartphone, a tablet-type information communication terminal device, a mobile phone, etc., and has a display unit (not shown) composed of a display element such as a liquid crystal panel or an organic EL (Electro Luminescence) panel, a speaker, an operation unit composed of a touch panel, a numeric keypad, etc. provided on the display unit. Further, the mobile communication terminal device 10 includes a camera module including a camera lens 91 for photographing, and a light source 92 composed of a light emitting element such as a white LED (Light Emitting Diode).

[0050] In addition to the camera lens 91 for photographing, the camera module has an optical system (not shown) such as a diaphragm mechanism for narrowing the light transmitted through the camera lens 91 for photographing, a shutter mechanism, an autofocus mechanism, and a shake correction mechanism. Further, the camera module has an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (complementary metal oxide Semiconductor) image sensor that receives the light transmitted through the camera lens 91 for photographing and outputs a signal corresponding to the received light, and supplies the signal output by the image sensor to an image generation unit (not shown) composed of a CPU (Central Processing Unit) to generate moving image data based on the light transmitted through the camera lens 91 for photographing. Note that the method of generating moving image data in the image generation unit is arbitrary. For example, bitmap data may be generated for each frame image, and the generated bitmap data may be arranged in time series to generate moving image data, or MPEG (Moving Picture Expert It may also be possible to generate moving image data in a format compressed by an inter-frame predictive coding method such as Group)2, H.264, or H.265. This camera module and image generation unit cooperate with the display unit and the operation unit to adjust the optical system based on a user input operation on the operation unit. Then, based on the light received by the image sensor, a frame image in which the subject (in this embodiment, the eye to be examined E) is captured is generated, and the image being captured is displayed on the display unit based on the generated frame image data. Note that the camera module equipped with an autofocus mechanism mounted on the mobile communication terminal device 10 is the same as the camera module used in conventional smartphones and the like.

[0051] Furthermore, the mobile communication terminal device 10 is installed (loaded) with an application program (hereinafter referred to as the "diagnosis app") for acquiring and displaying diagnosis support information from the diagnosis support server device 30 based on the captured moving image of the eye to be examined E. Then, by executing this diagnosis app, the mobile communication terminal device 10 uploads the moving image data generated by the camera module and the image generation unit to the diagnosis support server device 30 to acquire diagnosis support information from the diagnosis support server device 30. Then, based on the diagnosis support thus obtained, the mobile communication terminal device 10 displays information regarding the presence or absence of cataracts and their severity (specifically, indicators such as the NS grade expressing the severity of cataracts as a numerical value from 1 to 5), treatment methods, the necessity of surgery, etc. for the eye to be examined E on the display unit. Note that when generating the diagnosis support information in XML format, an existing browser application can also be used instead of the diagnosis app.

[0052] The device 20 for close-up photography is an optical device that is detachably attached to the mobile communication terminal device 10. It includes a slit light forming member 61 that generates slit light SL based on at least the light irradiated from the light source 92 of the mobile communication terminal device 10, and a convex lens member 93 that condenses the light including the reflected light RL of the slit light SL in the eye E to be examined onto the camera lens 91 for photography. Then, the device 20 for close-up photography generates the slit light SL as observation light based on the light source light (white diffused light) emitted from the light source 92, and condenses the light including the reflected light RL of the slit light SL in the eye E to be examined onto the camera lens 91 for photography. The mobile communication terminal device 10 captures a moving image of the eye E to be examined including a diagnostic frame image based on the light condensed onto the camera lens 91 for photography by the function of the device 20 for close-up photography. As a result, the mobile communication terminal device 10 (i.e., the smart eye camera) with the device 20 for close-up photography attached realizes the same function as an existing ophthalmic slit lamp microscope as a whole.

[0053] The diagnostic support server device 30 is a computer system provided on the network N. Based on the knowledge for extracting diagnostic frame images (hereinafter referred to as "extraction knowledge") obtained by machine learning or data mining, it extracts the diagnostic frames included in the moving image data uploaded from the mobile communication terminal device 10. Note that the extraction knowledge in this embodiment corresponds to, for example, the "first knowledge" of the present invention. Also, the specific method for obtaining the extraction knowledge is arbitrary. However, unlike the diagnosis of cataract, the determination of whether each frame image included in the moving image data corresponds to a diagnostic frame image can be determined only by whether the above conditions 1 to 3 are satisfied. Therefore, it is possible to determine whether it corresponds to a diagnostic frame image even without being an ophthalmology specialist. Thus, for example, an arbitrary person uses the annotation terminal device 40 to extract the diagnostic frame images for cataract from the moving image data uploaded from the mobile communication terminal device 10, and while using the extracted frame images as teacher data, the extraction knowledge may be obtained by executing at least one of machine learning and data mining in the diagnostic support server device 30.

[0054] In addition, the diagnostic support server device 30 of the present embodiment has diagnostic knowledge for estimating the state of cataract in the subject eye E based on the diagnosable frames extracted from the moving image data based on the extraction knowledge. Then, the diagnostic support server device 30 estimates the presence or absence of cataract, an index indicating the severity of cataract such as the NS grade, a treatment method, the necessity of surgery, etc. in the subject eye E based on the diagnosable frame image extracted from the moving image data and the diagnostic knowledge, generates diagnostic support information including the estimation result, and distributes it to the mobile communication terminal device 10 that is the upload source of the moving image data. Note that the diagnostic knowledge of the present embodiment constitutes, for example, the "second knowledge" of the present invention. Also, the method of acquiring the diagnostic knowledge by the diagnostic support server device 30 is arbitrary. For example, a computer for managing an electronic medical record installed in an ophthalmic facility (not shown) is connected to the diagnostic support server device 30 directly or via the network N, and the diagnostic results of patients who have visited the ophthalmic facility and ophthalmic diagnostic images taken at the ophthalmic facility are acquired from the electronic medical record of the ophthalmic facility. While using the acquired data as teacher data, it may be configured to execute at least one of machine learning and data mining to acquire diagnostic knowledge. However, when using this method, it is necessary to install a special function in the computer for managing the electronic medical record of the ophthalmic facility. Also, in this case, it becomes necessary to communicatively connect the computer for managing the electronic medical record of the ophthalmic facility and the diagnostic support server device 30 and manage the data acquired from each management computer by the diagnostic support server device 30, making it difficult to reduce the initial cost at the time of system construction. Furthermore, in this case, regarding the use of electronic medical record information, it is necessary to obtain the consent of the patient in advance on the side of the ophthalmic facility and strictly manage personal information, also making it difficult to reduce the running cost during system operation.

[0055] Therefore, in the diagnostic support system 1 of the present embodiment, diagnostic knowledge is acquired by generally the following method. (Step 1) First, each time moving image data captured by the mobile communication terminal device 10 is uploaded from the mobile communication terminal device 10, the diagnostic support server device 30 extracts a diagnosable frame image from the moving image data, transmits the diagnosable frame image to the annotation terminal device 40, and causes the annotation terminal device 40 to display the diagnosable frame image. (Step 2) Three doctors serving as annotators perform diagnoses regarding the presence or absence of cataracts, their severity (index such as NS grade), treatment methods, necessity of surgery, etc. based on the diagnosable frame images displayed on the annotation terminal device 40, and input the diagnostic results into the annotation terminal device 40. (Step 3) The annotation terminal device 40 generates diagnostic result information indicating the diagnostic results based on the input from the doctor, creates teacher data corresponding to each diagnosable frame image by tagging the corresponding diagnosable frame image with the diagnostic result information, and transmits it to the diagnostic support server device 30. (Step 4) The diagnostic support server device 30 accumulates the teacher data (that is, the tagged diagnosable frame images) acquired from the annotation terminal device 40 each time. Then, when the number of accumulated teacher data reaches α or more, which is the number of samples required for acquiring diagnostic knowledge (hereinafter simply referred to as "α"), the diagnostic support server device 30 executes at least one of machine learning and data mining using the accumulated teacher data to acquire diagnostic knowledge.

[0056] By adopting this method, a doctor acting as an annotator only needs to make a diagnosis regarding cataract based on the diagnosable frame images extracted by the diagnostic support server device 30, without having to find the diagnosable frame images from within the moving image data. Thus, the workload of the doctor during the acquisition of diagnostic knowledge can be reduced. Also, since the moving image data uploaded from the mobile communication terminal device 10 does not need to include information capable of identifying the patient, according to this method, the cost for personal information protection can be reduced. Furthermore, since only at least one annotation terminal device 40 is required, the system can be constructed at low cost. Note that the diagnostic knowledge may include, in addition to the knowledge obtained from the diagnostic results of the doctor based on the diagnosable frame images extracted from the moving image data, for example, knowledge described in medical books, medical dictionaries, other documents, or medical knowledge disclosed on the Internet. In this case, regarding the treatment method and the necessity of surgery, a configuration may be adopted in which it is estimated from the knowledge in medical books or the like according to the estimated NS grade of the cataract or the like.

[0057] When diagnostic support information is distributed from the diagnostic support server device 30, in the mobile communication terminal device 10, information such as the presence or absence of cataract in the eye to be examined E, indicators such as the NS grade, treatment method, and the necessity of surgery is displayed on the display unit based on the diagnostic support information, and becomes in a state where the user can view it.

[0058] The annotation terminal device 40 is, for example, a computer such as a PC (Personal Computer) or a tablet-type information communication terminal device, and has an operation unit composed of a display unit (not shown), a touch panel provided on the display unit, a mouse, a keyboard, etc., and is communicatively connected to the diagnostic support server device 30 directly or via the network N.

[0059] This annotation terminal device 40 displays the diagnosable frame images extracted by the diagnostic support server device 30. Further, the annotation terminal device 40 displays a GUI for a doctor serving as an annotator to input a diagnosis result based on the diagnosable frame image, and generates diagnosis result information corresponding to the input diagnosis result when the doctor inputs the diagnosis result. Then, the annotation terminal device 40 creates teacher data by tagging the corresponding diagnosable frame image with the diagnosis result information and transmits it to the diagnostic support server device 30. Note that the number of annotation terminal devices 40 is arbitrary, and it may be provided for the number of annotators (doctors at the time of acquiring diagnostic knowledge, arbitrary persons at the time of acquiring extraction knowledge), or one annotation terminal device 40 may be reused by a plurality of annotators.

[0060] [A3] Configuration of the close-up photographing device 20 Next, the configuration of the close-up photographing device 20 of the present embodiment will be described with reference to FIGS. 2 to 7. Note that FIGS. 2 to 7 are diagrams showing the configuration of the close-up photographing device 20A of the present embodiment. As shown in FIGS. 2 to 7, the close-up photographing device 20A of the present embodiment has a configuration that can be detachably attached to the mobile communication terminal device 10 and includes a housing 80 having a rectangular parallelepiped shape.

[0061] This housing 80 has a hollow interior and is composed of an outer wall portion 81 and a front plate 90 as shown in FIGS. 2 and 4. The outer wall portion 81 has a front wall 82, a rear wall 83, a left side wall 84, a right side wall 85, an upper wall 86, and a lower wall 87. An opening (not shown) into which the mobile communication terminal device 10 is inserted is formed in the lower wall 87. The opening is constituted by a through hole formed in the left - right direction from the side of the left side wall 84 to the side of the right side wall 85. The upper end portion of the mobile communication terminal device 10 provided with the out - camera module is inserted into the opening. The opening width in the front - rear direction is slightly wider than the thickness of the mobile communication terminal device 10, and the opening width in the left - right direction is slightly wider than the width of the mobile communication terminal device 10. Here, the front direction is defined as the direction from the rear wall 83 toward the front wall 82, and the opposite direction is the rear direction. Also, the left direction is defined as the direction from the right side wall 85 toward the left side wall 84, and the opposite direction is the right direction. Further, the upper direction is defined as the direction from the lower wall 87 toward the upper wall 86, and the opposite direction is the lower direction.

[0062] The front wall 82 has a peripheral portion where the upper edge, the lower edge, and the right edge are convex in a frame shape in the front direction. A front plate 90 having a width wider than the width of the peripheral portion is mounted on the peripheral portion. The front plate 90 is composed of an open left edge portion 90a, a right edge portion 90b, an upper edge portion 90c, and a lower edge portion 90d and is a frame - shaped body with an open central portion. The width of the front plate 90 is wider than the peripheral portion of the front wall 82. Therefore, the front plate 90 is provided so as to protrude inward of the peripheral portion. The protruding portion functions as upper and lower rails, and a plate - shaped color filter member 97 and a plate - shaped convex lens member 93 are slidably fitted into the upper and lower rails. The gap between the front wall 82 and the protruding portion of the front plate 90 is slightly larger than the thickness of the plate - shaped color filter member 97 and the plate - shaped convex lens member 93 and is formed with slidable dimensions. In the example of FIG. 2, the opening is the left edge portion 90a, but it may also be the right edge portion 90b.

[0063] The front wall 82 is provided with two holes. One hole 89 is provided at a position corresponding to the imaging camera lens 91 of the mobile communication terminal device 10, and the other hole 88 is provided at a position corresponding to the light source 92 of the mobile communication terminal device 10. Through these two holes 88 and 89, the light source light emitted from the light source 92 of the mobile communication terminal device 10 can be emitted forward, and the return light (for example, the light including the reflected light RL of the slit light SL) can be received by the imaging camera lens 91 of the mobile communication terminal device 10 to capture an image of the anterior segment or fundus of the eye to be examined E.

[0064] (Color filter member) The color filter member 97 is detachably provided above the light source 92. The color filter member 97 is a plate-like member, and as illustrated in FIGS. 3(A) to 3(C) described below, the color filter member 97 is slid to be attached to and detached from the light source 92. The color filter member 97 is preferably a blue filter that converts the white light emitted by the light source 92 of the mobile communication terminal device 10 into blue light. For example, it is preferably a blue filter that converts white light into blue light with a wavelength of 488 nm. As the blue filter, one obtained by coloring an acrylic resin can be adopted.

[0065] The hole 98 provided in the color filter member 97 is a hole for hanging a finger when sliding the color filter member 97. As long as it is for hanging a finger and sliding, it does not necessarily have to be a hole and can be a protrusion.

[0066] By sliding the color filter member 97 in the left - right direction, the light source 92 can be covered or uncovered. That is, by sliding the color filter member 97 and the convex lens member 93, the color filter member 97 can be attached to or detached from above the light source 92, or the convex lens member 93 can be attached to or detached from above the imaging camera lens 91. In this embodiment, by removing the color filter member 97 from the light source 92, the white diffused light as the light source light emitted from the light source 92 passes through the hole 88 as it is and is irradiated onto the eye - to - be - examined E as the observation light, and the light including the reflected light RL is received by the image sensor through the imaging camera lens 91, so that the eyelid, ocular surface, cornea, and conjunctiva of the eye - to - be - examined E can be observed and photographed. Also, by covering the light source 92 with the color filter member 97, the injuries generated on the cornea and conjunctiva can be observed and photographed. For example, by instilling a fluorescein solution as a contrast agent into the eye and adopting a blue - free filter for biological staining examination as the color filter member 97, the light source light is changed to blue light, irradiated onto the eye - to - be - examined E as the observation light, the injuries generated on the cornea and conjunctiva are changed to green, and the green light is condensed by the convex lens member 93 onto the imaging camera lens 91 and received by the image sensor, so that the injuries generated on the cornea and conjunctiva of the eye - to - be - examined E can be observed and photographed. Regarding the method of estimating the disease state in the cornea and conjunctiva of the eye - to - be - examined E using blue light and the estimation of the anterior eye disease state using white diffused light, they will be described in detail in the section of the modification example.

[0067] (Convex lens member) The convex lens member 93 is detachably provided on the imaging camera lens 91. This convex lens member 93 is a plate-shaped member, and as illustrated in FIGS. 3(A) to 3(C), the convex lens member 93 is slid to be attached to and detached from the imaging camera lens 91. This convex lens member 93 includes a convex lens 96 that condenses light onto the imaging camera lens 91 of the mobile communication terminal device 10. The convex lens 96 is arbitrarily selected in consideration of the focal length. The convex lens 96 is mounted in a hole 94 provided in the convex lens member 93. By the convex lens 96, the observation target tissue of the eye E to be examined is focused, image blurring is corrected, and the observation target tissue in the eye E to be examined can be clearly observed and photographed.

[0068] A hole 95 provided in the convex lens member 93 is a hole for hanging a finger when sliding the convex lens member 93, and also acts to improve the directivity of the light emitted from the light source 92 by sliding the convex lens member 93 and disposing the hole 95 above the light source 92 (see FIG. 3(C)).

[0069] FIG. 3 shows a form in which the color filter member 97 and the convex lens member 93 are slid. FIG. 3(A) shows a form in which the convex lens 96 of the convex lens member 93 is mounted on the imaging camera lens 91, and the color filter member 97 is slid to the right and removed from above the light source 92. FIG. 3(B) shows a form in which the convex lens 96 of the convex lens member 93 is mounted on the imaging camera lens 91, and the color filter member 97 is slid to the left and mounted on the light source 92. FIG. 3(C) shows a form in which the color filter member 97 is slid to the right and removed from above the light source 92, and the convex lens 96 of the convex lens member 93 is removed from the imaging camera lens 91. In FIG. 3(C), the hole 95 of the convex lens member 93 is disposed above the light source 92, and the directivity of the white diffused light as the light source light emitted from the light source 92 is adjusted when passing through the hole 88 and the hole 95, and is irradiated forward as the observation light.

[0070] (Slit light forming member) As shown in FIGS. 5 to 7, the slit light forming member 61 is a member that converts white diffused light, which is light source light emitted from the light source 92 of the mobile communication terminal device 10, into slit light SL using a cylindrical lens 62. The detaching means of the slit light forming member 61 is not particularly limited, but it is preferably detachably provided on the vertical rails for sliding the color filter member 97 and the convex lens member 93. For example, the slit light forming member 61, the color filter member 97, the holes 88 and 95 of the present embodiment constitute the "observation light irradiation member" of the present invention.

[0071] With such a slit light forming member 61, the slit light SL formed by the cylindrical lens 62 is irradiated onto the eye to be examined E, and the light including the reflected light RL is condensed by the convex lens member 93 onto the imaging camera lens 91 of the mobile communication terminal device 10, so that the anterior segment of the eye to be examined E can be observed and photographed in detail, and the fundus tissue can also be observed and photographed. As a result, the mobile communication terminal device 10 (i.e., the smart eye camera) in a state where the close-up photographing device 20 of the present embodiment is attached can realize the same functions as an ophthalmic slit lamp microscope as a whole.

[0072] Also, as shown in FIG. 7, the main body portion 61' of the slit light forming member 61 includes a first reflection mirror 65 that reflects light from the light source 92, a second reflection mirror 66 that reflects the light reflected by the first reflection mirror 65, and a slit portion 67 that allows the light reflected by the second reflection mirror 66 to pass through. The light that has passed through the slit portion 67 becomes slit light SL by the cylindrical lens 62. Then, this slit light SL is irradiated onto the eye E to be examined, and the light including the reflected light RL is condensed by the convex lens member 93 onto the imaging camera lens 91 of the mobile communication terminal device 10 while being received by the image sensor of the camera module, so that each tissue of the anterior eye part such as the eyelid, ocular surface, cornea, conjunctiva, lens, and anterior chamber of the eye E to be examined can be observed and photographed in detail, and the fundus tissue can also be observed and photographed. Note that this cylindrical lens 62 is not particularly limited, and can be selected and employed from various cylindrical lenses. The width of the slit portion 67 is preferably a narrow slit of about 1 mm, and is preferably formed to have a width in the range of about 0.7 to 1.5 mm.

[0073] The close-up photographing device 20A of the present embodiment having the above configuration can observe and photograph the anterior segment of the eye E to be examined (particularly, each tissue of the lens and the anterior chamber) by (a) attaching the slit light forming member 61 and the convex lens member 93 and removing the color filter member 97. (b) Also, by attaching only the convex lens member 93 and removing the slit light forming member 61 and the color filter member 97, each tissue of the eyelid, ocular surface, cornea, and conjunctiva of the eye E to be examined can be observed and photographed. By doing so, the lens tissue and the anterior chamber tissue can be observed and photographed in detail while using the slit light SL generated based on the light source light emitted from the light source 92 as the observation light. Further, the eyelid, ocular surface, cornea, and conjunctiva of the eye E to be examined can be observed and photographed in detail using the light source light (white diffused light) that has passed through the hole 88 as it is or the slit light SL as the observation light. In the present embodiment, in order to estimate the state of cataract in the eye E to be examined, the mobile communication terminal device 10 (smart eye camera) equipped with the close-up photographing device 20 in a state where the color filter member 97 is removed from the configuration shown in FIGS. 5 to 7 with the slit light forming member 61 and the convex lens member 93 attached is used to observe and photograph the lens tissue of the eye E to be examined. Further, the close-up photographing device 20A of the present embodiment can also observe the fundus tissue (retina) of the eye E to be examined with the slit light forming member 61 and the convex lens member 93 attached, but for detailed observation and photographing of the fundus tissue, it is preferable to attach a cylindrical member 180 (described later) for observing the fundus tissue.

[0074] As described above, the close-up photographing device 20A of the present embodiment can realize the same functions as an expensive ophthalmic slit lamp microscope simply by being attached to a mobile communication terminal device 10 such as a conventional smartphone. As a result, detailed images of the anterior segment tissues such as the eyelid, ocular surface, cornea, conjunctiva, anterior chamber, and lens of the eye E to be examined can be photographed at low cost and simply, and a moving image including a diagnostic frame image for cataract can be easily photographed.

[0075] [A4] Verification Results of Cataract Diagnosis Using a Smart Eye Camera In December 2018, the inventor conducted observations using a mobile communication terminal device 10 (smartphone, i.e., smart eye camera) equipped with the proximity shooting device 20A having the configuration shown in FIGS. 5 to 7, with the slit light forming member 61 and the convex lens member 93 attached and the color filter member 97 removed, for comparison with a smart eye camera and an existing ophthalmic slit lamp microscope. At this time, the cases were 58 eyes of the anterior segment (21 males and 37 females). The examination items were the presence or absence of eyelid and anterior segment diseases, cataract severity, left-right difference, shooting seconds, etc. Also, during this verification, the state of dry eye that developed in the test eye E was observed, etc., which will be described in detail in the section on modified examples.

[0076] At this time, for the proximity shooting device 20A, for focus adjustment, one equipped with a detachable convex lens 96 (focal length: 10 to 30 mm, magnification: 20 times) on the shooting camera lens 91 of the mobile communication terminal device 10 was used. This convex lens 96 is not particularly limited, but in this verification example, TK-12P (manufactured by T.S.K. Co., Ltd.) was used. The illuminance of the digital lux illuminator (model name: LX-1010B, manufactured by Zhangzhou WeiHua Electronic Co., Ltd.) of the mobile communication terminal device 10 used this time was 8000 lux. Also, for this verification, the proximity shooting device 20A designed for iPhone 7 (registered trademark) was used.

[0077] (Observation results of the anterior segment) Actually, FIG. 8 shows an example of an image taken of the above 58 eyes using the mobile communication terminal device 10 equipped with the proximity shooting device 20A shown in FIGS. 5 to 7. As shown in FIG. 8, evaluations of various anterior segment diseases such as pterygium, corneal opacity, epidemic keratoconjunctivitis, and intraocular lens eyes were possible for all cases. In FIG. 8, (A) is an image of corneal opacity, (B) is after cataract surgery, (C) is epidemic conjunctivitis, (D) is without cataract, (E) is moderate cataract, and (F) is severe cataract.

[0078] Fig. 9 and Fig. 10 show the comparison results between the cataract diagnosis results obtained by the mobile communication terminal device 10 equipped with the close-up photographing device 20A shown in Figs. 5 to 7 and the diagnosis results obtained by an existing slit lamp microscope. In Fig. 9, the vertical axis represents the NS grade indicating the severity of cataract, the EX on the horizontal axis represents the evaluation result by the existing device, and SEC1 to 3 represent the results diagnosed by three ophthalmologists based on the images taken by the mobile communication terminal device 10 equipped with the close-up photographing device 20A shown in Figs. 5 to 7.

[0079] As shown in Fig. 9, there were no significant differences among the observers, nor between the left and right eyes. Fig. 10 is a diagram showing the correlation between the diagnosis results obtained by the mobile communication terminal device 10 equipped with the close-up photographing device 20A by each ophthalmologist and the diagnosis results obtained by an existing ophthalmic slit lamp microscope. A correlation was recognized between the mobile communication terminal device 10 equipped with the close-up photographing device 20A and the existing ophthalmic slit lamp microscope, and a high correlation was also observed for the left and right eyes. These results indicate that the mobile communication terminal device 10 equipped with the close-up photographing device 20A shown in Figs. 5 to 7 has sufficient objectivity and reproducibility to obtain the observation phenotype of the anterior eye tissue of the eye to be examined E.

[0080] As described above, it was confirmed that the mobile communication terminal device 10 equipped with the close-up photographing device 20A of the present embodiment can observe various symptoms of the anterior eye of the eye to be examined E without inferiority compared to an existing ophthalmic slit lamp microscope.

[0081] Then, in the diagnosis support system 1 of the present embodiment, a moving image of the lens tissue of the eye to be examined E is taken using the mobile communication terminal device 10 equipped with the close-up photographing device 20A described above, and the state of cataract in the eye to be examined E is estimated based on the taken moving image, and diagnosis support information is generated. In addition to nuclear sclerosis, there are diseases such as atopic cataract and cortical cataract in cataracts, but for any of these diseases, a moving image including a diagnosable frame image can be taken by the same method as for nuclear sclerosis.

[0082] With this configuration, the diagnostic support system 1 of the present embodiment can capture a moving image including a diagnosable frame that can diagnose the state of cataracts in the eye E to be examined using the mobile communication terminal device 10 (i.e., a smart eye camera) equipped with the close-up photographing device 20A that can be manufactured at low cost without using an existing expensive ophthalmic slit lamp microscope, and can estimate the state of cataracts in the eye E to be examined. In addition, the smart eye camera of the present embodiment can basically capture a moving image including a diagnosable frame image by the same operation as a conventional device such as a smartphone. Therefore, even when a paramedical person or a patient's family member who does not have specialized knowledge, or even the user himself / herself (i.e., the patient himself / herself) takes a picture of his / her own eye, based on the diagnosable frame included in the captured moving image, the state of cataracts in the eye E to be examined can be appropriately estimated. When the user himself / herself takes a picture of his / her own eye as the eye E to be examined for self-check, while creating a state in which the slit light SL is irradiated on the eye E [i.e., the eye to be checked (for example, the right eye)], a moving image of the eye E (right eye) may be captured by the smart eye camera. However, in the case of the present embodiment, since the close-up photographing device 20A uses the out-camera module of the mobile communication terminal device 10, when actually taking a picture, open both eyes in front of a mirror, project the image being captured (i.e., the image of the right eye) displayed on the display unit of the mobile communication terminal device 10 onto the mirror while irradiating the slit light SL on the right eye that becomes the eye E to be examined, and while checking through the mirror what kind of image is being captured with the left eye (i.e., the eye opposite to the eye E to be examined), adjust the position and orientation of the smart eye camera and capture a moving image of his / her own right eye.

[0083] [A5] Configuration of the Diagnostic Support Server Device 30 Next, the configuration of the diagnostic support server device 30 of the present embodiment will be described with reference to FIGS. 11 and 12. Note that FIG. 11 is a block diagram showing a configuration example of the diagnostic support server device 30 of the present embodiment, and FIG. 12 is an image diagram for explaining the processing executed by the diagnostic processing unit 350 of the present embodiment.

[0084] As shown in FIG. 11, the diagnostic support server device 30 of the present embodiment includes a communication control unit 310 communicatively connected to the network N, a ROM / RAM 320 functioning as various memories, a storage device 330, a server management control unit 340 that controls the entire device, and a diagnostic processing unit 350 that executes processes of: (1) acquiring moving image data uploaded from the mobile communication terminal device 10; (2) extracting diagnosable frames from the moving image data; (3) estimating the state of cataract in the eye to be examined E based on the diagnosable frames; (4) generating diagnostic support information based on the estimation result; and (5) distributing the generated diagnostic support information to the corresponding mobile communication terminal device 10. Each of the above units is interconnected by a bus B, and data transfer between the respective components is executed.

[0085] The communication control unit 310 is a predetermined network interface and is communicatively connected to the mobile communication terminal device 10 via the network N. Further, the communication control unit 310 is communicatively connected to the annotation terminal device 40 directly or via the network N, and exchanges various data with the mobile communication terminal device 10 and the annotation terminal device 40.

[0086] The ROM / RAM 320 stores various programs necessary for driving the diagnostic support server device 30. Further, the ROM / RAM 320 is used as a work area when various processes are executed.

[0087] The storage device 330 is constituted by an HDD (Hard Disc Drive) or an SSD (Solid State Drive).

[0088] The storage device 330 is provided with: (a1) a program storage unit 331, (a2) an extraction knowledge storage unit 332, (a3) teacher data storage unit 333, and (a4) a diagnosis knowledge storage unit 334. Note that the extraction knowledge storage unit 332 and the diagnosis knowledge storage unit 334 of the present embodiment respectively constitute, for example, the "first storage means" and the "second storage means" of the present invention.

[0089] In the program storage unit 331, for example, together with programs such as BIOS (Basic Input Output System) and OS (Operating System), (b1) a diagnosable frame image extraction program for executing a process of extracting a diagnosable frame image based on the moving image data of the eye E to be examined uploaded from the mobile communication terminal device 10 and extraction knowledge, (b2) a diagnostic knowledge acquisition program for executing a process of acquiring diagnostic knowledge, and (b3) a diagnostic support information generation program for executing a process of generating and distributing diagnostic support information based on the diagnosable frame image and diagnostic knowledge are stored.

[0090] In the extraction knowledge storage unit 332, knowledge necessary for extracting a frame image that satisfies all of the above conditions 1 to 3 is stored in advance as knowledge for extracting a diagnosable frame image capable of diagnosing the state of cataract in the eye E to be examined. Note that the specific acquisition method and data format of the extraction knowledge in the present embodiment are arbitrary. For example, for the convolutional neural network (also referred to as a convolutional neural network) that constitutes the diagnosable frame extraction unit 352 described later, weights and biases obtained by inputting an arbitrary diagnosable frame image extracted by an annotator as teacher data may be acquired as extraction knowledge and stored in the extraction knowledge storage unit 332.

[0091] In the teacher data storage unit 333, teacher data acquired from the annotation terminal device 40 is stored each time moving image data is uploaded from the mobile communication terminal device 10.

[0092] In the diagnostic knowledge storage unit 334, diagnostic knowledge for estimating the presence or absence of cataract, NS grade, treatment method, necessity of surgery, etc. in the eye E to be examined is stored.

[0093] The server management control unit 340 is mainly composed of a CPU, and integrally controls each part of the diagnostic support server device 30 by executing programs such as an OS and BIOS.

[0094] The diagnosis processing unit 350 is configured by using the same CPU as the CPU that realizes the server management control unit 340, or is configured by a CPU independent of the server management control unit 340. Then, under the control of the server management control unit 340, the diagnosis processing unit 350 executes the program stored in the program storage unit 331, thereby realizing (c1) a moving image data acquisition unit 351, (c2) a diagnosable frame extraction unit 352, (c3) a diagnosis knowledge acquisition unit 353, (c4) a health state estimation processing unit 354, (c5) a diagnosis support information generation unit 355, and (c6) a diagnosis support information distribution unit 356.

[0095] (Moving image data acquisition unit 351) While interlocking with the communication control unit 310, the moving image data acquisition unit 351 acquires moving image data obtained by photographing the eye to be examined E from the mobile communication terminal device 10, and inputs the acquired moving image data to the diagnosable frame extraction unit 352 (step [1] in FIG. 12). Note that the moving image data acquisition unit 351 of the present embodiment interlocks with the communication control unit 310 and constitutes, for example, the "acquisition means" of the present invention.

[0096] (Diagnosable frame extraction unit 352) The diagnosable frame extraction unit 352 is constructed as, for example, a convolutional neural network using a CPU. Then, according to the diagnosable frame image extraction program, the diagnosable frame extraction unit 352 executes the diagnosable frame image extraction process shown in FIG. 13 based on the moving image data input from the moving image data acquisition unit 351 and the extraction knowledge, and extracts the diagnosable frame image from the moving image data (step [2] in FIG. 12). Then, the diagnosable frame extraction unit 352 inputs the extracted diagnosable frame image to each of the diagnosis knowledge acquisition unit 353, the health state estimation processing unit 354, and the diagnosis support information generation unit 355 (step [3] in FIG. 12). Note that the method by which the diagnosable frame extraction unit 352 inputs the diagnosable frame image to each of the units 353, 354, and 355 is arbitrary, and it may be directly input from the diagnosable frame extraction unit 352 to each of the units 353, 354, and 355. However, in the present embodiment, the description will be made assuming that the diagnosable frame image extracted by the diagnosable frame extraction unit 352 is once stored in the ROM / RAM 320, and each of the units 353, 354, and 355 reads it from the ROM / RAM 320 as necessary. Also, the number of frame images constituting the moving image data is arbitrary. For example, about 200 frames out of 600 to 1200 frames obtained by shooting with a smart eye camera at a frame rate of 30 fps for 20 to 40 seconds may be used. Note that in FIG. 13, only 7 frames (frames 1 to 7) included in these 200 frames are extracted and illustrated in order to prevent the figure from becoming complicated. Also, the diagnosable frame extraction unit 352 of the present embodiment constitutes, for example, the "first extraction means" of the present invention.

[0097] [A5.1] Principle of diagnosable frame image extraction process Next, the principle of the diagnosable frame image extraction process executed by the diagnosable frame extraction unit 352 of the present embodiment will be described with reference to FIG. 13. Note that FIG. 13 is an image diagram for explaining the diagnosable frame image extraction process executed by the diagnosable frame extraction unit 352 of the present embodiment according to the diagnosable frame image extraction program.

[0098] As shown in FIG. 13, in this process, the diagnosable frame extraction unit 352 determines whether the subject eye E is captured in each frame image included in the input moving image data. When the subject eye E is not captured, the diagnosable frame extraction unit 352 classifies the frame image into the Other class. Since the subject eye E is not captured in the frame images classified into the “Other” class, the diagnosable frame extraction unit 352 thins out the frames and determines for each of the remaining frame images whether at least a part of the in-focus crystalline lens tissue is captured together with the reflected light RL of the slit light SL in the crystalline lens tissue.

[0099] Then, the diagnosable frame extraction unit 352 classifies only the frame images in which at least a part of the in-focus crystalline lens tissue is captured together with the reflected light RL into the Diagnosable class. On the other hand, the diagnosable frame extraction unit 352 classifies other frame images (i.e., frame images in which at least a part of the crystalline lens tissue is not captured together with the reflected light RL of the slit light SL in the crystalline lens tissue or frame images in which the crystalline lens tissue is out of focus) into the Eye class. As a result, by this process, only the frame images that satisfy the above three conditions are classified into the “Diagnosable” class, and the other frame images are classified into the “Other” class or the “Eye” class. For example, FIG. 13 shows an example in which frames 1 and 2 are classified into the “Other” class, frames 3, 4, and 7 are classified into the “Eye” class, and frames 5 and 6 are classified into the “Diagnosable” class based on the above conditions.

[0100] At this time, as shown in FIG. 13, the diagnosable frame extraction unit 352 calculates the probability (or likelihood) corresponding to each of the classes of "Other", "Eye", and "Diagnosable" for each frame image included in the moving image data. Then, the diagnosable frame extraction unit 352 is configured to classify which class of "Other", "Eye", or "Diagnosable" each frame image belongs to based on the calculated corresponding probability.

[0101] Note that at this time, for all frame images included in the moving image data, the probability corresponding to each class may be calculated. However, if the configuration is such that the corresponding probability is calculated for all frame images, the processing load on the diagnosable frame extraction unit 352 may increase. For this reason, in the present embodiment, while pre-labeling the tissue names for the in-focus tissue in each frame image included in the moving image data, only the frame images in focus on the lens tissue are extracted based on the labels attached to each frame image, and a configuration is adopted in which the probability corresponding to each class is calculated for each of the extracted frame images.

[0102] With this configuration, the diagnostic support server device 30 of the present embodiment can preliminarily thin out frame images that are out of focus on the lens tissue and extract diagnosable frame images from the remaining frame images. As a result, the processing load of the diagnosable frame image extraction process can be reduced, and the image classification problem at the time of extracting diagnosable frame images can be simplified, improving the accuracy and recall at the time of extracting diagnosable frame images. Note that the specific method for labeling the tissue name for the in-focus tissue in each frame image is arbitrary. For example, a standard label engine for eye tissues may be installed in the diagnosable frame extraction unit 352, and the tissue name may be labeled for the in-focus tissue in each frame image using this standard label engine. Further, when the eye E to be examined is photographed from the front, for example, a plurality of anterior eye tissues such as eyelids, ocular surface, cornea, and conjunctiva may be simultaneously captured in focus in one frame image. Therefore, the diagnosable frame extraction unit 352 is configured to perform multi-labeling for labeling the tissue name written in each pixel in each frame image. Note that the specific criteria for classifying each frame image into each class based on the probability calculated by the diagnosable frame extraction unit 352 are arbitrary. For example, the probabilities corresponding to each of the three classes may be compared and classified into the class with the highest corresponding probability, or may be classified into the class where the corresponding probability exceeds a predetermined threshold (for example, 60%).

[0103] Then, based on the calculated probability, the diagnosable frame extraction unit 352 extracts diagnosable frame images from all the frame images included in the moving image data. Note that the specific criteria for the diagnosable frame extraction unit 352 to extract diagnosable frame images based on the calculated probability are arbitrary. For example, it may be configured to uniformly extract all the frame images classified into the "Diagnosable" class (e.g., frames 5 and 6 in FIG. 13) as diagnosable frame images, or it may be configured to extract frame images whose probability of belonging to the "Diagnosable" class exceeds a predetermined threshold (e.g., 60%) as diagnosable frame images. However, in the present embodiment, for the sake of concretizing the explanation, it is assumed that a configuration is adopted in which the upper predetermined number of frame images with a high probability of belonging to the "Diagnosable" class (e.g., the upper several frames with a high corresponding probability, or the upper several % of frames with a high corresponding probability) are extracted as diagnosable frame images, and the explanation will be given accordingly.

[0104] When diagnosable frame images are extracted by the above-described diagnosable frame image extraction process, the diagnosable frame extraction unit 352 associates the extracted diagnosable frame images with the probability that the diagnosable frame images belong to the "Diagnosable" class and stores them in the ROM / RAM 320. As a result, when the health state estimation unit 354 later estimates the presence or absence of cataract and the NS grade, etc. in the eye E to be examined based on each diagnosable frame image, the probability that each diagnosable frame image belongs to the "Diagnosable" class can be specified.

[0105] (Diagnosis knowledge acquisition unit 353) The diagnostic knowledge acquisition unit 353 is constructed as, for example, a convolutional neural network using a CPU. Then, the diagnostic knowledge acquisition unit 353 transmits the diagnosable frame images extracted by the diagnosable frame extraction unit 352 to the annotation terminal device 40 according to the diagnostic knowledge acquisition program, thereby acquiring teacher data from the annotation terminal device 40 and storing it in the teacher data storage unit 333. Further, when the number of teacher data accumulated in the teacher data storage unit 333 reaches α or more, the diagnostic knowledge acquisition unit 353 acquires diagnostic knowledge by performing at least one of machine learning and data mining based on the teacher data, and stores it in the diagnostic knowledge storage unit 334. Note that the specific method for the diagnostic knowledge acquisition unit 353 to acquire diagnostic knowledge and the specific content of the diagnostic knowledge are arbitrary in this embodiment. For example, it may be configured to acquire, as diagnostic knowledge, the parameter of weights and biases obtained by inputting the teacher data accumulated in the teacher data storage unit 333 to the convolutional neural network constituting the diagnostic knowledge acquisition unit 353, and store the parameter in the diagnostic knowledge storage unit 334. Further, the diagnostic knowledge acquisition unit 353 of this embodiment is linked with the communication control unit 310 and the annotation terminal device 40, and constitutes, for example, the "learning means" of the present invention. Furthermore, the specific number of α in this embodiment is arbitrary. However, in order to acquire diagnostic knowledge, it is necessary to use at least about several hundred pieces of teacher data as samples. However, in the diagnostic support system 1 of this embodiment, a plurality of diagnosable frame images (for example, the top frames with high probability or the frames of the top percentage) are extracted from one moving image data, and teacher data is created for each diagnosable frame image. Therefore, a large amount of teacher data can be created from a small amount of moving image data to acquire diagnostic knowledge. For example, when 1000 pieces of teacher data are required as α, and 5 frames of diagnosable frame images are extracted from 1 piece of moving image data, 1000 pieces of teacher data can be created from 200 pieces of moving image data to acquire the necessary diagnostic knowledge.

[0106] (Health state estimation processing unit 354) The health state estimation processing unit 354 is configured as, for example, a convolutional neural network using a CPU. Then, according to the health state estimation program, the health state estimation processing unit 354 estimates the presence or absence of cataracts and the NS grade, etc. in the eye E to be examined based on the diagnostic knowledge stored in the diagnostic knowledge storage unit 334 and the diagnosable frame image extracted by the diagnosable frame extraction unit 352. Note that the health state estimation processing unit 354 of the present embodiment constitutes, for example, the "estimation means" of the present invention.

[0107] In particular, as a characteristic matter in the present embodiment, the health state estimation processing unit 354 is configured to estimate the NS grade, etc. for each diagnosable frame image based on each of a plurality of diagnosable frame images extracted by the diagnosable frame extraction unit 352 (see FIG. 12). Then, the health state estimation processing unit 354 estimates the most likely state of cataracts in the eye E to be examined based on the value of the NS grade estimated based on each diagnosable frame image. For example, in FIGS. 12 and 13, when frames 5 and 6 included in the moving image data are extracted as diagnosable frame images and the NS grades of "1.8" and "2.3" are estimated for each frame, based on these estimation results, an example is shown in which the health state estimation processing unit 354 estimates the value of the NS grade of "2.1" as the most likely health state of the eye E to be examined.

[0108] At this time, as a method for the health state estimation processing unit 354 to estimate the most likely health state of the eye E to be examined, (1) a method of taking the average of the NS grade estimation values corresponding to each diagnosable frame image, and (2) a method of taking a majority vote of the estimation results for each diagnosable frame image are also conceivable. However, in the present embodiment, the health state estimation processing unit 354 is described as executing ensemble machine learning based on the NS grade estimation values corresponding to each diagnosable frame image to estimate the most likely health state of the eye E to be examined. Specifically, the health state estimation processing unit 354 weights the NS grade estimation values based on each diagnosable frame image by the probability corresponding to the "Diagnosable" class calculated by the diagnosable frame extraction unit 352, and estimates the most likely health state of the eye E to be examined. For example, when five frame images are extracted as diagnosable frame images and NS grade estimation values are obtained for each of them, the health state estimation processing unit 354 estimates the most likely health state of the eye E to be examined by regressing the NS grade estimation values with [estimation value, probability (Diagnosable)]*5 as the input.

[0109] Generally, when estimating the NS grade of cataracts in the eye E to be examined using a classifier, an estimated value based on a diagnosable frame image with a high probability of belonging to the "Diagnosable" class has a higher correct answer rate than an estimated value based on a diagnosable frame image with a lower corresponding probability. Therefore, by weighting the estimated value of the NS grade according to the probability of belonging to the "Diagnosable" class calculated for each diagnosable frame image with the above configuration, the weight of the estimated value with a high correct answer rate is increased, and the most likely health state of the eye E to be examined can be appropriately estimated, thereby improving the reliability of the diagnostic support information. When strict reliability of the diagnostic support information is not required, in the diagnosable frame image extraction process, a configuration may be adopted in which the state of cataracts in the eye E to be examined is estimated based only on the best frame with the highest probability of belonging to the "Diagnosable" class. Also, regarding the estimation method for states other than the NS grade, it is arbitrary. For example, it may be determined by a majority vote, or the estimation result based on the diagnosable frame image with the highest probability of belonging to "Diagnosable" may be used.

[0110] (Diagnostic support information generation unit 355) The diagnostic support information generation unit 355 generates diagnostic support information including the estimation result of the health state of the eye E to be examined by the health state estimation unit 354 by executing processing according to a diagnostic support information generation program. Note that the health state estimation unit 354 of the present embodiment constitutes, for example, the "generation means" and the "second extraction means" of the present invention.

[0111] In particular, as a characteristic feature of the present embodiment, the diagnostic support information generation unit 355 extracts a diagnostic frame image in which the state of the cataract in the subject eye E is most easily recognizable based on the labeling of the tissue name performed by the diagnosable frame extraction unit 352, and generates diagnostic support information including the diagnosable frame image. Generally, a diagnosable frame image in which the lens tissue is most prominently captured is the image in which the state of the cataract is most easily recognizable. Therefore, in the present embodiment, the diagnostic support information generation unit 355 extracts, as a frame image for presenting to the user (hereinafter referred to as "presentation frame image"), a diagnosable frame image in which the area of the pixel region in which the lens tissue is captured is the largest based on the diagnosable frame image in which the tissue name is labeled for each pixel in the diagnosable frame extraction unit 352. Then, the diagnostic support information generation unit 355 generates diagnostic support information including the estimation result regarding the state of the cataract and the presentation frame image.

[0112] With this configuration, according to the diagnostic support system 1 of the present embodiment, the diagnostic support information including the image in which the user can most easily recognize the state of the cataract can be generated. As a result, when the user explains the symptoms to the patient or the patient himself / herself self-checks the health condition of his / her own eyes, the user experience of the patient can be dramatically improved.

[0113] (Diagnostic support information distribution unit 356) The diagnostic support information distribution unit 356 distributes the diagnostic support information generated by the diagnostic support information generation unit 355 to the mobile communication terminal device 10 which is the upload source of the moving image data in conjunction with the communication control unit 310. Note that the diagnostic support information distribution unit 356 of the present embodiment constitutes, for example, the "distribution means" of the present invention in conjunction with the communication control unit 310. Further, the target to which the diagnostic support information distribution unit 356 distributes the diagnostic support information is not limited to the mobile communication terminal device 10 which is the upload source of the moving image data, and may be, for example, another communication terminal device used by the user (for example, a PC) or a communication terminal device unrelated to the user (for example, a computer system owned by a cataract research institution, etc.).

[0114] [A6]Operation of the Diagnostic Support System 1 [A6.1]Diagnostic Knowledge Acquisition Process Next, with reference to FIG. 14, the diagnostic knowledge acquisition process executed in the diagnostic support system 1 of the present embodiment will be described. Note that FIG. 14 is a flowchart showing the diagnostic knowledge acquisition process executed by the diagnostic processing unit 350 of the present embodiment in conjunction with the mobile communication terminal device 10 and the annotation terminal device 40 according to the diagnostic knowledge acquisition program and the diagnosable frame image extraction program.

[0115] Prior to this process, it is assumed that the extraction knowledge storage unit 332 stores in advance the extraction knowledge necessary for extracting the diagnosable frame image for cataract.

[0116] In addition, when acquiring the diagnostic knowledge for cataract in the diagnostic support system 1 of the present embodiment, it is necessary to capture a moving image including the diagnosable frame image for cataract using the smart eye camera. For this reason, prior to this process, the user removes the color filter member 97 from the proximity photographing device 20A constituting the smart eye camera and attaches the convex lens member 93 and the slit light forming member 61 so that the proximity photographing device 20A has the configuration shown in FIGS. 5 to 7.

[0117] In this state, when the user performs a predetermined input operation on an operation unit (not shown) of the mobile communication terminal device 10, in the mobile communication terminal device 10, according to the diagnostic application, the camera module is activated. As a result, in the mobile communication terminal device 10, together with the image being taken, a string of text instructing operation procedures such as "Adjust so that the slit light SL hits the black pupil part of the eye to be diagnosed, press the start button, and take a picture of the eye for about 20 to 40 seconds. At this time, while gradually changing the irradiation angle of the slit light, if you take a picture while slowly scanning the black pupil part, the possibility of accurate diagnosis will increase. Also, after pressing the stop button when the shooting is completed, please press the send button." and buttons for "Shooting start", "Stop", and "Send" are displayed on the display unit. Note that the operation procedures and each button may be displayed superimposed on the image being taken, or the taken image and the display area may be separated for display. Furthermore, the operation procedures may be guided by voice.

[0118] Then, when the user selects the shooting start button according to the operation procedures displayed on the display unit, in the mobile communication terminal device 10, the light source 92 emits light, the slit light SL is irradiated forward, and at the same time, shooting of a moving image is started in the camera module. In this state, when the user irradiates the crystalline lens tissue of the eye to be examined E with the slit light SL and slowly scans the black pupil part of the eye to be examined E with the slit light SL while changing the irradiation angle of the slit light SL, the light including the reflected light RL of the slit light SL in the eye to be examined E is condensed by the convex lens member 93 of the close-up photographing device 20A onto the photographing camera lens 91, and moving image data including a series of frame images as illustrated in FIG. 13 is generated. At this time, in the camera module of the mobile communication terminal device 10, while automatically focusing on the crystalline lens tissue by the autofocus mechanism, a moving image is taken, and in this regard, it is the same as a conventional mobile communication terminal device such as a smartphone.

[0119] Then, after the user follows the operation procedure and takes a picture of the lens tissue of the eye E to be examined for about 20 to 40 seconds, for example, when an input operation is performed to select buttons in the order of the stop button and the transmission button, the mobile communication terminal device 10 uploads the captured moving image data to the diagnostic support server device 30 according to the diagnostic application.

[0120] When the moving image data uploaded from the mobile communication terminal device 10 in this way is received by the communication control unit 310 of the diagnostic support server device 30 (step Sa1), in the diagnostic processing unit 350, the diagnosable frame extraction unit 352 reads the extraction knowledge from the extraction knowledge storage unit 332 (step Sa2). Then, the diagnosable frame extraction unit 352 executes the diagnosable frame image extraction process shown in FIG. 13 according to the diagnosable frame image extraction program, extracts the diagnosable frame images included in the moving image data, and stores them in the ROM / RAM 320 (step Sa3).

[0121] Next, the diagnostic knowledge acquisition unit 353 transmits all the diagnosable frame images stored in the ROM / RAM 320 to the annotation terminal device 40 (step Sa4). Regarding the form when the diagnostic knowledge acquisition unit 353 transmits the diagnosable frame images to the annotation terminal device 40, it is arbitrary. The diagnosable frame images may be broadcast to all the annotation terminal devices 40, or a configuration may be adopted in which all the diagnosable frame images are transmitted only to some of the annotation terminal devices 40.

[0122] When receiving the diagnosable frame images extracted from the moving image data in this way, the annotation terminal device 40 displays, together with each diagnosable frame image, for example, a character string such as "Please input the diagnosis result of cataract based on this image." and a GUI for inputting the diagnosis result on a monitor (not shown). Note that the display mode at this time is arbitrary. By dividing the display area of the monitor and associating each diagnosable frame image with a diagnosis result input GUI in each display area respectively, a diagnosis screen based on all the diagnosable frame images may be displayed at once, or the diagnosable frame images may be displayed one by one while switching.

[0123] In this state, a doctor serving as an annotator makes a diagnosis such as the presence or absence of cataract, NS grade, treatment method, necessity of surgery, etc. based on each displayed diagnosable frame image, and inputs the diagnosis result according to the GUI. Then, the annotation terminal device 40 generates diagnosis result information in response to the input operation. And the annotation terminal device 40 creates teacher data corresponding to each diagnosable frame image by tagging the corresponding diagnosable frame image with the diagnosis result information, and returns the created teacher data to the diagnostic support server device 30. For example, when 5 diagnosable frame images are extracted in the diagnosable frame image extraction process, the annotation terminal device 40 creates 5 pieces of teacher data corresponding to each diagnosable frame image and transmits them to the diagnostic support server device 30.

[0124] On the other hand, in the diagnostic support server device 30, when the communication control unit 310 receives teacher data from the annotation terminal device 40 (step Sa5 "Yes"), the diagnostic knowledge acquisition unit 353 stores the received teacher data in the teacher data storage unit 333 (step Sa6), and then determines whether the number of teacher data stored in the teacher data storage unit 333 is α or more (step Sa7). And when the number of teacher data is less than α (step Sa7 "No"), the diagnostic knowledge acquisition unit 353 ends the process.

[0125] Each time moving image data is uploaded from the mobile communication terminal device 10, the processes of steps Sa1 to 7 are repeatedly executed in the diagnosis processing unit 350, and teacher data is sequentially accumulated in the teacher data storage unit 333. For example, when α is set to 1000 and a method of extracting the top 5 frames with a high corresponding probability as diagnosable frame images is adopted, teacher data is sequentially accumulated in the teacher data storage unit 333 until the 199th upload of moving image data, and the process ends. On the other hand, when the 200th moving image data is uploaded, since the number of teacher data stored in the teacher data storage unit 333 is α (1000) or more, the diagnosis knowledge acquisition unit 353 determines "Yes" in step Sa7, and acquires diagnosis knowledge by executing at least one of machine learning and data mining based on the teacher data accumulated in the teacher data storage unit 333 (step Sa8), stores it in the diagnosis knowledge storage unit 334 (step Sa9), and ends the process.

[0126] [A6.2] Diagnostic support information generation process Next, the diagnostic support information generation process executed in the diagnostic support server device 30 of the present embodiment will be described with reference to FIG. 15. Note that FIG. 15 is a flowchart showing the diagnostic support information generation process executed by the diagnosis processing unit 350 in conjunction with the mobile communication terminal device 10 according to the diagnostic support information generation program and the diagnosable frame image extraction program.

[0127] Prior to this process, it is assumed that the necessary knowledge has been stored in the extraction knowledge storage unit 332 and the diagnosis knowledge storage unit 334. Also, it is assumed that the user has configured the proximity photographing device 20A constituting the smart eye camera in a state where it can photograph a moving image including a diagnosable frame image for cataract as described above. That is, it is assumed that the proximity photographing device 20A is configured such that the slit light forming member 61 and the convex lens member 93 are attached and the color filter member 97 is removed.

[0128] In this state, when the user takes a picture of the eye to be examined E, selects the transmission button, and moving image data is uploaded from the mobile communication terminal device 10 (step Sb1 "Yes"), in the diagnosis processing unit 350, after the diagnosable frame extraction unit 352 reads the extraction knowledge from the extraction knowledge storage unit 332 (step Sb2), it executes the diagnosable frame image extraction process according to the diagnosable frame image extraction program. As a result, the diagnosable frame images included in the moving image data are extracted, and the probability that the corresponding diagnosable frame image belongs to the "Diagnosable" class is stored in the ROM / RAM 320 in association with each diagnosable frame image (step Sb3).

[0129] Next, the health state estimation processing unit 354 reads the diagnostic knowledge (step Sb4), and based on the diagnosable frame images stored in the ROM / RAM 320 and the diagnostic knowledge, executes the health state estimation process (step Sb5) to estimate the cataract state in the eye to be examined E. At this time, as shown in FIG. 12, the health state estimation processing unit 354 estimates the cataract state for each diagnosable frame image. Then, the health state estimation processing unit 354 estimates the most likely state of cataract in the eye to be examined E while weighting the estimation results by probability.

[0130] Next, the diagnostic support information generation unit 355 extracts the presentation frame images from the diagnosable frame images stored in the ROM / RAM 320 (step Sb6), and generates the diagnostic support information including the estimation result in step Sb5 and the presentation frame images (step Sb8). The diagnostic support information distribution unit 356 distributes the diagnostic support information generated in this way to the corresponding mobile communication terminal device 10 (step Sb9) and ends the process.

[0131] As described above, the diagnostic support system 1 of the present embodiment extracts a diagnosable frame image from a moving image captured by the mobile communication terminal device 10 (i.e., a smart eye camera) with the proximity shooting device 20A attached thereto, and based on the diagnosable frame image, estimates the presence or absence of cataract, NS grade, treatment method, necessity of surgery, etc. in the eye E to be examined. Then, while generating diagnostic support information including the estimation result, it is configured to distribute it to the corresponding mobile communication terminal device 10. With this configuration, the diagnostic support system 1 of the present embodiment can appropriately estimate the state of cataract in the eye E to be examined and generate and use diagnostic support information even when a layperson captures the eye E to be examined without using an expensive image capturing device for ophthalmic diagnosis.

[0132] Also, in the diagnostic support system 1 of the present embodiment, only the diagnosable frame image is extracted from the moving image captured by the mobile communication terminal device 10 and can be presented to the annotator (doctor) in the annotation terminal device 40. Therefore, the diagnostic support system 1 of the present embodiment can reduce the work burden of the doctor when acquiring diagnostic knowledge and create teaching data to acquire diagnostic knowledge. In particular, the diagnostic support system 1 of the present embodiment is configured to acquire diagnostic knowledge using a moving image captured by a smart eye camera. Since a moving image is likely to include a plurality of diagnosable frame images, highly accurate diagnostic knowledge can be acquired even when the number of samples of the moving image data is small. In the above embodiment, a configuration is adopted in which the functions of each of the units 351 to 356 constituting the diagnostic processing unit 350 of the diagnostic support server device 30 are realized by the CPU that executes the application program. However, the functions of each unit constituting the diagnostic processing unit 350 can of course be realized by hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0133] [A7]Modification of the First Embodiment [A7.1]Modification 1 In the above-described first embodiment, in the diagnostic support server device 30, while extracting diagnosable frame images from moving image data (step Sa3 in FIG. 14), the extracted diagnosable frame images are transmitted to the annotation terminal device 40 (step Sa4), and the teacher data acquired from the annotation terminal device 40 is stored in the teacher data storage unit 333 (step Sa6). However, the diagnostic support server device 30 may directly transmit the moving image data uploaded from the mobile communication terminal device 10 to the annotation terminal device 40 (step Sa4), acquire diagnostic result information corresponding to the doctor's diagnostic result based on the moving image from the annotation terminal device 40 (step Sa5), create teacher data based on the diagnostic result information in the diagnostic support server device 30, and store it in the teacher data storage unit 333 (step Sa6).

[0134] In this case, the annotation terminal device 40 is configured to display, on a monitor (not shown), the moving image of the subject eye E and the GUI for inputting the diagnostic result based on the moving image data received from the diagnostic support server device 30, and allow the doctor to input the diagnostic result based on the moving image.

[0135] Also, in this case, the diagnosable frame extraction unit 352 executes the diagnosable frame image extraction process and stores all the frame images classified into the "Diagnosable" class as diagnosable frame images in the ROM / RAM 320. Then, the diagnostic knowledge acquisition unit 353 creates teacher data while tagging all the diagnosable frame images stored in the ROM / RAM 320 with the diagnostic result information acquired from the annotation terminal device 40, and stores it in the teacher data storage unit 333 (step Sa6). In this way, when the number of teacher data accumulated in the teacher data storage unit 333 reaches α or more (step Sa7), the diagnostic knowledge acquisition unit 353 acquires diagnostic knowledge (step Sa8) and stores it in the diagnostic knowledge storage unit 334. With the configuration of this modification example, it is possible to facilitate sample collection when acquiring diagnostic knowledge and increase the number of samples to improve the accuracy of diagnostic knowledge.

[0136] [A7.2] Variant Example 2 In the above-described first embodiment, the annotation terminal device 40 is provided separately from the diagnostic support server device 30. However, the functions of the annotation terminal device 40 may be integrated into the diagnostic support server device 30, and the annotation terminal device 40 may be omitted. In this case, a monitor, keyboard, mouse, etc. (not shown) may be provided in the diagnostic support server device 30, and while the diagnostic frame image extracted by the diagnostic frame extraction unit 352 is displayed on the monitor of the diagnostic support server device 30, the doctor may be allowed to input the diagnostic result. Then, the corresponding diagnostic frame image may be tagged with the diagnostic result information corresponding to the input result to create teacher data. In this case, similar to Variant Example 1, while displaying a moving image, the doctor may be allowed to input the diagnostic result to create teacher data.

[0137] Also, the diagnostic support server device 30 may be connected to a computer (not shown) of an existing ophthalmic facility, and the computer of the ophthalmic facility may be used as the annotation terminal device 40, and the annotation terminal device 40 may be omitted. In this case, data of the electronic medical record may be acquired from the computer of the ophthalmic facility and used as additional teacher data. By adopting this configuration, the number of samples of teacher data available for obtaining diagnostic knowledge can be increased, and the quality of diagnostic knowledge can be improved.

[0138] [A7.3] Variant Example 3 In the first embodiment described above, the diagnosable frame extraction unit 352 labels the name of the tissue on the in-focus tissue in each frame image and extracts the diagnosable frame image from the frame images after the labeling. However, a configuration may be adopted in which the labeling process in the diagnosable frame extraction unit 352 is not performed. In this case, in step Sb6, an arbitrary diagnosable frame image may be extracted as the presentation frame image, or the diagnosable frame image with the highest probability corresponding to the "Diagnosable" class may be extracted as the presentation frame image. Furthermore, diagnostic support information that does not include the presentation frame image may be generated (step Sb7) and distributed.

[0139] [A7.4] Modification Example 4 In the first embodiment described above, a configuration is adopted in which the presence or absence of cataract, NS grade, treatment method, necessity of surgery, etc. in the subject eye E are estimated based on the two-dimensional moving image captured by the mobile communication terminal device 10. However, in the diagnosable frame extraction unit 352, a three-dimensional image may be constructed based on the moving image data, and the state of cataract in the subject eye E may be estimated based on the constructed three-dimensional image.

[0140] When capturing a moving image of the subject eye E using the camera module of the mobile communication terminal device 10, the moving image will be captured while focusing on each frame image by the autofocus mechanism of the camera module. At this time, each frame image is captured in a state of being in focus at a different focal length. Therefore, the in-focus tissues in each frame image are each at a slightly different distance from the imaging camera lens 91 of the mobile communication terminal device 10.

[0141] In particular, when using the slit light SL as the observation light, by making the slit light SL incident on the eye E to be examined from an oblique direction, while cutting the eye E to be examined in the cross-sectional direction with the slit light SL, a moving image including information in the cross-sectional direction of the eye E to be examined can be captured. In this case, the tissues in focus in each frame image will have different distances from the imaging camera lens 91. Therefore, by stacking each frame image in order according to the focal length, a three-dimensional image including information in the cross-sectional direction of the anterior eye tissue of the eye E to be examined can be constructed. In this case, it is desirable to stack only the diagnosable frame images to construct the three-dimensional image. However, it is not necessarily required to use only the diagnosable frame images, and frame images other than the diagnosable frame images may also be stacked to construct the three-dimensional image. Also, in this case, the diagnosable frame extraction unit 352 constitutes the "three-dimensional image construction means" of the present invention. Furthermore, in this case, it is also possible to construct a three-dimensional image including information such as unevenness on the surface of the eye E to be examined by using light other than the slit light SL as the observation light. That is, in this modification example, the observation light used for observing the eye E to be examined is not limited to the slit light SL.

[0142] The three-dimensional image constructed by this method can be used not only for estimating the state of cataract in the eye E to be examined but also as a presentation frame image. On the other hand, when estimating the health state of the eye E to be examined using the three-dimensional image, it is necessary to use knowledge for estimating the state of cataract in the eye E to be examined based on the three-dimensional image as diagnostic knowledge.

[0143] Therefore, when this method is adopted, the diagnosable frame extraction unit 352 extracts a diagnosable frame image in step Sa3 of FIG. 14, constructs a three-dimensional image, and stores the three-dimensional image in the ROM / RAM 320. Then, the diagnostic knowledge acquisition unit 353 is configured to transmit the three-dimensional image stored in the ROM / RAM 320 to the annotation terminal device 40 in step Sa4. On the other hand, the annotation terminal device 40 may be configured to display the received three-dimensional image, create teacher data by tagging the three-dimensional image with diagnostic result information based on the doctor's diagnostic result based on the three-dimensional image, and transmit the teacher data to the diagnostic support server device 30.

[0144] Also, in this case, when the communication control unit 310 receives teacher data from the annotation terminal device 40 (step Sa5 in FIG. 14), the diagnostic knowledge acquisition unit 353 of the diagnostic support server device 30 stores the teacher data in the teacher data storage unit 333 (step Sa6 in FIG. 14). Each time moving image data is uploaded from the mobile communication terminal device 10, the diagnosable frame extraction unit 352 accumulates the teacher data based on the constructed three-dimensional image in the teacher data storage unit 333 while constructing the three-dimensional image. Then, when the number of teacher data stored in the teacher data storage unit 333 reaches α or more (step Sa7 “Yes” in FIG. 14), based on the teacher data stored in the teacher data storage unit 333 (that is, the three-dimensional image tagged with diagnostic result information), the diagnostic knowledge acquisition unit 353 executes at least one of machine learning and data mining, acquires diagnostic knowledge for cataract based on the three-dimensional image (step Sa8 in FIG. 14), and stores the diagnostic knowledge in the diagnostic knowledge storage unit 334 (step Sa9 in FIG. 14).

[0145] In this way, when the knowledge for diagnosing cataract based on the three-dimensional image is acquired, when moving image data is later uploaded from the mobile communication terminal device 10, in the diagnosis processing unit 350, basically, the same diagnosis support information generation process as in FIG. 15 is executed, so that the diagnosis support information is generated (step Sb7 in FIG. 15) and distributed to the corresponding mobile communication terminal device 10 (step Sb8 in FIG. 15).

[0146] Also, in this case, the diagnosable frame extraction unit 352 extracts a diagnosable frame image from the moving image data uploaded from the mobile communication terminal device 10 in the diagnosable frame image extraction process of step Sb3, constructs a three-dimensional image, and stores it in the ROM / RAM 320. Then, in step Sb5, the diagnosis support information generation unit 355 estimates the state of cataract in the subject eye E based on the knowledge for diagnosing cataract based on the three-dimensional image stored in the diagnosis knowledge storage unit 334 and the three-dimensional image stored in the ROM / RAM 320.

[0147] Furthermore, in this case, the diagnosis support information generation unit 355 generates the diagnosis support information (step Sb7) while using the three-dimensional image stored in the ROM / RAM 320 as a presentation frame image (step Sb6), and distributes it to the corresponding mobile communication terminal device 10 (step Sb8). As a result, in the mobile communication terminal device 10, the three-dimensional image will be displayed in a state viewable by the user together with the estimation results regarding the presence or absence of cataract, the NS grade estimated value, the treatment method, and the necessity of surgery in the subject eye E. Note that also in this case, as the presentation frame image, instead of the three-dimensional image, a diagnosable frame image with the largest area of the pixel region in which the lens tissue is reflected may be extracted and used. Usually, more information is included in the three-dimensional image than in the planar image. Therefore, with the configuration of this modification example, it is possible to improve the estimation accuracy of the state of cataract in the subject eye E, improve the reliability of the diagnosis support information, and present the diagnosis support information including the three-dimensional image to the user, thus greatly improving the user experience (UX).

[0148] [A7.5] Modification Example 5 In the above-described first embodiment, the slit light forming member 61 and the convex lens member 93 are attached to the close-up photographing device 20A that constitutes the smart eye camera, the color filter member 97 is removed, and the eye to be examined E is photographed in a state where the slit light SL is irradiated onto the eye to be examined E as observation light, thereby adopting a configuration for estimating the state of cataract in the eye to be examined E. On the other hand, this modified example is for estimating the presence or absence of diseases other than cataract and the severity thereof.

[0149] Here, in order to estimate the presence or absence of diseases other than cataract in the eye to be examined E and the severity thereof, it is necessary to photograph a moving image including one or more diagnosable frame images that can be used for the diagnosis of the disease with the smart eye camera. Therefore, in this case, the user needs to photograph a moving image of the eye to be examined E while changing the configuration of the close-up photographing device 20A according to the disease to be estimated for the severity and the like. Hereinafter, the state estimation method for each disease will be specifically described.

[0150] [A7.5.1] Method for estimating the state of diseases such as iritis and uveitis that occur in the anterior chamber tissue When estimating the presence or absence of onset and the severity of diseases such as iritis and uveitis that occur in the anterior chamber tissue of the eye to be examined E, similar to the first embodiment, the slit light SL is generated by the close-up photographing device 20A, and in a state where the slit light SL is irradiated onto the anterior chamber tissue of the eye to be examined E as observation light, the light including the reflected light RL of the slit light SL in the anterior chamber tissue is condensed by the convex lens member 93 onto the photographing camera lens 91, and it is necessary to photograph a moving image including one or more diagnosable frame images corresponding to these diseases. Therefore, in this case, similar to the first embodiment, the slit light forming member 61 and the convex lens member 93 are attached to the close-up photographing device 20A that constitutes the smart eye camera, the color filter member 97 is removed, and while the close-up photographing device 20A is configured as shown in FIGS. 5 to 7, it is necessary to photograph a moving image of the eye to be examined E.

[0151] Also, in this case, the extraction knowledge storage unit 332 is pre-stored with extraction knowledge for extracting diagnosable frame images corresponding to iritis and uveitis. Note that the conditions for corresponding to the diagnosable frame image in this case are the same as those for the diagnosable frame image for cataract diagnosis, except that the tissue irradiated with the slit light SL is the anterior chamber tissue instead of the lens tissue.

[0152] Also, in this case, the diagnostic knowledge for iritis or uveitis is basically acquired by the same knowledge acquisition process as shown in FIG. 14. However, the process at this time is the same as that of the first embodiment, except that the disease to be the acquisition target of the diagnostic knowledge is iritis or uveitis instead of cataract, so the details are omitted.

[0153] Then, when moving image data is uploaded from the mobile communication terminal device 10 in a state where the diagnostic knowledge storage unit 334 stores the diagnostic knowledge for iritis or uveitis, the diagnostic support information is basically generated by the same diagnostic support information generation process as shown in FIG. 15 in the diagnostic processing unit 350 (step Sb7) and is distributed to the corresponding mobile communication terminal device 10 (step Sb8). Note that the process at this time is the same as that of the first embodiment, except that the target disease is iritis or uveitis instead of cataract in the same way as the diagnostic knowledge acquisition process.

[0154] [A7.5.2] Method for estimating the state of diseases such as chalazion and hordeolum that occur in the eyelid tissue, allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, and corneal opacity that occur in the corneal tissue and conjunctival tissue When estimating the presence or absence and severity of diseases such as chalazion and hordeolum that occur in the eyelid tissue, and allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, and corneal opacity that occur in the corneal tissue and conjunctival tissue, it is necessary to capture a moving image including a diagnosable frame image corresponding to these diseases with a smart eye camera. Here, as methods for capturing a moving image including a diagnosable frame image corresponding to these diseases, there are the following two methods. <Shooting Method 1> This imaging method is a method of capturing a moving image of the eye E to be examined while irradiating the eye E to be examined with slit light SL as observation light, with the close-up imaging device 20A constituting the smart eye camera having the same configuration as in the first embodiment (Figs. 5 to 7). <Imaging method 2> In this imaging method, a convex lens member 93 is attached to the close-up imaging device 20A that constitutes the smart eye camera, and the color filter member 97 and the slit light forming member 61 are removed, creating a state in which the light source light (white diffused light) emitted from the light source 92 passes through the holes 88 and 95 as it is and is irradiated forward, and while irradiating the eye E to be examined with the white diffused light as observation light, this is a method of capturing a moving image of the eye E to be examined.

[0155] However, in this case, the conditions for corresponding to a diagnosable frame image are different between the case of adopting Imaging method 1 and the case of adopting Imaging method 2. The conditions in the case of adopting Imaging method 1 are basically the same as in the case of cataract, but the irradiation target of the slit light SL is a tissue other than the lens, such as the eyelid and the ocular surface. On the other hand, the conditions in the case of adopting Imaging method 2 are that the reflected light RL of the white diffused light is reflected together with at least a part of the observation target tissue in a state where the white diffused light is irradiated as observation light, and it is a frame image in focus.

[0156] Therefore, when estimating the states of these diseases, the imaging method to be used is determined in advance, and the extraction knowledge corresponding to the imaging method is stored in the extraction knowledge storage unit 332. It is also possible to adopt a configuration in which a plurality of extraction knowledge corresponding to each imaging method are stored in the extraction knowledge storage unit 332 in advance, and the extraction knowledge to be used is switched according to the imaging method selected by the user to extract a diagnosable frame image. In this case, the user is made to specify the imaging method to be used on the diagnostic application, and information indicating the imaging method is uploaded from the mobile communication terminal device 10 together with the moving image data. Then, in the diagnosable frame image extraction process, the diagnosable frame extraction unit 352 may be configured to extract a diagnosable frame image while switching the extraction knowledge to be used based on the information.

[0157] Also, in this case, the diagnostic knowledge for these diseases, regardless of the imaging method adopted, will basically be acquired by the same diagnostic knowledge acquisition process as shown in FIG. 14. However, the process at this time is the same as that of the first embodiment except that the diseases for which diagnostic knowledge is to be acquired are not cataracts but chalazion, hordeolum, allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, corneal opacity, etc. that occur in the corneal tissue and conjunctival tissue. Therefore, the details are omitted.

[0158] And when moving image data is uploaded from the mobile communication terminal device 10 with the diagnostic knowledge for these diseases stored in the diagnostic knowledge storage unit 334, the diagnostic processing unit 350 basically generates diagnostic support information by the same diagnostic support information generation process as shown in FIG. 15 (step Sb7) and distributes it to the corresponding mobile communication terminal device 10 (step Sb8). Note that the process at this time is the same as that of the first embodiment except that the target disease is not cataract but chalazion, hordeolum, allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, corneal opacity, etc., similar to the diagnostic knowledge acquisition process.

[0159] Also, when adopting this configuration, the moving images captured by the smart camera may include diagnostic frame images that can be used for the diagnosis of multiple diseases. At this time, the user may specify which disease to target using the mobile communication terminal device 10, or the states of multiple diseases may be estimated at once. The method for estimating the states of multiple diseases at once will be described in detail in the section of Modification Example 6.

[0160] [A7.5.3] Method for estimating the state of diseases such as punctate superficial keratitis and corneal ulcer When estimating the presence or absence and severity of diseases such as punctate superficial keratitis and corneal ulcer, it is necessary to capture a moving image including a frame image capable of diagnosing the state of the wound on the cornea using a smart eye camera. For this reason, in this case, a color filter member 97 and a convex lens member 93 constituted by a blue-free filter for biological staining examination are attached to the close-up photographing device 20A constituting the smart eye camera, and the slit light forming member 61 is removed to configure the close-up photographing device 20A as shown in Fig. 3(B). Then, the light source light emitted from the light source 92 is changed to blue light by the color filter member 97, and a moving image of the subject eye E is captured in a state where the subject eye E is irradiated with the blue light as observation light. At this time, by instilling a fluorescein solution for contrast into the subject eye E, the wounds on the cornea and conjunctiva are changed to green, and the green light is received by the image sensor through the photographing camera lens 91 to capture the wounds on the cornea and conjunctiva of the subject eye E.

[0161] And in this case, in the extraction knowledge storage unit 332, extraction knowledge for extracting a diagnosable frame image that can be used for diagnosing diseases such as punctate superficial keratitis and corneal ulcer is stored in advance. In this case, the condition for corresponding to a diagnosable frame image is that it is a frame image in focus in which the reflected light RL of the blue light in the observation target tissue is reflected together with at least a part of the observation target tissue in a state where the observation target tissue is irradiated with blue light as the observation light. Therefore, in this case, it is necessary to store in advance in the extraction knowledge storage unit 332 the extraction knowledge for extracting a frame image that satisfies the condition.

[0162] Also, in this case, the diagnostic knowledge corresponding to diseases such as punctate superficial keratitis and corneal ulcer is basically obtained by the same diagnostic knowledge acquisition process as in Fig. 14. However, the process at this time is the same as that of the first embodiment except that the target disease of the diagnostic knowledge to be acquired is not cataract but punctate superficial keratitis, corneal ulcer, etc.

[0163] When moving image data is uploaded from the mobile communication terminal device 10 while diagnostic knowledge for diseases such as punctate superficial keratitis and corneal ulcer is stored in the diagnostic knowledge storage unit 334, the diagnostic processing unit 350 basically generates diagnostic support information by the same diagnostic support information generation process as shown in FIG. 15 (step Sb7) and distributes it to the corresponding mobile communication terminal device 10 (step Sb8). Note that the processing at this time is the same as that of the first embodiment except that the target disease is not cataract but diseases such as punctate superficial keratitis and corneal ulcer, similar to the diagnostic knowledge acquisition process. Also, in this case as well, the moving image captured by the smart eye camera may include diagnostic frame images that can be used for the diagnosis of multiple diseases. At this time, the user may specify the target disease using the mobile communication terminal device 10, or the states of multiple diseases may be estimated at once.

[0164] [A7.5.4] Method for estimating the state of dry eye disease (DED) When estimating the state of DED in the eye E to be examined, it is necessary to observe the state of the wounds on the cornea and conjunctiva and the state of the tears on the ocular surface. Therefore, in this case, similar to the case of estimating the state of punctate superficial keratitis and the like, a color filter member 97 and a convex lens member 93 constituted by a blue-free filter for vital staining examination are attached to the proximity photographing device 20A constituting the smart eye camera. The light source light emitted from the light source 92 is changed to blue light by the color filter member 97, and the observation target tissue of the eye E to be examined is photographed in a state where the blue light is irradiated as observation light to the eye E to be examined. Also, in this case, it is necessary to photograph the ocular surface, cornea, and conjunctiva of the eye E to be examined while instilling fluorescein solution into the eye E to be examined to contrast the wounds.

[0165] [A7.5.4.1] Regarding the verification results of DED diagnosis using a smart eye camera In order to verify whether the smart eye camera with the above configuration can diagnose DED, the inventor observed with a mobile communication terminal device 10 (smartphone) equipped with a close-up photographing device 20A to which a convex lens member 93 and a color filter member 97 were attached, and compared it with an existing device for DED observation. In this verification, as the close-up photographing device 20A, a device with the same configuration as that designed for the iphone7 (registered trademark) used in the above verification of cataract was used, and a color filter member 97 composed of an acrylic resin blue filter (PGZ 302K 302, manufactured by Kuraray Co., Ltd.) was attached.

[0166] In addition, in this verification, as Verification Example 1, in a DED mouse model related to graft-versus-host disease (GVHD), the images taken by the mobile communication terminal device 10 equipped with the verification close-up photographing device 20A and an existing device were compared, and an eye examination including the tear film break-up time (TFBUT) of the mouse model was performed. As Verification Example 2, TFBUT measurement, which is a diagnostic criterion for DED, was performed on a total of 42 people, including DED patients and normal people without DED. Note that TFBUT is an essential parameter used for the diagnosis of DED.

[0167] [Verification Example 1] DED is caused by a decrease in tear volume, a rapid breakdown of the tear film, and increased evaporation of tears, and TFBUT is one of the core mechanisms of DED. As for past DED research, DED mouse models have been studied, but methods for measuring human TFBUT have not been established in such DED mouse models. There are several reasons why the evaluation of TFBUT in mouse models cannot be directly applied to humans. First, the width of the mouse cornea is only 2-3 mm, which is small in size and difficult to focus for application to humans. Second, existing ophthalmic clinical slit lamp microscopes are used for examining the anterior segment of the eye (each tissue of the eyelid, ocular surface, cornea, conjunctiva, anterior chamber, and lens), but the device is large, not easily movable, and does not have an image recording function itself. Third, existing slit lamp microscopes are expensive and have a low cost-effectiveness. To avoid these problems, the use of tear secretion (TS) and corneal fluorescein score (CFS) for diagnosis in DED mouse models is increasing. However, even in such DED mouse models, a device that can be easily applied to humans has not yet been established.

[0168] Regarding Verification Example 1, FIG. 16 shows an image taken by the mobile communication terminal device 10 (smart eye camera) equipped with the close-up photographing device 20A having the above configuration. The upper row of FIG. 16 is a representative photograph of the eye irradiated with white light, and the lower row is a representative photograph showing a fluorescein staining image. The images in the left column of FIG. 16 are example images taken by an existing device, and the images in the right column are example images taken using the mobile communication terminal device 10 equipped with the close-up photographing device 20A having the above configuration. As the existing device, one widely used for evaluating the eyes of the DED mouse model was used. Specifically, a microscope (product name: SZ61), a camera (product name: DP70), and a light source (product name: LG-PS2) manufactured by Olympus Corporation were used. In Verification Example 2, a portable slit lamp microscope (product name: SL-15, Kowa Company Limited) was used as a comparative existing device.

[0169] (Mouse GVHD group-related DED model) For the DED mouse model, Zhang's method that reproduces the phenotype of GVHD-related DED was selected, similar to the clinical cases. The B10.D2 and BALB / cCrSlc (BALB / c) mice (7 weeks old) used were purchased from Sankyo Laboratories (Tokyo, Japan). After acclimating to the SPF environment for one week, the mice were divided into three groups (5 mice per group). For the DED (GVHD group model) group, allogeneic bone marrow transplantation (BMT) was performed using 8-week-old male B10.D2 and female BALB / c mice as donors and recipients. For the negative control (non-GVHD group), syngeneic BMT was performed by transplanting donor cells from male BALB / c mice to female BALB / c mice. Six hours before BMT, these recipient mice were irradiated with 700 cGy using a Gammacel 137Cs radiation source (Hitachi Medico Ltd.), and then the donor cells were injected via tail vein injection. For the healthy control (normal group control), female BALB / c mice of the same age were selected.

[0170] Three mouse models (GVHD group, non-GVHD group, and normal group control) were used for comparison. Since the DED phenotype in this DED mouse model appears 3 weeks after BMT, ocular phenotypes such as body weight, tear film break-up time (TFBUT), corneal fluorescein score (CFS), and tear secretion (TS) were collected once a week from before BMT (8 weeks old) to 12 weeks old. All the captured data recorded by the smart eye camera were manually transferred to an iMac (Apple Inc., USA) via Bluetooth (registered trademark) and converted to mp4 video data for safe storage.

[0171] (Evaluation of Tear Film Break-Up Time) Stability was measured using the tear film break-up time (TFBUT). The observer held the mouse with one hand and then used a micropipette to inject 1 μL of 0.5% sodium fluorescein into the conjunctival sac. After instilling the eye drops three times, the observer used the mobile communication terminal device 10 equipped with the above-described close-up photographing device 20A with the right hand to take a picture of the eye. To compare this method with the existing method, the TFBUT obtained with the existing device was evaluated by the conventional method.

[0172] (Evaluation of corneal fluorescein score) Corneal epithelial damage was evaluated using the corneal fluorescein score (CFS) evaluated 90 seconds after instillation of fluorescein. Each cornea was divided into four quadrants and recorded individually. The CFS was calculated using a four - level evaluation. Evaluation 1 was for cases showing slightly punctate staining of "<30 spots", evaluation 2 was for cases of punctate staining of ">30 spots" without diffusion, evaluation 3 was for cases with severe diffuse staining but no positive plaques, and evaluation 4 was for cases of fluorescein - positive plaques.

[0173] (Evaluation of tear secretion) Tear secretion (TS) was measured using the modified Schirmer test. A phenol red thread was placed on the temporal side of the upper eyelid margin for 15 seconds. The length of the wet part from the end was within 0.5 mm.

[0174] (Data analysis) Data analysis was performed using Prism software (Mac version 6.04; GraphPad Software, Inc., USA). The D’Agostino - Pearson omnibus normality test was used to evaluate whether the data showed a normal distribution. The Mann - Whitney U test was used to compare the differences between normal and target (GVHD group and non - GVHD group) for several parameters including body weight and TFBUT, CFS, and TS. The Wilcoxon signed - rank test was used to compare the differences between the results evaluated by the existing method and the smart eye camera. The Friedman test was used to compare the differences in TFBUT evaluations by three different ophthalmologists who photographed using the smart eye camera. The Lin's concordance correlation coefficient was used to evaluate the possible correlation between TFBUT and CFS using the existing technique and the smart eye camera. The data were presented as mean ± standard deviation (SD), and a P - value of less than 0.05 was considered statistically significant.

[0175] [Results] (Body weight) Figure 17(A) is a graph measuring the relationship between the age and body weight of mice, and Figure 17(B) is a table showing the progression of body weight by group (green: normal group, blue: non-GVHD group, red: GVHD group). To verify the applicability of the smart eye camera in a mouse model of mice, first, bone marrow transplantation (BMT) was evaluated. From the results in Figure 17, since the body weight of mice was initially adjusted for each group, there was no difference in body weight among the normal group, non-GVHD group, and GVHD group before BMT (8 weeks old). However, in the non-GVHD group and GVHD group at 9 and 10 weeks old, the body weight significantly decreased compared to the normal group. As a result, at 9 weeks old, P = 0.016 and 0.016, and at 10 weeks old, P = 0.016 and 0.032 for the normal group vs. non-GVHD group and normal group vs. GVHD group, respectively. The body weight decreased only in the GVHD group, with P = 0.032 and 0.032 at 11 and 12 weeks old, respectively, compared to the body weight in the normal group at 11 and 12 weeks old.

[0176] (Tear secretion volume) Figure 18(A) is a graph measuring the relationship between the age of mice and tear secretion volume (TS: Tear Secretion), and Figure 18(B) is a table showing the progression of continuous tear secretion (TS) by group (green: normal group, blue: non-GVHD group, red: GVHD group). A significant difference in TS was observed between the normal group and non-GVHD group, and between the normal group and GVHD group at 9 - 12 weeks old. Furthermore, at 12 weeks old, TS was significantly shorter in the GVHD group compared to the non-GVHD group. With N = 5 per group, the evaluation of significance (P < 0.05) was performed using the Mann-Whitney U test.

[0177] From the results of Fig. 18, TS showed no difference in terms of body weight before BMT, but these values significantly decreased in the non-GVHD group and GVHD group compared to the normal group at 9 - 12 weeks of age. For the normal group vs. non-GVHD group and normal group vs. GVHD group, P = 0.008 and 0.016 at 9 weeks of age, P = 0.008 and 0.008 at 10 weeks of age, P = 0.016 and 0.024 at 11 weeks of age, and P = 0.008 and 0.008 at 12 weeks of age, respectively. Furthermore, the amount of TS in the GVHD group at 12 weeks of age was 1.65 ± 1.01, which was significantly lower compared to 3.70 ± 0.33 in the non-GVHD group. Also, the non-GVHD group vs. GVHD group at that time had P = 0.008.

[0178] (Tear film break-up time) Fig. 19(A) is a graph measuring the relationship between the age of mice and the tear film break-up time (TFBUT), and Fig. 19(B) is a table showing the progress of the tear film break-up time by group (green: normal group, blue: non-GVHD group, red: GVHD group). The significant difference was determined by the Mann-Whitney U test with n = 5 per group and P < 0.05 considered significant. TFBUT was evaluated in the right eye.

[0179] No difference in TFBUT was observed among the normal group, non-GVHD group, and GVHD group before BMT (8 weeks of age). However, TFBUT significantly decreased in the GVHD group when compared to the normal group at 10 - 12 weeks of age (P = 0.024, 0.008, and 0.008 at 10 weeks, 11 weeks, and 12 weeks of age, respectively). TFBUT also decreased in the non-GVHD group when compared to the normal group at 11 weeks of age (normal group vs. non-GVHD group was 5.80 ± 0.84 vs. 4.00 ± 0.71, and P = 0.024). Furthermore, when compared to the post-BMT groups, the GVHD group had a significantly shorter TFBUT than the non-GVHD group at 11 weeks and 12 weeks of age (P = 0.040 and 0.008 at 11 weeks and 12 weeks of age, respectively).

[0180] Figure 20 shows a photograph of a continuous tear film stained with a fluorescein solution. In this way, disruption of the tear layer was observed. The upper part of Figure 20 is an example of the GVHD group in which the tear layer ruptured in 3 seconds (TFBUT = 3 seconds), and the lower part of Figure 20 is an example of the normal group in which the tear film stabilized in 3 seconds and disintegrated in 6 seconds (TFBUT = 6 seconds). The photograph is of the right eye of a 12-week-old female BALB / c mouse. These results indicate that the smart eye camera can evaluate the continuous TFBUT of the GVHD group DED mouse model.

[0181] (Continuous corneal fluorescein score) Figure 21(A) is a graph measuring the relationship between the age of the mouse and the continuous corneal fluorescein score (CFS), and Figure 21(B) is a table regarding the progress of CFS by group (green: normal group, blue: non-GVHD group, red: GVHD group). The significant difference was determined by the Mann-Whitney U test with n = 5 per group and P < 0.05 considered significant. CFS was evaluated in the right eye.

[0182] When compared with the normal groups at 9 weeks, 11 weeks, and 12 weeks of age, as shown in Figure 21(B), significant differences were observed in the GVHD group (P = 0.008, 0.008, and 0.032, respectively). The non-GVHD group showed a tendency of higher CFS than the normal group, and the GVHD group showed a tendency of higher CFS than the non-GVHD group, but this did not reach statistical significance (all P > 0.05). From this result, it was confirmed that in the DED mouse model of the GVHD group, continuous CFS can also be evaluated with the smart eye camera.

[0183] (Comparison with existing devices) Figure 22 shows the results of comparing the measurement results of TFBUT and CFS for the smart eye camera and existing technologies. (A) is a graph for TFBUT, (B) is a graph for CFS, and green: normal group, blue: non-GVHD group, red: GVHD group. Figure 21(C) is a table summarizing these. The significant difference was determined by the Mann-Whitney U test with n = 5 per group and P < 0.05 considered significant.

[0184] In each graph, two bars are side by side. From these results, it can be seen that there is no significant difference in the use of the existing device and the smart eye camera between the normal non-GVHD group and the GVHD group. And for TFBUT, there was no significant difference among the normal control group, non-GVHD group, and GVHD group between the results obtained with the smart eye camera and those obtained with the portable slit lamp microscope (existing device) (0.50, 0.99, and 0.99 respectively, all P>0.05). Similarly, for CFS, no significant difference was observed among the normal control group, non-GVHD group, and GVHD group between the results obtained with the smart eye camera and the existing slit lamp microscope (P = 0.99, 0.75, and 0.50 respectively, all P>0.05).

[0185] Figure 23 is a graph showing the correlation between the smart eye camera and the existing device. (A) is a graph for TFBUT, and (B) is a graph for CFS. In each graph, the Y-axis shows the numerical values evaluated by the smart eye camera, and the X-axis shows the evaluation by the existing device. n = 15. For TFBUT (R = 0.868, 95%, CI: 0.656~ 0.953) and CFS (R = 0.934, 95%, CI: 0.823~0.976), a high correlation was observed.

[0186] From the results of Figure 23(A), the TFBUT of the smart eye camera and the existing device was r = 0.871 with P<0.001, showing a significant correlation. Also, from the results of Figure 23(B), the CFS of the smart eye camera and the existing device was also P<0.001, r = 0.941, showing a significant correlation. These results indicate that the smart eye camera has equivalent quality in obtaining the ocular surface phenotype of the mouse model compared to the existing device.

[0187] (Summary) From the above verification examples, it was possible to verify that the mobile communication terminal device 10 (i.e., the smart eye camera) equipped with the close-up photographing device 20A of the first embodiment is applicable in the DED mouse model. This model is characterized by weight loss, shortening of TS, and exacerbation of corneal epitheliitis, which was reflected in CFS. FIG. 24 is a graph that organizes them. From the results, the TFBUT of the GVHD group in the DED mouse model was decreased compared with the normal group and the non-GVHD group. As shown in FIG. 24, this indicates that the trend of TFBUT is similar to the trend of TS and opposite to the trend of CFS.

[0188] As shown in FIG. 22, there was no difference in the observation results between the smart eye camera and the existing device for both TFBUT and CFS. Also, as can be seen from the correlation analysis shown in FIG. 23, the results by the smart eye camera and the existing device showed a significantly high correlation in both TFBUT and CFS. These mean that the results obtained by photographing with the smart eye camera have the same quality as the results obtained by photographing with the existing device. Furthermore, although not shown, similar results have been obtained by different observers.

[0189] [Verification Example 2] In this verification example 2, using the mobile communication terminal device 10 (smart eye camera) equipped with the close-up photographing device 20A having the above configuration, TFBUT measurement, which is the diagnostic criterion for dry eye, was performed on a total of 42 dry eye patients and normal patients without dry eye, and the left-right difference, photographing seconds, etc. were also verified. The results are shown in FIG. 25. The TFBUT of the dry eye patient group (represented by "DED+") was 3.64 ± 1.81 seconds, and the TFBUT of the normal patients without dry eye (represented by "DED-") was 6.50 ± 0.71 seconds. Statistically significantly, the dry eye group had a shorter TFBUT result. The significant difference was determined by the Mann-Whitney U test with P < 0.05 considered significant. From this, it can be said that the smart eye camera can evaluate eye findings such as TFBUT.

[0190] As described above, it has been confirmed that the mobile communication terminal device 10 (i.e., the smart eye camera) equipped with the close-up photographing device 20A of the present embodiment can evaluate continuous eye phenotypes such as TFBUT and CFS.

[0191] (Method for acquiring diagnostic knowledge corresponding to DED) When actually estimating the state of DED in the eye E to be examined, it is necessary to previously store in the extraction knowledge storage unit 332 the extraction knowledge for extracting the diagnosable frame image that can be used for the diagnosis of DED, and to store in the diagnostic knowledge storage unit 334 the diagnostic knowledge corresponding to DED.

[0192] Here, in order to diagnose DED in the eye E to be examined, in order to score CFS, blue light is irradiated as observation light on the tissue to be observed (cornea, conjunctiva, ocular surface), and at least a part of the tissue to be observed is reflected. It is necessary to extract a well-focused frame image in which the light RL is also reflected as a diagnosable frame image. For this reason, in this case, it is necessary to previously store in the extraction knowledge storage unit 332 the extraction knowledge for extracting the frame image that satisfies the above conditions.

[0193] In addition, when estimating the state of DED in the eye E to be examined, it is necessary to estimate the state of DED while measuring continuous CFS, TS, and TFBUT based on the change over time of the diagnosable frame image. When measuring these, it is necessary to observe the secretion state of tears and the change state over time of the tear film layer on the ocular surface, corneal epithelium, and conjunctival epithelium of the eye E to be examined. For this reason, when estimating the state of DED in the eye E to be examined, it is difficult to estimate the state based on a single still image frame as in the above disease examples.

[0194] Therefore, when estimating the state of DED in the eye E to be examined, in step Sa3 of the diagnostic knowledge acquisition process (Fig. 14), the diagnosable frame extraction unit 352 extracts, from the frame images classified into the "Diagnosable" class, the frame image captured at the earliest timing as the diagnosable frame image corresponding to "0" seconds, and stores the diagnosable frame image in the ROM / RAM 320 in association with the time information (0 seconds). At this time, the diagnosable frame extraction unit 352 also extracts, from the frame images (for example, the frame images captured at the timings 3 seconds and 6 seconds after) captured at the timing after a predetermined time has elapsed from the diagnosable frame image corresponding to "0" seconds, the frame images classified into the "Diagnosable" class as the diagnosable frame images at the corresponding timings, and stores each diagnosable frame image in the ROM / RAM 320 in association with the corresponding time information (for example, 3 seconds, 6 seconds, etc.). Then, in step Sa4, the diagnostic knowledge acquisition unit 353 is configured to transmit the time information in association with each diagnosable frame image stored in the ROM / RAM 320 to the annotation terminal device 40.

[0195] When receiving the diagnosable frame images associated with the time information from the diagnostic support server device 30 in this way, the annotation terminal device 40 displays the diagnosable frame images in time series while associating each diagnosable frame image with the time information, for example, the diagnosable frame image at 0 seconds, the diagnosable frame image at the 3rd second, the diagnosable frame image at the 6th second, and displays the diagnosable frame images on the monitor so that the doctor can confirm the state of the tears and the change state of the tear film in the eye E to be examined.

[0196] Then, based on the diagnosable frame images displayed in association with the time information, the physician measures the continuous CFS, TS, and TFBUT, and inputs the diagnostic results such as the presence or absence of DED and its severity. The annotation terminal device 40 arranges each diagnosable frame image in time series based on the time information to generate a change image (a kind of moving image) composed of a plurality of diagnosable frame images. Then, while displaying the change image, the annotation terminal device 40 causes the physician serving as the annotator to diagnose DED, and creates teacher data by tagging the change image with the diagnostic result information, and returns it to the diagnostic support server device 30. As a result, teacher data related to the change image is stored in the teacher data storage unit 333 of the diagnostic support server device 30 (step Sa6). At this time, the physician serving as the annotator may measure the TFBUT based on the change image and diagnose that DED has occurred if the measured value of the TFBUT is equal to or less than a predetermined threshold (for example, 5 seconds or less).

[0197] Each time the moving image data is uploaded from the mobile communication terminal device 10, the above processing is repeated. When the number of teacher data related to the change image stored in the teacher data storage unit 333 reaches α or more, the diagnostic knowledge acquisition unit 353 acquires diagnostic knowledge corresponding to DED based on the teacher data stored in the teacher data storage unit 333 (step Sa8), and stores it in the diagnostic knowledge storage unit 334 (step Sa9). At this time, the diagnostic knowledge acquisition unit 353 may execute at least one of machine learning and data mining based on the change image included in the teacher data and the diagnostic result information to acquire diagnostic knowledge corresponding to DED. Note that the generation timing of the change image is not limited to the above generation timing, and can be generated by the diagnostic knowledge acquisition unit 353 when transmitting the diagnosable frame image and the time information to the annotation terminal device 40. Also, in the diagnosable frame image extraction process, the diagnosable frame extraction unit 352 may be configured to generate a change image.

[0198] On the other hand, when moving image data is uploaded from the mobile communication terminal device 10 in a state where diagnostic knowledge corresponding to DED is stored in the diagnostic knowledge storage unit 334, diagnostic support information is basically generated by the same diagnostic support information generation process as in FIG. 15 in the diagnostic processing unit 350 (step Sb7), and is to be distributed to the corresponding mobile communication terminal device 10 (step Sb8).

[0199] However, in this case, it is necessary to estimate the state of DED in the eye E to be examined using a change image similar to the change image obtained in the above-described diagnostic knowledge acquisition process (for example, a change image composed of diagnosable frame images at 0 seconds, 3 seconds, 6 seconds, etc.). For this reason, in this case, the diagnosable frame extraction unit 352 extracts, in step Sb3, diagnosable frame images obtained at the same timing as the timing at which the diagnosable frame images were extracted in the diagnostic knowledge acquisition process (for example, diagnosable frame images obtained at each timing of 0 seconds, 3 seconds, 6 seconds), and stores them in the ROM / RAM 320 while associating them with time information.

[0200] Also, in this case, the health state estimation processing unit 354 generates a change image by arranging the diagnosable frame images stored in the ROM / RAM 320 in time series based on time information in the health state estimation processing of step Sb5, and estimates the state of DED in the eye E to be examined based on the change image and the diagnostic knowledge of DED stored in the diagnostic knowledge storage unit 334. Note that in this case, it is optional whether the health state estimation processing unit 354 weights the estimation result based on the probability corresponding to the "Diagnosable" class and estimates the most likely state of DED. For example, when weighting is performed, when extracting the diagnosable frame image in step Sb3, as the diagnosable frame image for generating the change image, for example, 5 corresponding to the "0 second timing", 5 corresponding to the "3 second timing", 5 corresponding to the "6 second timing", and so on, a plurality of sets are extracted. Then, while the diagnostic processing unit 350 generates a change image for each set, the state of DED is estimated for each change image. Then, the estimation result is weighted according to the probability corresponding to the "Diagnosable" class for each change image, and the most likely state of DED in the eye E to be examined may be estimated. Note that the method of calculating the probability that each change image corresponds to the "Diagnosable" class is arbitrary in this case. For example, the average value of the probabilities corresponding to the "Diagnosable" class of the diagnosable frame images constituting each change image may be used.

[0201] Then, in step Sb6, the diagnostic support information generation unit 355 generates diagnostic support information including the changed image and the DED estimation result while presenting the changed image generated in step Sb5 as a presentation frame image (step Sb7), and distributes it to the corresponding mobile communication terminal device 10 (step Sb8). In this modification example, the case of generating a changed image while extracting diagnostic frame images corresponding to 0 seconds, 3 seconds, and 6 seconds respectively has been described as an example. However, in the moving image data uploaded from the smart eye camera, the first diagnostic frame image is set to 0 seconds, and all diagnostic frame images included in the moving images taken in the subsequent 10 seconds are extracted, and they are all connected to generate a changed image, and a method of generating a changed image (a kind of moving image) for 10 seconds may be adopted.

[0202] [A7.6] Modification Example 6 In the above first embodiment and each modification example, a configuration is adopted in which the state of one disease (for example, cataract, etc.) in the anterior segment of the eye to be examined E is estimated based on the moving image taken by the mobile communication terminal device 10. However, in actual patients, for example, cases where a patient suffers from multiple diseases simultaneously, such as cataract and epidemic keratoconjunctivitis, are also assumed. Therefore, it is desirable to estimate the states of multiple diseases at once and make them available by taking a single moving image.

[0203] Therefore, in this modified example, extraction knowledge for a plurality of diseases is stored in advance in the extraction knowledge storage unit 332, and diagnostic knowledge for a plurality of diseases is stored in the diagnostic knowledge storage unit 334. When moving image data is uploaded from the mobile communication terminal device 10, the presence or absence of the onset of a disease in the eye E to be examined and its severity are estimated for each disease based on the moving image data, and diagnostic support information including the estimation result for each disease is generated (step Sb7), and the configuration is adopted to distribute it to the corresponding mobile communication terminal device 10 (step Sb8). As methods for observing and photographing the anterior segment of the eye E to be examined, as described above, (Method 1) The slit light forming member 61 and the convex lens member 93 are attached, and the proximity photographing device 20A in a state where the color filter member 97 is removed is used to irradiate the eye E to be examined with the slit light SL as the observation light for observation and photographing, (Method 2) Only the convex lens member 93 is attached, and the eye E to be examined is irradiated with white diffused light as the observation light for observation and photographing, (Method 3) There are three methods, namely, the proximity photographing device 20A in a state where the color filter member 97 and the convex lens member 93 are attached irradiates the eye E to be examined with blue light as the observation light for observation and photographing. In this modified example, the moving image photographed by any method is used.

[0204] At this time, the diagnosis processing unit 350 basically generates diagnostic support information by executing diagnostic support information generation processing similar to that in FIG. 15. However, in this modified example, since it is necessary to be able to estimate the states related to a plurality of diseases at once, the following method is adopted.

[0205] First, the diagnosable frame extraction unit 352 extracts diagnosable frame images for each disease based on each frame image included in the moving image data and extraction knowledge corresponding to each disease in the diagnosable frame image extraction process of step Sb3. At this time, the diagnosable frame extraction unit 352 labels the in-focus tissues in each frame image included in the moving image data, and extracts frame images in which the tissues where the target disease develops are in focus. Then, the diagnosable frame extraction unit 352 extracts diagnosable frame images for each disease from the extracted frame images based on the extraction knowledge. At this time, the diagnosable frame extraction unit 352 calculates the probability corresponding to the "Diagnosable" class for each of the extracted frame images, and associates the calculated probability with the corresponding disease name and stores it in the ROM / RAM 320 in association with the diagnosable frame image.

[0206] Then, in this modified example, the health state estimation processing unit 354 reads out the diagnostic knowledge corresponding to the relevant disease from the diagnostic knowledge storage unit 334 based on the disease name stored in association with the diagnosable frame image extracted in step Sb3 (step Sb4), and estimates the state for each disease (step Sb5). At this time, the health state estimation processing unit 354 estimates the state of the disease corresponding to each diagnosable frame image stored in the ROM / RAM 320. Then, while weighting by the probability corresponding to the "Diagnosable" class stored in association with the diagnosable frame image for the estimation result, the most likely state of each disease in the eye E to be examined is estimated. Note that the processing for estimating the most likely state of each disease is the same as that in the first embodiment above.

[0207] Also, in step Sb6, the diagnostic support information generation unit 355 extracts the presentation frame images for each disease, and generates diagnostic support information while associating the estimated results for each disease with the presentation frame images (step Sb7). Then, the diagnostic support information distribution unit 356 distributes the thus generated diagnostic support information to the corresponding mobile communication terminal device 10 (step Sb8), and ends the process. In this case, the method for extracting the presentation frame images corresponding to each disease in step Sb6 is the same as that of the first embodiment except that the diagnostic frame image with the largest area of the pixel region in which the tissue affected by the target disease appears is used. Also, in this case, the format of the diagnostic support information generated by the diagnostic support information generation unit 355 in step Sb7 is arbitrary. For example, it may be configured to generate in the form of a list associating the disease name, information such as the estimated value of the presence or absence and severity of the onset of the disease, treatment methods, necessity of surgery, etc. with the presentation frame images.

[0208] Also, in this modification example as well, basically, diagnostic knowledge corresponding to each disease is acquired by the same diagnostic knowledge acquisition process as shown in FIG. 14 and stored in the diagnostic knowledge storage unit 334. As acquisition methods in this case, the following two methods can be adopted. (Acquisition Method 1) This method is a method of acquiring the corresponding diagnostic knowledge by executing the same process as in the first embodiment (FIG. 14) for each disease. (Acquisition Method 2) This method is a method of acquiring diagnostic knowledge corresponding to a plurality of diseases at once based on the moving image data uploaded from the mobile communication terminal device 10. However, regarding Acquisition Method 1, since it is the same as the first embodiment except that the target disease is other than cataract, only Acquisition Method 2 will be described below.

[0209] (Acquisition Method 2) When adopting this method, in the diagnostic frame image extraction process of step Sa3 in FIG. 14, the diagnostic frame extraction unit 352 extracts the diagnostic frame images for each disease by the same method as above, associates the corresponding disease name with the diagnostic frame images, and stores them in the ROM / RAM 320.

[0210] Next, in step Sa4, the diagnostic knowledge acquisition unit 353 transmits the disease names stored in the ROM / RAM 320 to the annotation terminal device 40 while associating them with the diagnosable frame images.

[0211] Upon receiving the diagnosable frame images transmitted in this manner, the annotation terminal device 40 causes the monitor to display the diagnosable frame images received from the diagnostic support server device 30 while associating them with the disease names. In this state, the doctor serving as the annotator diagnoses the conditions of each disease based on the diagnosable frame images for each disease displayed on the monitor and inputs the diagnostic results. Note that the display form at this time is arbitrary. For example, the display area of the monitor may be divided into a plurality of parts, and (a) the disease name, (b) the corresponding diagnosable frame image, and (c) a GUI for inputting the diagnostic results regarding the disease may be associated and displayed in each area.

[0212] Then, when the doctor makes a diagnosis for each disease based on the diagnosable frame images and disease names displayed in each area and inputs the diagnostic results, the annotation terminal device 40 tags the diagnosable frame images of the corresponding diseases with the diagnostic result information for each disease corresponding to the input results, thereby creating teacher data corresponding to each disease, and transmits the created teacher data to the diagnostic support server device 30 while associating it with the corresponding disease name.

[0213] On the other hand, in the diagnostic support server device 30, when the communication control unit 310 receives the teacher data transmitted from the annotation terminal device 40 (step Sa5), it stores the teacher data in the teacher data storage unit 333 while associating it with the disease name (step Sa6).

[0214] The above processing is repeated each time moving image data is uploaded from the mobile communication terminal device 10, and teacher data for each disease is accumulated in the teacher data storage unit 333.

[0215] When the number of teacher data corresponding to each disease stored in the teacher data storage unit 333 reaches α or more, the determination in step Sa7 changes to "Yes", and the diagnostic knowledge acquisition unit 353 acquires diagnostic knowledge corresponding to each disease based on the teacher data for each disease stored in the teacher data storage unit 333 (step Sa8), and stores it in the diagnostic knowledge storage unit 334 in association with the disease name (step Sa9). In steps Sa7 and Sa8, the number of teacher data stored in the teacher data storage unit 333 may be compared with α for each disease, and the diagnostic knowledge regarding the disease for which the number of teacher data has reached α or more may be acquired (step Sa8). When there are a plurality of diseases for which the number of teacher data has reached α or more, the diagnostic knowledge acquisition process may be repeated for each disease (step Sa8) so as to acquire the diagnostic knowledge corresponding to the plurality of diseases at once. Then, in step Sa9, the diagnostic knowledge acquisition unit 353 may store the diagnostic knowledge for each disease acquired in step Sa8 in the diagnostic knowledge storage unit 334 in association with the disease name of the corresponding disease.

[0216] In addition, when the test eye E is photographed from the front by the smart eye camera, the tissues of the cornea, anterior chamber, and lens will be imaged overlapping each other. For example, when irradiating slit light SL as observation light, by making the slit light SL enter the test eye E from an oblique direction, while cutting the test eye E in the cross-sectional direction with the slit light SL, an image including three-dimensional information of the test eye E can be photographed. In this case, by adopting a configuration in which multi-labeling is performed for each pixel, it is possible to specify which tissue is imaged in which pixel. Therefore, regarding a plurality of diseases occurring in a plurality of tissues that appear overlapping when photographed from the front, the state can be estimated at once with the same configuration, diagnostic support information can be generated, and it can be distributed to the corresponding mobile communication terminal device 10.

[0217] [A7.7] Variant 7 In the above-described embodiments and each modification example, a configuration for estimating the disease state in the eye E to be examined is adopted for the human eye. However, the diagnostic support device, diagnostic support system, and program of the present invention are applicable to animals other than humans. As described above, by using the smart eye camera, it is possible to observe and photograph the eyes of mice and other animals in detail. Therefore, while using the eye of an animal other than a human as the eye E to be examined, by photographing a moving image of the eye, it is possible to estimate the health state of the animal's eye in the same manner as for a human, generate diagnostic support information, and utilize it. Also in this case, except that the subject to be photographed is an animal other than a human, it is the same as the above-described embodiments and modification examples, so details are omitted.

[0218] [A7.8] Modification Example 8 In the above-described embodiments and each modification example, a method of estimating the health state of the eye E to be examined using only the moving image photographed by the smart eye camera is adopted. However, since the smart eye camera is realized by the mobile communication terminal device 10 equipped with the close-up photographing device 20, it can be used to estimate the health state of the eye E to be examined regardless of the country or region, and it is possible to acquire position information (for example, latitude and longitude information) indicating the photographing location of the moving image. Therefore, it is also possible to perform the following applications by utilizing the property peculiar to this smart eye camera (that is, the property of being able to acquire the position information of the photographing location). Note that since the position information can be acquired by using the GPS (Global Positioning System) function installed in the smartphone used as the existing mobile communication terminal device 10, details regarding the acquisition method are omitted.

[0219] <Application Method 1> For example, by adopting the method of this modification example, it is possible to identify in which region and what kind of eye disease has a high incidence rate, and according to the region where the moving image is captured, an application form that reflects the estimated result of the health state can be realized. In this case, the position information of the shooting location is uploaded to the diagnostic support server device 30 together with the captured image of the smart eye camera. Also, in this case, the diagnostic support server device 30 transmits the diagnostic frame image extracted from the captured image and the position information to the annotation terminal device 40 while associating them, and the annotation terminal device 40 displays the position information together with the diagnostic frame image (for example, plots the position on a map, etc.). Then, when a doctor acting as an annotator makes a diagnosis based on the displayed diagnostic frame image and position information, the annotation terminal device 40 tags the corresponding diagnostic frame image with the diagnostic result information and the position information to create teacher data. On the other hand, the diagnostic knowledge acquisition unit 353 acquires diagnostic knowledge based on the teacher data tagged with position information. Note that other processes at this time are the same as those in FIG. 14, so the details are omitted.

[0220] Also, when generating diagnostic support information by this method, a configuration is adopted in which the position information is uploaded in association with the moving image captured by the smart eye camera. Then, when generating diagnostic support information in the diagnostic support server device 30, (i) the diagnostic frame image extracted from the uploaded moving image data, (ii) the position information associated with the moving image data, and (iii) the diagnostic knowledge acquired by the above method are used to estimate the health state of the eye to be examined E. By this method, it is possible to accurately estimate the state of the disease with the highest incidence frequency in the shooting region, and it is also possible to estimate the state of the disease with high accuracy regarding endemic diseases peculiar to the shooting region. For example, diseases such as onchocerciasis hardly exist in patients in Japan, but are common diseases in the African region south of the Sahara. Therefore, by estimating the state based on the position information and the diagnostic frame image, the state of this kind of endemic disease can be estimated with high accuracy.

[0221] <Application Method 2> In addition, for example, by adopting the method of this modification example, it can also be applied to identify areas with a high prevalence of various eye diseases and to grasp the epidemiological prevalence of various diseases. In this case, by creating a database while associating the estimated results of the health status with the location information and leaving a diagnosis history, it becomes possible to epidemiologically investigate which diseases have a high incidence in which areas. Note that in this case, the database of the diagnosis history may be configured to be provided in the diagnostic support server device 30, or may be configured to be provided in a computer system or the like owned by an epidemiological research institution related to eye diseases.

[0222] [B]Second Embodiment The above first embodiment is for estimating the disease state in the anterior eye tissue of the eye to be examined E by the diagnostic support server device 30 and distributing and using the diagnostic support information including the estimation result to the corresponding mobile communication terminal device 10. In contrast, this embodiment is for estimating the state of diseases (for example, hypertensive retinopathy, retinal thinning, glaucoma, diabetic retinopathy, retinal detachment, central serous chorioretinopathy, age-related macular degeneration, exudative retinopathy, central retinal artery occlusion, etc.) in the fundus tissue of the eye to be examined E and distributing the diagnostic support information including the estimation result to the mobile communication terminal device 10. Note that the diagnostic support system of this embodiment differs from the diagnostic support system 1 of the first embodiment only in the configuration of the proximity photographing device 20 and the contents of the extraction knowledge and diagnostic knowledge, and the other configurations are basically the same as those of the first embodiment. Therefore, unless otherwise specified, the diagnostic support system of this embodiment is assumed to be realized with the same configuration as that of the first embodiment.

[0223] [B1]Configuration of Proximity Photographing Device 20B Next, with reference to FIGS. 26 to 28, the configuration of the proximity photographing device 20B of this embodiment will be described. FIGS. 26 to 28 are diagrams showing the configuration of the proximity photographing device 20B of this embodiment, respectively. FIG. 26 is a perspective view of the proximity photographing device 20B, FIG. 27 is a front view of the proximity photographing device 20B of this embodiment, and FIG. 28 is an exploded configuration diagram of the proximity photographing device 20B of this embodiment.

[0224] The close-up photographing device 20B of this embodiment is configured to include a cylindrical member 180 for fundus tissue observation and photographing.

[0225] (Cylindrical member) As shown in FIGS. 26 to 28, the cylindrical member 180 is a member having a convex lens 183 at its tip. This cylindrical member 180 is also detachably attached to the mobile communication terminal device 10. The cylindrical member 180 is at least composed of a mounting portion 181 for mounting on the mobile communication terminal device 10, a cylindrical portion 182 (182a, 182b), a convex lens 183, an opening 184, a convex lens holding portion 185, and a mounting portion 186. With such a cylindrical member 180, the focal length with the fundus can be adjusted and maintained in an appropriate state. As a result, observation and photographing of the fundus can be appropriately performed.

[0226] At the opening 184 of the cylindrical member 180, a color filter (orange) 4 and a polarizing filter (horizontal polarization) 3 are arranged in this order at the optical path position of the light source light (white diffused light) emitted from the light source 92. With this configuration, the light source light emitted from the light source 92 changes to orange light when passing through the orange color filter 4 and is converted into linearly polarized light (for example, P-polarized light) by the polarizing filter 3. The observation light (linearly polarized light) obtained by converting the light source light in this way passes through the convex lens 183 and irradiates the fundus tissue of the eye E to be examined. On the other hand, a polarizing filter (vertical polarization) 2 is provided at the optical path position immediately before the reflected light RL (i.e., the return light), which is linearly polarized light reflected by the fundus tissue, reaches the photographing camera lens 91. Note that as long as one of the polarizing filters 2 and 3 is vertically polarized and the other is horizontally polarized, either one can be vertically polarized or horizontally polarized. With this configuration, a phase difference is generated between the observation light and the reflected light RL, interference between the observation light (forward path light) and the reflected light RL (return path light) inside the cylindrical member 180 is prevented, and observation and photographing of the fundus tissue (retina) by the reflected light RL can be performed cleanly.

[0227] The orange color filter 4 is a member for converting the observation light into light that can reach the fundus tissue as easily as possible.

[0228] The mounting portion 186 for mounting on the mobile communication terminal device 10 is configured to be slidably mounted on the mobile communication terminal device 10 from the side of the right side wall 85 as shown in FIG. 28. After mounting, in the cylindrical member 180, a color filter (orange) 4 and a polarizing filter 3 are provided in front of the light source 92, and another polarizing filter 2 is provided in front of the imaging camera lens 91.

[0229] In the example of the figure, the cylindrical portion 182 is configured by combining two cylindrical portions 182a and 182b, but it is not particularly limited. The convex lens 183 is mounted on the convex lens holding portion 185 in front of the cylindrical member 180, irradiates the fundus tissue with linearly polarized light as observation light, and condenses the reflected light RL of the observation light (linearly polarized light) in the fundus tissue onto the imaging camera lens 91 through the polarizing filter 2. Note that the cylindrical member 180 may be configured as a detachable attachment for the close-up photographing device 20A shown in FIGS. 2 to 7. When it is an attachment, the cylindrical member 180 is configured such that the color filter 4, the polarizing filters 3 and 2 are arranged in the positional relationship shown in FIGS. 26 to 28, and in the close-up photographing device 20A shown in FIGS. 2 to 7, the slit light forming member 61, the convex lens member 93 and the color filter member 97 are all removed, and the cylindrical member 180 may be detachably mounted on the housing 80.

[0230] [B2] Verification results regarding the diagnosis of diseases in the fundus tissue using a smart eye camera In December 2018, the inventor observed the fundus tissue of the eye to be examined E using the mobile communication terminal device 10 (that is, a smart eye camera) equipped with the close-up photographing device 20B having the cylindrical member 180 configured as shown in FIGS. 26 to 28. The cases were 41 eyes of the fundus (19 males and 22 females), and the presence or absence of optic nerve abnormalities and fundus diseases was evaluated.

[0231] An example of an image taken of the above-mentioned 41 eyes by the mobile communication terminal device 10 (smartphone) equipped with the close-up photographing device 20 shown in FIGS. 26 to 28 is shown in FIG. 29. In FIG. 29, (A) is a normal fundus, (B) is a hypertensive fundus, (C) is retinal thinning, and (D) is an image of enlargement of the optic disc pit (suspected glaucoma). Including the example of FIG. 29, the fundus tissue could be evaluated in 85% or more of the cases. The cases in which evaluation was impossible were due to opacity of the intermediate diaphanous body such as severe cataract, which could not be evaluated even by existing ophthalmic devices. From this result, it was confirmed that the smart eye camera equipped with the cylindrical member 180 having the configuration shown in FIGS. 26 to 28 can preferably observe and photograph the fundus of the eye.

[0232] FIG. 30 shows the correlation between doctors regarding the evaluation results of the hypertensive fundus evaluated by the mobile communication terminal device 10 equipped with the cylindrical member 180 having the configuration shown in FIGS. 26 to 28. There was no significant difference between the doctors. Also, the glaucoma diagnosis rates among the doctors were 25.7% and 18.9%, and there was no significant difference in this regard either. From the above results, it was confirmed that the smart eye camera equipped with the cylindrical member 180 shown in FIGS. 26 to 28 can observe and photograph the fundus tissue of the test eye E in detail without being inferior to existing ophthalmic devices.

[0233] When actually estimating the presence or absence and severity of various diseases that develop in the fundus tissue of the test eye E using the diagnostic support system of the present embodiment, it is necessary to attach the close-up photographing device 20B having the cylindrical member 180 shown in FIGS. 26 to 28 to the mobile communication terminal device 10, and while irradiating the fundus tissue of the test eye E with linearly polarized light as the observation light, capture a moving image including a diagnosable frame image that can be used for the diagnosis of the corresponding disease.

[0234] And in this case, the extraction knowledge storage unit 332 stores in advance extraction knowledge for extracting diagnosable frame images that can be used for diagnosing diseases that occur in fundus tissues, such as hypertensive retinopathy, retinal thinning, glaucoma, diabetic retinopathy, retinal detachment, central serous chorioretinopathy, age-related macular degeneration, exudative retinopathy, and central retinal artery occlusion. Note that the conditions for diagnosable frame images that can be used for diseases that occur in fundus tissues such as hypertensive retinopathy, retinal thinning, glaucoma, diabetic retinopathy, retinal detachment, central serous chorioretinopathy, age-related macular degeneration, exudative retinopathy, and central retinal artery occlusion are that the observation target tissue is irradiated with linearly polarized light as observation light, and at least a part of the observation target tissue is reflected together with the reflected light RL of the linearly polarized light in the observation target tissue and is a focused image. Therefore, a configuration may be adopted in which extraction knowledge for extracting frame images that satisfy the conditions is stored in the diagnosis knowledge storage unit 334 in advance.

[0235] Also, in this embodiment as well, the diagnosis processing unit 350 basically acquires diagnosis knowledge by executing diagnosis knowledge acquisition processing similar to that in FIG. 14 (step Sa8) and stores it in the diagnosis knowledge storage unit 334 (step Sa9). Then, while using the acquired diagnosis knowledge, the diagnosis processing unit 350 executes diagnosis support information generation processing similar to that in FIG. 15, estimates the state of the disease in the fundus tissue (step Sb5), generates diagnosis support information (step Sb7), and distributes it to the corresponding mobile communication terminal device 10 (step Sb8). Note that the processing at this time is the same as that in the first embodiment except that the target disease is a disease that occurs in the fundus tissue such as hypertensive retinopathy instead of cataract. Also, in this embodiment as well, a configuration may be adopted in which the states of a plurality of diseases are estimated at once for each disease in the same manner as in the above-described modification example 6, and diagnosis support information including the estimation results for each disease is generated and distributed. Also, in this embodiment as well, a configuration may be adopted in which the most likely state of the disease in the fundus tissue of the eye to be examined E is estimated while weighting the estimation results according to the probability that each diagnosable frame image belongs to the "Diagnosable" class. However, since this point is the same as that in the first embodiment, the details are omitted.

[0236] As described above, the diagnostic support system of the present embodiment extracts a diagnosable frame image from a moving image captured by the mobile communication terminal device 10 with the proximity imaging device 20B having the cylindrical member 180 attached thereto, acquires diagnostic knowledge corresponding to a disease that develops in the fundus tissue based on the diagnosable frame image, and based on the diagnostic knowledge and the moving image data uploaded from the mobile communication terminal device 10, estimates the presence or absence, severity, treatment method, necessity of surgery, etc. of a disease in the fundus tissue of the eye E to be examined. When the mobile communication terminal device 10 with the proximity imaging device 20B attached is used, not only paramedical personnel who are not used to taking ophthalmic diagnostic images but also laypersons such as patient families can easily take a moving image including one or more diagnosable frame images that can be used for diagnosing a disease that develops in the fundus tissue with a simple operation similar to that of a conventional smartphone or the like. Therefore, according to the configuration of the present embodiment, without using an expensive ophthalmic diagnostic imaging device, based on a moving image taken by a layperson, the state of a disease in the fundus tissue of the eye E to be examined can be appropriately estimated, diagnostic support information can be generated, and used. In addition, since the proximity imaging device 20B can be manufactured at an extremely low cost compared to an ophthalmic fundus imaging device, the state of a disease in the fundus tissue of the eye E to be examined can be estimated and diagnostic support information can be generated even in an area where the budget for an inspection device cannot be secured.

[0237] [C] Third Embodiment In the above-described first and second embodiments and their modified examples, according to the process shown in FIG. 14, the diagnosable frames included in the moving image captured by the smart eye camera are directly transmitted to the annotation terminal device 40, and a doctor serving as an annotator is made to perform a diagnosis, and by tagging the corresponding diagnosable frame image with the diagnostic result information, teacher data corresponding to each diagnosable frame image is created while acquiring diagnostic knowledge.

[0238] On the other hand, in the diagnostic support system 1 of the present embodiment, when the diagnostic support server device 30 transmits the frame image extracted from the moving image as a diagnosable frame image to the annotation terminal device 40, the diagnosable frame extraction unit 352 performs machine learning to cut out the area in which the eye tissue is reflected from the diagnosable frame image. Note that the diagnostic support system 1 of the present embodiment is basically realized by the same configuration as that of the first and second embodiments and the modified example. That is, also in the present embodiment, each component constituting the diagnostic support system 1 has the same functions as those in the first embodiment and the like and performs the same operations. Therefore, in the following, only the points different from those in the first embodiment and the like will be described.

[0239] [C1] Method for cutting out an area in which an eye tissue is reflected from a diagnosable frame image Next, with reference to FIG. 31, a method for the diagnostic support server device 30 in the diagnostic support system 1 of the present embodiment to cut out an image of an area in which an eye tissue is reflected from a diagnosable frame image (hereinafter, also referred to as a "region image") will be described. Note that FIG. 31 is a diagram for explaining the annotation work for cutting out the region image from the diagnosable frame image, and FIGS. (A) and (B) each show a diagram for explaining the annotation work for cutting out the area in which the eye tissue is reflected in the diagnosable frame image taken when white diffused light is irradiated as the observation light and when blue light is irradiated, respectively.

[0240] In this embodiment, first, a method is adopted in which a human performs annotation for cutting out an area in a diagnosable frame image where an eye tissue is captured. At this time, the diagnosable frame extraction unit 352 of the diagnostic support server device 30 extracts a diagnosable frame image that can be used for diagnosing various diseases from the moving image data uploaded from the mobile communication terminal device 10 (smart eye camera), and transmits the extracted diagnosable frame image to the annotation terminal device 40 and presents it to the annotator. Then, the annotator performs a cutting operation on the area in the diagnosable frame image where the eye tissue is captured. Note that it is desirable for a doctor to perform the annotation at this time. However, since an area where an eye tissue is captured in the diagnosable frame image can be discriminated even by a person who is not an ophthalmologist, any person may be used as the annotator.

[0241] At this time, the annotation terminal device 40 displays the diagnosable frame image sent from the diagnostic support server device 30. Then, the annotator uses the annotation tool installed in the annotation terminal device 40 to perform annotation by surrounding the area where each eye tissue is captured in the diagnosable frame image with a rectangular area as shown in FIG. 31.

[0242] Specifically, at this time, as illustrated in FIGS. 31(A) and (B), the annotator performs annotation by surrounding the "area where the cornea is reflected" with a blue rectangle and the "area where the entire eye is reflected" with a red rectangle in the diagnosable frame image. In the annotation, in this way, while tagging the diagnosable frame image using the data of the upper left and lower right coordinates of the obtained rectangular area, teacher data is created, and by performing at least one of machine learning and data mining, knowledge for cutting out the area image from the diagnosable frame image (hereinafter, also referred to as "cutting-out knowledge") is obtained. Specifically, at this time, in order to obtain the cutting-out knowledge, teacher data is created from a pair of the diagnosable frame image and the annotated rectangular area, and at least one of machine learning and data mining is executed. Note that the specific learning algorithm used for obtaining the cutting-out knowledge is arbitrary. For example, a configuration may be adopted in which teacher data is input to the Objective Detection algorithm and learning is performed so that the error becomes small. Although Objective Detection can theoretically estimate a plurality of regions for one image, since only one eye appears in the diagnosable frame, the one with the maximum probability may be selected. As a result, it becomes possible to uniquely cut out the regions of the cornea and the entire eye for one frame. Also, the cutting-out knowledge may be stored in the extraction knowledge storage unit 332 together with the extraction knowledge, for example, or a storage area for the cutting-out knowledge may be allocated in the storage device 330 and a cutting-out knowledge storage unit (not shown) may be provided. Further, in FIG. 31, the case of cutting out the cornea from a moving image captured using white diffused light and blue light as observation lights is illustrated, but this embodiment is not limited to being applicable only to the cornea and can be applied to any tissue of the eye, such as the conjunctiva, a part of the eyelid tissue, the iris, and the fundus tissue. Also, the observation lights that can be used in this embodiment are not limited to white diffused light and blue light and are also applicable to moving images using slit light SL or linearly polarized light as the observation light.In this case, the annotator may surround the area where the target tissue is reflected in the diagnosable frame image with a blue rectangle and surround the area where the entire eye is reflected with a red rectangle. In short, by specifying the range surrounded by the blue rectangle according to the target tissue, it becomes possible to cut out the area image where any tissue is reflected. When using linearly polarized light, the proximity photographing device 20B of the second embodiment is attached to the mobile communication terminal device 10 to photograph a moving image of the fundus tissue, and the annotator may surround, for example, the tissue such as the macula reflected in the diagnosable frame image with a blue rectangle and the entire image with a red rectangle as the area to be cut out in the fundus tissue.

[0243] [C2] Method for estimating and cutting out an area image where the tissue of the eye is reflected Next, with reference to FIG. 32, a method for estimating the area where the tissue of the eye is reflected in the diagnosable frame image and cutting out the area image will be described. Note that FIG. 32 is a diagram for explaining a method in which the diagnosable frame extraction unit 352 of the diagnostic support server device 30 of the present embodiment estimates the area where the tissue of the eye is reflected from the diagnosable frame image and cuts out the area image.

[0244] With the configuration of the present embodiment, the diagnosable frame extraction unit 352 uses the learned knowledge for cutting out to perform area detection on the diagnosable frame image, and estimates the area where the cornea is reflected (the yellow rectangular area part) and the area where the entire eye is reflected (the green rectangular area part) from each diagnosable frame image as shown in FIG. 32, and cuts out the area image. As a result, as illustrated in FIG. 32, it was found that the area where the cornea is reflected and the area where the entire eye is reflected can be estimated and cut out almost accurately. In actuality, when acquiring the diagnostic knowledge and generating the diagnostic support information described later, the area image is cut out as the smallest square including this area at the center in the diagnosable frame image.

[0245] [C3] Method for acquiring diagnostic knowledge When acquiring diagnostic knowledge in a state where the knowledge for cutting out has been obtained by the above method, in this embodiment as well, diagnostic knowledge will be obtained by basically the same processing as in FIG. 14.

[0246] However, in this embodiment, the diagnosable frame extraction unit 352 reads out the extraction knowledge and the knowledge for cutting out from the storage device 330 in step Sa2, and in step Sa3, after extracting the diagnosable frame image based on the extraction knowledge, based on the knowledge for cutting out, an area image in which the eye tissue is reflected in the diagnosable frame image is cut out as described above. Note that the diagnosable frame extraction unit 352 of this embodiment constitutes, for example, the "cutting-out means" of the present invention.

[0247] Then, in step Sa4, the diagnostic knowledge acquisition unit 353 transmits the area image cut out from the diagnosable frame image to the annotation terminal device 40. The annotation terminal device 40 displays the area image and causes a doctor serving as an annotator to input a diagnostic result based on the area image, creates teacher data tagged with diagnostic result information corresponding to the input content, and the diagnostic knowledge acquisition unit 353 acquires the created teacher data from the annotation terminal device 40 (step Sa5 "Yes"). Then, the diagnostic knowledge acquisition unit 353 stores the teacher data in the teacher data storage unit 333 (step Sa6), and when the number of stored teacher data reaches α or more (step Sa7 "Yes"), acquires diagnostic knowledge based on the stored teacher data (step Sa8), stores it in the diagnostic knowledge storage unit 334 (step Sa9), and ends the process.

[0248] [C4] Method for generating diagnostic support information In this way, when the diagnostic knowledge based on the region image cut out from the diagnosable frame image and the knowledge for cutting out are acquired, when moving image data is later uploaded from the mobile communication terminal device 10, in the diagnostic processing unit 350, basically by the same processing as in FIG. 15, diagnostic support information is generated (step Sb7 in FIG. 15) and distributed to the corresponding mobile communication terminal device 10 (step Sb8 in FIG. 15).

[0249] However, in the present embodiment, after the diagnosable frame extraction unit 352 extracts a diagnosable frame image from the moving image data uploaded from the smart eye camera in step Sb3, the region image is cut out from the extracted diagnosable frame image by the above method based on the knowledge for cutting out.

[0250] Then, the health state estimation processing unit 354 generates diagnostic support information basically by the same processing as in FIG. 15 based on the region image and the diagnostic knowledge. At this time, the health state estimation processing unit 354 estimates the health state of the eye E to be examined photographed by the smart eye camera (step Sb5) based on the region image cut out from the diagnosable frame image and the diagnostic knowledge in step Sb5, extracts a presentation frame image (step Sb6), generates diagnostic support information (step Sb7), distributes it to the mobile communication terminal device 10 that is the upload source of the moving image (step Sb8), and ends the processing. Note that the processing at this time is basically the same as that in the first embodiment, so the details are omitted. Also, in this case, the presentation frame image extracted in step Sb6 may be extracted by the same method as in the first embodiment, or only the region image of the eye cut out from the diagnosable frame image extracted as the presentation frame image may be used as the presentation frame image.

[0251] With the above configuration, the diagnostic support system 1 of the present embodiment can estimate the health state of the eye E to be examined with very high accuracy, generate and utilize diagnostic result information including a more accurate diagnostic result. In particular, when capturing a moving image of the eye E to be examined using a smart eye camera, even when the frame images with different zooms are included, the diagnostic frame images are normalized to the same size, so that the accuracy of machine learning during the acquisition of diagnostic knowledge can be improved, and the health state of the eye E to be examined can be estimated with high accuracy.

[0252] Using the diagnostic support system 1 of the present embodiment, while actually irradiating the eye E to be examined with each observation light of slit light, white diffused light, linearly polarized light, and blue light, a moving image of the eye E to be examined is captured by a smart eye camera, a region image is cut out from the diagnostic frame images included in the captured moving image, and based on the region image, diseases such as iritis and uveitis that occur in the anterior chamber tissue, chalazion and hordeolum that occur in the eyelid tissue, allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, corneal opacity, punctate superficial keratitis and corneal ulcer, DED, and diseases that occur in the fundus tissue are estimated. As a result, it was found that the diagnostic accuracy was significantly improved according to the method of the present embodiment.

[0253] In particular, in the diagnosis of DED, it is possible to perform the diagnosis of DED with very high accuracy while estimating the TFBUT with high accuracy. For example, it was found that it is also possible to diagnose various subtypes of DED, such as (a) spot break, (b) area break, (c) dimple break, (d) line break, (e) random break, etc.

[0254] As described above, according to the diagnostic support system 1 of the present embodiment, it is possible to acquire diagnostic knowledge while cutting out a region image in which the target tissue of the eye is reflected from the diagnosable frame image included in the moving image captured by the smart eye camera. Further, even when estimating (diagnosing) the health state of the subject eye E, a highly accurate estimation result can be obtained by using the region image, and highly practical and highly accurate diagnostic support information can be generated and used. In the configuration of the present embodiment, it is desirable to adopt a configuration in which the states of a plurality of diseases can be estimated and used at once by a single moving image capture as in Modification 6. In this case, the cut-out knowledge for a plurality of diseases and the diagnostic knowledge based on the region image are stored in advance in the storage device 330. When moving image data is uploaded from the smart eye camera, a region image for estimating the state of each disease is cut out for each disease, and based on the cut-out region image and the diagnostic readiness awareness for the corresponding disease, a configuration may be adopted in which the states of a plurality of diseases occurring in the subject eye E are estimated at once. Further, in this case, similar to Modification 6, a configuration may be adopted in which diagnostic support information describing the estimation results for each disease based on the region image in a list format is generated. Furthermore, in this case, similar to Modification 6, a configuration may be adopted in which diagnostic knowledge corresponding to a plurality of diseases is acquired at once. Also, in the configuration of the present embodiment, similar to Modification 4, a three-dimensional image may be constructed from the region image, and diagnostic knowledge may be acquired based on the three-dimensional image, or the health state of the subject eye E may be estimated based on the three-dimensional image formed by stacking the region images. In this case, a configuration may be adopted in which the region images cut out from each diagnosable frame image are stacked according to the focal length to generate a three-dimensional image. Since other points are the same as those in Modification 4, the details are omitted.

[0255] [D]Fourth Embodiment In each of the above embodiments and modifications, a configuration is adopted in which diagnostic knowledge is acquired based only on the moving image captured by the smart eye camera.

[0256] In contrast, in the diagnostic support system 1 of the present embodiment, not only the smart eye camera but also information parameter values obtained from various ophthalmic devices installed in an ophthalmic facility (not shown) (specifically, measurement values such as the length, angle, and area of a part of the tissue in the eye E to be examined) are associated with the diagnostic frame images, and teacher data is created. By performing at least one of machine learning and data mining based on the teacher data, a method for acquiring diagnostic knowledge is adopted. Then, in the present embodiment, a configuration is adopted in which the value of the information parameter in the eye E to be examined is estimated from the moving image captured by the smart eye camera using the acquired diagnostic knowledge. Note that the diagnostic support system 1 of the present embodiment is basically realized by the same configuration as that of the first and second embodiments and the modification examples. That is, also in the present embodiment, each component constituting the diagnostic support system 1 has the same functions as those in the first embodiment and the like and performs the same operations. Therefore, in the following, only the points different from those in the first embodiment and the like will be described.

[0257] In the present embodiment, the annotation terminal device 40 is installed in an ophthalmic facility (not shown), and a doctor serving as an annotator uses the following various ophthalmic devices installed in the ophthalmic facility to acquire various information parameter values regarding the eye E to be examined. A method is adopted. Then, in the diagnostic support system 1 of the present embodiment, when acquiring diagnostic knowledge, teacher data is created while associating the information parameters acquired by these ophthalmic devices with the diagnostic frame images, and the diagnostic knowledge is acquired using the teacher data. Note that the diagnostic frame images used for acquiring diagnostic knowledge in the present embodiment may be directly used as those extracted from the moving image as in the first embodiment, or region images cut out from the diagnostic frame images may be used as in the third embodiment.

[0258] [D1] Regarding the types of ophthalmic devices available in the present embodiment and the information parameter values that can be acquired by each device Regarding specifically which ophthalmic device to use and which information parameter values regarding the eye E to be examined to obtain while creating teacher data in this embodiment, it is arbitrary. For example, it is possible to configure to obtain the following information parameter values using the following ophthalmic devices and create teacher data while associating the information parameter values with diagnosable frame images.

[0259] (1) Anterior segment OCT (Optical Coherence Tomography) When using anterior segment OCT, it is possible to obtain information parameter values such as the corneal thickness, anterior chamber depth, corneal radius of curvature, angle of the anterior chamber, astigmatism degree, pupil diameter, and higher-order aberration of the eye E to be examined. Therefore, teacher data associating these information parameter values with diagnosable frame images can be created, diagnostic knowledge can be acquired, and by using the diagnostic knowledge, it becomes possible to estimate these information parameter values from the smart eye camera image.

[0260] (2) Auto refractometer When using an auto refractometer, it is possible to obtain information parameter values such as the refraction, corneal radius of curvature, myopia, hyperopia, and astigmatism degree of the eye E to be examined. Therefore, teacher data associating these information parameter values with diagnosable frame images can be created, diagnostic knowledge can be acquired, and by using the diagnostic knowledge, it becomes possible to estimate these information parameter values from the smart eye camera image.

[0261] (3) Non-contact tonometer When using a non-contact tonometer, it is possible to obtain the information parameter value of the intraocular pressure of the eye E to be examined. Therefore, teacher data associating the parameter value with a diagnosable frame image can be created, diagnostic knowledge can be acquired, and by using the diagnostic knowledge, it becomes possible to estimate the intraocular pressure of the eye E to be examined from the smart eye camera image.

[0262] (4) Specular microscope When using a specular microscope, information parameter values regarding the corneal endothelial cells of the eye E to be examined can be obtained. Therefore, teacher data in which the information parameter values are associated with diagnostic frame images is created, diagnostic knowledge is acquired, and by using the diagnostic knowledge, it becomes possible to estimate these information parameter values from a smart eye camera image.

[0263] (5) Fundus camera When using a fundus camera, a fundus photograph of the eye E to be examined can be obtained. Therefore, teacher data is created while tagging the diagnostic frame image with diagnostic result information corresponding to the diagnostic results of a doctor based on fundus photographs corresponding to various fundus diseases, diagnostic knowledge is acquired, and by using the diagnostic knowledge, it becomes possible to accurately estimate the states of various diseases that have developed in the fundus from a smart eye camera image. Regarding the fundus photograph, the diagnostic frame image obtained by the method of the second embodiment may be directly used for creating teacher data, or a region image may be cut out and used for creating teacher data.

[0264] (6) OCT or optical coherence tomograph for axial length measurement When using OCT or an optical coherence tomograph for axial length measurement, information parameter values such as the retinal thickness, optic disc diameter, and choroidal blood vessels of the eye E to be examined can be measured. Therefore, teacher data in which these information parameter values are associated with diagnostic frame images is created, diagnostic knowledge is acquired, and by using the diagnostic knowledge, it becomes possible to estimate these information parameter values from a smart eye camera image. In this case as well, it is possible to create teacher data based on a region image.

[0265] (7) Ultrasonic tomography device When using an ultrasonic tomography device, an information parameter value of the axial length of the eye E to be examined can be obtained. Therefore, teacher data in which the information parameter value is associated with a diagnostic frame image is created, diagnostic knowledge is acquired, and by using the diagnostic knowledge, it becomes possible to estimate the axial length from a smart eye camera image.

[0266] (8) Perimeter When using a perimeter, information parameter values regarding the static and dynamic visual fields of the eye E to be examined can be obtained. Therefore, teacher data in which these information parameter values are associated with diagnosable frame images is created to acquire diagnostic knowledge, and by using this diagnostic knowledge, it becomes possible to estimate these information parameter values from the smart eye camera images.

[0267] (9) Accommodation function analysis device When using an accommodation function analysis device, information parameter values regarding the accommodation function of the eye E to be examined can be obtained. Therefore, teacher data in which these information parameter values are associated with diagnosable frame images is created to acquire diagnostic knowledge, and by using this diagnostic knowledge, it becomes possible to estimate information parameter values regarding the accommodation function of the eye from the smart eye camera images.

[0268] (10) Flicker meter When using a flicker meter, information parameter values of the flicker value of the eye E to be examined can be obtained. Therefore, teacher data in which these information parameter values are associated with diagnosable frame images is created to acquire diagnostic knowledge, and by using this diagnostic knowledge, it becomes possible to estimate the flicker value of the eye E to be examined from the smart eye camera images.

[0269] [D2] Method for acquiring diagnostic knowledge In the diagnostic support system 1 of the present embodiment, diagnostic knowledge is basically acquired by the same processing as in FIG. 14.

[0270] However, when estimating information parameter values in the eye E to be examined by the method of the present embodiment, additional conditions are required for the above three conditions. Therefore, in step Sa3, the diagnosable frame extraction unit 352 adopts a method of extracting a frame image that satisfies the additional conditions as a diagnosable frame image. The details of this additional condition will be described later.

[0271] Also, in this embodiment, when a diagnosable frame image is transmitted from the diagnostic support server device 30 in step Sa4 of FIG. 14 to the annotation terminal device 40, the annotation terminal device 40 displays the diagnosable frame image and also displays a GUI for allowing an annotator to input a diagnostic result based on the diagnosable frame image and information parameter values acquired by the ophthalmic device. Then, when a doctor serving as an annotator inputs (a) a diagnostic result based on the diagnosable frame image and (b) information parameter values acquired by the ophthalmic device, the annotation terminal device 40 tags (a) diagnostic result information corresponding to the input diagnostic result and (b) the input information parameter values to the corresponding diagnosable frame image, creates teacher data, and transmits it to the diagnostic support server device 30. In this case, it is necessary to acquire information parameter values related to the same eye as the test eye E photographed by the smart eye camera and create teacher data by associating them with the diagnosable frame image. However, the specific method at that time is arbitrary, and for example, either one of the following methods a and b can be adopted.

[0272] (Method a) (Step a1) In this method a, first, in the ophthalmic facility where the annotation terminal device 40 is installed, the eye of the patient (i.e., the eye to be examined E) is photographed by the smart eye camera, and the moving image data of the eye to be examined E photographed is uploaded to the diagnostic support server device 30. (Step a2) The diagnostic support server device 30 extracts a diagnosable frame image from the uploaded moving image data and distributes it to the annotation terminal device 40 installed in the corresponding ophthalmic facility. (Step a3) Then, the doctor serving as the annotator makes a diagnosis based on the diagnosable frame image, measures various information parameter values regarding the eye of the patient (i.e., the eye to be examined E) by an ophthalmic device, and inputs the measured various information parameter values together with the diagnosis result to the annotation terminal device 40. (Step a4). The annotation terminal device 40 creates teaching data while tagging the diagnosable frame image with the diagnosis result information input by the doctor and the measured various information parameter values.

[0273] (Method b) (Step b1) In this method b, first, a patient ID for uniquely identifying each patient is pre-assigned within the diagnostic support system 1. (Step b2) When uploading the moving image captured by the smart eye camera, the patient ID assigned to the patient is input and uploaded while associating it with the moving image data. (Step b3) Also at the ophthalmic facility, while inputting the patient ID, the measurement results from the ophthalmic device are input into an electronic medical record or the like, and the input measurement values are stored in a database (not shown) while associating them with the patient ID. (Step b4) When the moving image data associated with the patient ID is uploaded from the smart eye camera, the diagnostic support server device 30 extracts a diagnosable frame image from the moving image and transmits it to the annotation terminal device 40 while associating it with the patient ID. (Step b5) The annotation terminal device 40 acquires the measurement values stored in association with the patient ID in the database. (Step Sb6) The annotation terminal device 40 creates teacher data while associating the diagnosis result by the doctor and the measurement values acquired from the database with the diagnosable frame image.

[0274] When the teacher data created in this way is received from the annotation terminal device 40 (Fig. 14, Step Sa5 "Yes"), the diagnostic knowledge acquisition unit 353 stores the teacher data in the teacher data storage unit 333 (Fig. 14, Step Sa6). When the number of teacher data reaches α or more, based on the teacher data, at least one of machine learning and data mining is executed to acquire diagnostic knowledge (Step Sa8). After storing it in the diagnostic knowledge storage unit 334, (Step Sa9), the process ends. At this time, the diagnostic knowledge acquisition unit 353, in conjunction with the health state estimation processing unit 354, executes at least one of machine learning and data mining so that the error between the measurement value acquired by the ophthalmic device and the estimated value based on the diagnosable frame image becomes "0", and acquires diagnostic knowledge.

[0275] [D3] Method for generating diagnostic support information Also in the diagnostic support system 1 of the present embodiment, information parameter values regarding the eye E to be examined photographed by the smart eye camera are basically estimated by the same process as in FIG. 15, and diagnostic support information including the estimated information parameter values is generated.

[0276] However, in the configuration of the present embodiment, in order to estimate the information parameter value in the eye E to be examined, it is necessary to extract a frame image that satisfies the following additional conditions as a diagnosable frame image as described above. Therefore, in step Sb3 of FIG. 15, the diagnosable frame extraction unit 352 extracts a diagnosable frame image that satisfies the condition, and adopts a method of estimating the information parameter value in the eye E to be examined based on the diagnosable frame image and diagnostic knowledge. Further, in the present embodiment, the health state estimation processing unit 354 estimates the information parameter value related to the eye E to be examined in step Sb5. Note that the diagnostic support information generated by the diagnostic processing unit 350 of the diagnostic support server device 30 in the present embodiment may include (1) only the estimated information parameter value, (2) only the estimated result of the disease state in the eye E to be examined as in the first embodiment, or (3) both the estimated result of the disease state and the estimated result of the information parameter value. As a method for generating diagnostic support information including both, (a) a method of estimating step by step, such as first estimating the disease state in the eye E to be examined by the same process as in the first embodiment and then estimating the information parameter value, may be adopted, or (b) a method of estimating the disease state and the information parameter value at once based on diagnostic knowledge may be adopted. In the former case, the same process as in FIG. 15 may be repeated for estimating the disease state and the information parameter value. In the latter case, (i) the health state estimation processing unit 354 may be configured to estimate the disease state and the information parameter value in the eye E to be examined at once in step Sb5 based on the diagnostic knowledge obtained from the teacher data tagged with the diagnostic result information and the information parameter value, or (ii) a configuration may be adopted in which the diagnostic knowledge includes knowledge described in medical books, medical dictionaries, medical papers, and other documents, or medical knowledge disclosed on the Internet, and the health state estimation processing unit 354 estimates the disease state in the eye E to be examined based on various information parameter values estimated from the smart eye camera image and the knowledge of medical books and the like, and describes both the information parameter value and the disease state in the diagnostic support information.In any case of adopting a method, it is basically the same as FIG. 15 except that (1) conditions corresponding to the diagnosable frame image extracted in step Sb3 are added and (2) the health state estimation unit 354 estimates information parameter values in step Sb5.

[0277] [D4] Application Example of the Present Embodiment Next, an application example of the diagnosis mode using the diagnostic support system 1 of the present embodiment will be described. Similar to the first and second embodiments and their modified examples, the diagnostic support system 1 of the present embodiment can (1) diagnose cataracts in the eye to be examined E, (2) diagnose diseases such as iritis and uveitis that occur in the anterior chamber tissue, (3) diagnose chalazion and hordeolum that occur in the eyelid tissue, allergic conjunctivitis, epidemic keratoconjunctivitis, keratoconus, corneal opacity, etc. that occur in the corneal tissue and conjunctival tissue, (4) diagnose diseases such as punctate superficial keratitis and corneal ulcer, (5) diagnose DED, and (6) estimate the disease state in the fundus tissue, and generate and utilize diagnostic support information including the estimation result. In particular, according to the diagnostic support system 1 of the present embodiment, the following applications can be performed using the information parameter values measured by the ophthalmic device.

[0278] (1) Estimation of Anterior Chamber Depth (ACD) In order to verify the applicability of the diagnostic support system 1 of the present embodiment, the present inventor conducted an experiment to actually estimate the anterior chamber depth in the eye to be examined E based on the moving image of the eye to be examined E captured by the smart eye camera.

[0279] (1-1) Overall Flow of This Experiment In this experiment, as shown in FIG. 33, first, the anterior chamber tissue of the eye to be examined E is photographed by the smart eye camera, and the diagnosable frame extraction unit 352 extracts a diagnosable frame image from the photographed moving image (step i in FIG. 33), and extracts the region image of the iris part from the diagnosable frame image (step ii). At this time, for the extraction knowledge, the extraction knowledge for extracting the region image in which the iris is reflected from the diagnosable frame image was used.

[0280] (1-2) Regarding the diagnosable frame image used Here, in order to capture an image capable of estimating the anterior chamber depth by a smart eye camera, it is necessary to use slit light SL as the observation light, irradiate the slit light SL obliquely to the eye to be examined E, and capture a moving image while cutting the eye to be examined E obliquely. Therefore, in this experiment, a slit light forming member 61 and a convex lens member 93 were attached to the close-up photographing device 20A constituting the smart eye camera, and the color filter member 97 was removed. The eye to be examined E was photographed in a state where the slit light SL was irradiated to the eye to be examined E, and a moving image including a diagnosable frame image as illustrated in FIG. 34(A) was photographed.

[0281] Also, when estimating the anterior chamber depth from the moving image captured by the smart eye camera, it is necessary to use, as the diagnosable frame image, a frame image capable of discriminating the part indicated by the green line at the tip of the red line in FIG. 34(A). For this reason, in this experiment, as the frame image (that is, the diagnosable frame image) that can be used for estimating the anterior chamber depth, in addition to the above three conditions, (Condition 4) the slit light SL is irradiated to the pupil and the slit light SL is divided vertically with the pupil in between as shown in FIG. 33(A). Extraction knowledge was used to extract a frame image that satisfies this additional condition as the diagnosable frame image. Then, in step Sa3 of FIG. 14 described above, the diagnosable frame extraction unit 352 extracts a frame image that satisfies these four conditions as the diagnosable frame image, and tags the diagnosable frame image with the information parameter value measured by the ophthalmic device and the diagnostic result information to obtain diagnostic knowledge. In this case, it is desirable to use, as the diagnosable frame image, a frame image in which the slit light SL is irradiated to the center of the pupil. It has been found that the estimation accuracy of the anterior chamber depth can be improved by using such a frame image. However, it has also been found in this experiment that it is sufficient if the slit light SL is irradiated to the pupil, and it is not necessarily required to be irradiated to the center of the pupil.

[0282] At this time, an area image with the iris reflected therein is cut out from the diagnosable frame image extracted based on the extraction knowledge, and while tagging the area image cut out from the diagnosable frame image with the information parameter values of the anterior chamber depth and axial length (AL: Axial Length) measured by anterior segment OCT, teacher data is created (step iii in FIG. 33), and diagnostic knowledge is acquired.

[0283] Here, the anterior chamber depth is indicated by the numerical value "ACD = 3.149 mm" shown by the green circle in the measurement result GUI of anterior segment OCT shown in FIG. 34(B), and substantially coincides with the distance from the red point "conea - B" to the red point "lens - F" in (B). Also, the measured value of the anterior chamber depth measured by this anterior segment OCT theoretically indicates the same distance as the distance shown by the green line at the tip of the red line in FIG. 34(A). For this reason, in this experiment, while defining the length of this green line as the measured value of the anterior chamber depth measured by anterior segment OCT, tagging was performed on the diagnosable frame image to create teacher data.

[0284] At this time, the annotator inputs the area image cut out from the diagnosable frame image, the measured value of the anterior chamber depth measured by anterior segment OCT (for example, the numerical value "ACD = 3.149 mm" in FIG. 34(B)), and the axial length obtained by anterior segment OCT (AL = 32 mm exemplified in FIG. 33) into the annotation terminal device 40, thereby tagging the diagnosable frame image with the measured values of the anterior chamber depth and axial length to create teacher data. Then, learning is performed so that the error between the value predicted from the diagnosable frame image and the measured value of anterior segment OCT becomes "0", and diagnostic knowledge is acquired (step Sa8 in FIG. 14). Note that the specific processing at this time is the same as that in FIG. 14 except that the knowledge used as extraction knowledge is the knowledge for extracting the knowledge that satisfies the above four conditions, and the frame images that satisfy the four conditions are extracted as diagnosable frame images in step Sa3.

[0285] And in this experiment, based on the diagnostic knowledge obtained by the above method, the anterior chamber depth was estimated using the moving images captured by the smart eye camera. The operation during the anterior chamber depth estimation is basically the same as that in FIG. 15. However, when estimating the anterior chamber depth, since it is necessary to use the diagnosable frame image that satisfies the above four conditions as described above, in step Sb3, the diagnosable frame extraction unit 352 adopted a method of extracting a frame image that satisfies the above four conditions as a diagnosable frame image based on the extraction knowledge. Also, at this time, after the diagnosable frame extraction unit 352 extracted the diagnosable frame image that satisfies the above four conditions (step i in FIG. 33), it cut out the region image with the iris reflected therein from the diagnosable frame image in the same manner as above (step ii). At this time, the diagnostic knowledge used was obtained by tagging the region image with the anterior chamber depth measurement value and the axial length measurement value (step iii). Then, based on the diagnostic knowledge and the captured image by the smart eye camera, the anterior chamber depth was estimated by the health state estimation processing unit 354 (step iv in FIG. 33).

[0286] As a result of the above experiment, the estimated value of the anterior chamber depth based on the diagnosable frame image extracted from the moving image captured by the smart eye camera was within the error range of 150 ± 360 μm compared with the value actually measured by anterior segment OCT, and it was found that the anterior chamber depth could be estimated with very high accuracy.

[0287] (2) Regarding the determination of the risk of glaucoma attack According to the present embodiment as described above, since the anterior chamber depth of the eye E to be examined can be estimated with high accuracy, it is also possible to determine the onset risk of glaucoma in the eye E to be examined based on the moving image captured by the smart eye camera. Generally, the value of the anterior chamber depth is related to the onset risk of glaucoma, and when the anterior chamber depth is less than 2.5 mm, it can be determined that there is a risk of onset of angle-closure glaucoma (for example, refer to https: / / en.wikipedia.org / wiki / Anterior_chamber_of_eyeball). Therefore, with the configuration of the present embodiment, it is also possible to estimate the anterior chamber depth of the eye E to be examined, determine the presence or absence of the onset risk of glaucoma, and generate diagnostic support information including the presence or absence of the onset risk of glaucoma.

[0288] (3) Estimation of corneal radius of curvature In addition, in order to confirm the applicability of the diagnostic support system 1 of the present embodiment, the present inventor measured the corneal radius of curvature in the eye E to be examined with an ophthalmic device such as an anterior segment OCT, tagged the diagnostic frame image or the region image of the region where the cornea was reflected with the measured value to create teacher data, acquired diagnostic knowledge, and estimated the corneal radius of curvature of the eye E to be examined based on the diagnostic knowledge and the moving image captured by the smart eye camera. As a result, it was found that the corneal radius of curvature value can also be estimated with high accuracy, similar to the anterior chamber depth. In this case, since the conditions for corresponding to the diagnostic frame image are the same as those for the anterior chamber depth and satisfy the above four conditions, also in this case, for the extraction knowledge, the extraction knowledge for extracting the frame image that satisfies the above four conditions was used.

[0289] (4) Estimation of axial length of the eye Here, it is known that the eyeball has a shape close to a sphere, and the shape of the entire eyeball depends on the shape of the anterior segment of the eye. For example, a strong correlation is observed between the anterior chamber depth and the axial length of the eye (see, for example, https: / / pubmed.ncbi.nlm.nih.gov / 26107475 / ). Also, a strong correlation is observed between the corneal radius of curvature and the axial length of the eye (see, for example, https: / / pubmed.ncbi.nlm.nih.gov / 32209342 / ). Furthermore, a strong correlation is observed between the spherical equivalent (SE) and the axial length of the eye (see, for example, https: / / pubmed.ncbi.nlm.nih.gov / 33326192). Therefore, by tagging these values to the diagnosable frame image or region image while associating them with the measured value of the axial length to create training data, it is also possible to configure to estimate the axial length from information parameter values such as the anterior chamber depth, corneal radius of curvature, and refractive spherical power estimated from the diagnosable frame image.

[0290] (5) Estimation of biopsy (biopsy) results In the diagnostic support system 1 of this embodiment, when estimating the biopsy result of the eye E to be examined from the moving image captured by the smart eye camera, the diagnostic knowledge is basically acquired by the same process as that shown in FIG. 14. At this time, the doctor who serves as the annotator acquires the biopsy result in the target tissue of the biopsy and inputs the diagnostic result based on the biopsy result into the annotation terminal device 40. The annotation terminal device 40 tags the diagnosable frame image with the diagnostic result information corresponding to the input diagnostic result to create teacher data, and transmits it to the diagnostic support server device 30. Then, in the diagnostic support server device 30, the diagnostic knowledge acquisition unit 353 stores the teacher data in the teacher data storage unit 333 (step Sa6). When the number of teacher data reaches α or more (step Sa7 “Yes”), the diagnostic knowledge is acquired and stored based on the teacher data (steps Sa8 and Sa9), and the process ends. Note that the target of the biopsy can be the eyelid tissue, the anterior eye tissue, the intraocular tissue, and the fundus tissue. At this time, when the target tissue is the eyelid and the anterior eye tissue, slit light SL, white diffused light, and blue light can be used as the observation light. On the other hand, when the fundus tissue is the target, it is necessary to attach the proximity photographing device 20B in the second embodiment to the mobile communication terminal device 10 and use linearly polarized light as the observation light. Note that when the fundus tissue is the target, in addition to the biopsy result, information parameter values such as the retinal thickness, optic disc diameter, and choroidal blood vessels measured by OCT are tagged to the diagnosable frame image or region image together with the diagnostic result information to create teacher data, so that these information parameter values can be estimated with high accuracy, and the presence or absence of a disease in the fundus tissue of the eye E to be examined can be estimated with high accuracy. In particular, since the moving image captured by the smart eye camera can be captured at a very high resolution (for example, 4K or 8K resolution) using the out-camera module of the mobile communication terminal device 10, when uploading the moving image data, the region image including the target site (for example, the eyelid, etc.) is cut out and enlarged for use, so that the pathological findings that can only be known by biopsy can be estimated with high accuracy from the moving image captured by the smart eye camera.

[0291] As described above, according to the diagnostic support system 1 of the present embodiment, since the values of various information parameters regarding the eye E to be examined can be estimated from the moving image captured by the smart eye camera, even in regions lacking equipment and doctors, such as developing countries and remote areas where various ophthalmic devices cannot be installed, the health condition of the patient's eyes can be estimated in detail and appropriately and used for the diagnosis of various diseases. In the configuration of the present embodiment, similar to the first embodiment, for each diagnosable frame image, the probability corresponding to "Diagnosable" is calculated, a plurality of diagnosable frame images are extracted, the information parameter values are estimated for each diagnosable frame image, and while weighting is performed on each estimated value according to the probability corresponding to "Diagnosable", a configuration may be adopted in which the most likely information parameter value in the eye E to be examined is estimated. In this case, it is the same as the first embodiment except that the estimation target is the information parameter value instead of the disease state. Also, in the configuration of the present embodiment, similar to Modification Example 4, a three-dimensional image may be constructed, diagnostic knowledge may be acquired based on the three-dimensional image, and the information parameter value may be estimated based on the diagnostic knowledge and the three-dimensional image. Furthermore, in the configuration of the present embodiment, similar to Modification Example 6, it is desirable to adopt a configuration in which a plurality of information parameter values (for example, anterior chamber depth and corneal radius of curvature, etc.) can be estimated at once by a single moving image capture and used. In this case, a plurality of diagnostic knowledges for estimating each information parameter value are stored in advance in the diagnostic knowledge storage unit 334, and when moving image data is uploaded from the smart eye camera, the diagnosable frame images for estimating each information parameter value are extracted for each information parameter value, and based on the extracted diagnosable frame images and the diagnostic awareness for the corresponding information parameter values, a configuration may be adopted in which a plurality of information parameter values in a part of the tissue of the eye E to be examined are estimated at once. Also, in this case, similar to Modification Example 6, a configuration may be adopted in which diagnostic support information describing the estimation results of a plurality of information parameter values in a list format is generated.

Explanation of Signs

[0292] 1…Diagnostic support system, 2…Polarizing filter (vertical polarization), 3…Polarizing filter (horizontal polarization), 4…Color filter (orange), 8…Plate-shaped filter, 10…Mobile communication terminal device, 20, 20A, 20B…Device for close-up photography, 30…Diagnostic support server device, 310…Communication control unit, 320…ROM / RAM, 330…Storage device, 331…Program storage unit, 332…Extraction knowledge storage unit, 333…Teacher data storage unit, 334…Diagnostic knowledge storage unit, 340…Server management control unit, 350…Diagnostic processing unit, 351…Moving image data acquisition unit, 352…Diagnosable frame extraction unit, 353…Diagnostic knowledge acquisition unit 353, 354…Health status estimation processing unit, 355…Diagnostic support information generation unit, 356…Diagnostic support information distribution unit, 61…Slit light forming member, 61’…Main body part, 62…Cylindrical lens, 63…Upper holding member, 64…Lower holding member, 65…First reflection mirror, 66…Second reflection mirror, 67…Slit, 68…Mounting part, 69…Step part, 80…Housing, 81…Outer wall part, 82…Front wall, 83…Rear wall, 84…Left wall, 85…Right wall, 86…Upper wall, 87…Lower wall, 88…Hole in the front wall, 90…Front plate, 90a…Open left edge part, 90b…Right edge part, 90c…Upper edge part, 90d…Lower edge part, 91…Camera lens for photography, 92…Light source, 93…Convex lens member, 94…Convex lens mounting hole, 95…Hole, 96…Convex lens, 97…Color filter member, 98…Hole, 180…Cylindrical member, 181…Mounting part, 182, 182a, 182b…Cylindrical part, 183…Convex lens, 84…Opening

Claims

1. An image obtained by photographing an eye to be examined with the mobile communication terminal device equipped with the proximity photographing device that is attached to the mobile communication terminal device including a light source and a photographing camera lens, the proximity photographing device including at least: (a) an observation light irradiation member that irradiates an observation target tissue of the eye to be examined with any one of slit light, blue light, and linearly polarized light generated based on the light source light emitted from the light source, or irradiates the observation target tissue with the light source light as it is as the observation light; and (b) a convex lens member that condenses light including reflected light of the observation light in the observation target tissue onto the photographing camera lens. The image includes at least one acquisition means for acquiring a photographed image including one or more first frame images that can be used to estimate at least one of (i) the health state of the eye to be examined and (ii) information parameter values including any one or more of distance, angle, and area in a part of the tissue of the eye to be examined. A first storage means in which first knowledge for extracting the first frame image from the acquired photographed image is stored in advance. A first extraction means for extracting the first frame image included in the acquired photographed image based on the first knowledge. A second storage means in which second knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) the information parameter values is stored based on the photographed image. An estimation means for estimating at least one of (i) the health state of the eye to be examined reflected in the acquired photographed image and (ii) the information parameter values based on the extracted first frame image and the second knowledge. A generation means for generating diagnostic support information including at least one of the estimated health state of the eye to be examined and the information parameter values. A distribution means for distributing the generated diagnostic support information to an external device. A diagnostic support device characterized by having the above.

2. The first extraction means calculates the probability that each of the frame images included in the photographed image corresponds to the first frame image based on the frame image included in the photographed image and the first knowledge, and extracts the first frame image based on the calculated probability. The diagnostic support device according to Claim 1.

3. The first extraction means extracts a plurality of the frame images with high calculated probabilities as the first frame images, The estimation means Based on the plurality of the extracted first frame images and the second knowledge, for each of the first frame images, at least one of (i) the health state of the eye to be examined and (ii) the information parameter value is estimated, and while weighting the estimated health state and information parameter value by the calculated probability, at least one of the most likely health state and information parameter value of the eye to be examined is estimated. The diagnostic support device according to claim 2.

4. The acquisition means acquires the captured image that is captured while irradiating at least one of the eyelid and the anterior eye tissue of the eye to be examined with the slit light generated based on the light source light as the observation light. In the first storage means as the first knowledge, (i) the state of diseases occurring in the eyelid and the anterior eye tissue of the eye to be examined, and (ii) knowledge for extracting the first frame image that can be used for estimating at least one of the information parameter values related to the eyelid and the anterior eye tissue are stored in advance, and In the second storage means as the second knowledge, (i) the state of diseases occurring in the eyelid and the anterior eye tissue of the eye to be examined, and (ii) knowledge for estimating at least one of the information parameter values related to the eyelid and the anterior eye tissue are stored. The estimation means estimates at least one of (i) the state of diseases in at least one of the eyelid and the anterior eye tissue of the eye to be examined and (ii) at least one of the information parameter values related to the eyelid and the anterior eye tissue based on the extracted first frame image and the second knowledge. The diagnostic support device according to any one of claims 1 to 3.

5. The acquisition means acquires the captured image that is captured while irradiating at least one of the cornea and the conjunctiva of the eye to be examined with the blue light generated based on the light source light as the observation light in a state where a wound occurring in at least one of the cornea and the conjunctiva of the eye to be examined is contrasted with a contrast agent. In the first storage means as the first knowledge, knowledge for extracting the first frame image that can be used for diagnosing the state of a wound occurring in at least one of the cornea and the conjunctiva of the eye to be examined is stored in advance, and In the second storage means as the second knowledge, knowledge for estimating the state of diseases in at least one of the cornea and the conjunctiva from the state of a wound occurring in at least one of the cornea and the conjunctiva of the eye to be examined is stored. The estimation means The diagnostic support device according to any one of claims 1 to 3, which estimates the disease state in at least one of the cornea and conjunctiva tissues of the eye to be examined based on the extracted first frame image and the second knowledge.

6. The acquisition means acquires the captured image obtained by irradiating the fundus tissue of the eye to be examined with linearly polarized light generated based on the light source light as the observation light while capturing an image. In the first storage means as the first knowledge, knowledge for extracting the first frame image that can be used for estimating at least one of (i) the disease state that develops in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue is stored in advance, and In the second storage means as the second knowledge, knowledge for estimating at least one of (i) the disease state in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue is stored. The estimation means estimates at least one of (i) the disease state in the fundus tissue of the eye to be examined and (ii) the information parameter value regarding the fundus tissue based on the extracted first frame image and the second knowledge. The diagnostic support device according to any one of claims 1 to 3.

7. The acquisition means acquires the captured image obtained by irradiating the tissue to be observed of the eye to be examined with the light source light as the observation light while capturing an image in the state as it is. In the first storage means as the first knowledge, knowledge for extracting the first frame image that can be used for estimating at least one of (i) the disease state that develops in at least one of the eyelid, ocular surface, cornea, and conjunctiva tissues of the eye to be examined and (ii) the information parameter value regarding at least one of the eyelid, ocular surface, cornea, and conjunctiva tissues of the eye to be examined is stored in advance, and In the second storage means as the second knowledge, knowledge for estimating at least one of (i) the disease state in at least one of the eyelid, ocular surface, cornea, and conjunctiva tissues of the eye to be examined and (ii) the information parameter value regarding at least one of the eyelid, ocular surface, cornea, and conjunctiva tissues of the eye to be examined is stored. The estimation means Based on the extracted first frame image and the second knowledge, (i) estimate the disease state in at least one of the tissues of the eyelid, ocular surface, cornea, and conjunctiva of the eye to be examined, and (ii) estimate at least one of the information parameter values related to at least one of the tissues of the eyelid, ocular surface, cornea, and conjunctiva of the eye to be examined. The diagnostic support device according to any one of claims 1 to 3.

8. Based on the first frame image extracted by the first extraction means, further comprising a learning means for executing at least one of machine learning and data mining to acquire the second knowledge and storing the acquired second knowledge in the second storage means. The estimation means Based on the second knowledge stored in the second storage means by the learning means and the extracted first frame image, estimate at least one of the disease state and the information parameter value in the eye to be examined. The diagnostic support device according to any one of claims 1 to 7.

9. The learning means Obtain diagnostic result information indicating the diagnostic result of a doctor based on the first frame image extracted by the first extraction means, and while using the diagnostic result information and the corresponding first frame image as teacher data, execute at least one of machine learning and data mining to acquire the second knowledge and store it in the second storage means. The diagnostic support device according to claim 8.

10. In the first storage means, As the first knowledge, a plurality of knowledges for extracting the first frame image available for diagnosing each disease that can occur in each tissue constituting the eye to be examined are stored. The first extraction means Based on the first knowledge, extract the first frame image available for disease state estimation for each disease, and The learning means Obtain the diagnostic result information related to the corresponding disease based on the first frame image corresponding to each extracted disease, and while using the diagnostic result information and the corresponding first frame image as teacher data, execute at least one of machine learning and data mining to acquire the second knowledge necessary for diagnosing each disease, and store the acquired second knowledge in the second storage means in association with the corresponding disease. The diagnostic support device according to claim 9.

11. In the first storage means, As the first knowledge, a plurality of pieces of knowledge for extracting the first frame images that can be used for diagnosing each disease that can develop in each tissue constituting the eye to be examined are stored, and in the second storage means, as the second knowledge, a plurality of pieces of knowledge for estimating the state of each disease are stored for each disease, the first extraction means, based on the first knowledge, extracts the first frame images that can be used for diagnosing the disease for each disease, and the estimation means, based on the first frame images extracted for each disease and the second knowledge of the corresponding disease, estimates the state of each disease in the eye to be examined, the generation means, generates, as the diagnostic support information, information including the estimated state of each disease according to any one of claims 1 to 10.

12. labeling means for labeling the corresponding tissue name for the in-focus tissue of the eye to be examined in each frame image included in the captured image; and second extraction means for extracting, as the second frame image, the frame image having the largest area of the pixel region in which the in-focus tissue is reflected in each of the labeled frame images, further comprising: the generation means, generates the diagnostic support information including the extracted second frame image according to any one of claims 1 to 11.

13. further comprising three-dimensional image construction means for stacking each frame image included in the acquired captured moving image according to the focal length to construct a three-dimensional image of the eye to be examined, in the second storage means, as the second knowledge, knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the three-dimensional image is stored, the estimation means, estimates at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the generated three-dimensional image and the second knowledge according to any one of claims 1 to 12.

14. further comprising cutting-out means for cutting out, as a region image, the region in which the tissue to be estimated for the health state is reflected in the first frame image extracted by the first extraction means, in the second storage means, As the second knowledge, knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) information parameter values including any one or more of distance, angle, and area in a part of the tissue of the eye to be examined based on a region image cut out from the captured image is stored. The estimation means The diagnostic support device according to any one of claims 1 to 8, wherein the estimation means estimates at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the region image and the second knowledge.

15. The diagnostic support device further includes a cutting-out means for cutting out, as a region image, a region in the first frame image extracted by the first extraction means in which a tissue to be estimated for the health state is imaged. In the second storage means As the second knowledge, knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) information parameter values including any one or more of distance, angle, and area in a part of the tissue of the eye to be examined based on a region image cut out from the captured image is stored. The estimation means The diagnostic support device according to claim 9 or 10, wherein the estimation means estimates at least one of (i) the health state of the eye to be examined and (ii) the information parameter value based on the region image and the second knowledge.

16. The learning means The diagnostic support device according to claim 15, wherein the learning means executes at least one of machine learning and data mining based on the cut-out region image and the diagnostic result information to acquire the second knowledge.

17. The learning means The diagnostic support device according to claim 15 or 16, wherein the learning means acquires the information parameter value in the eye to be examined and acquires the second knowledge based on (a) the acquired information parameter value, (b) the diagnostic result information, and (c) the first frame image or the cut-out region image.

18. A mobile communication terminal device including a light source and a camera lens for photographing, comprising: (a) an observation light irradiation member that irradiates an observation target tissue of the eye to be examined with any one of slit light, blue light, and linearly polarized light generated based on the light source light emitted from the light source, or irradiates the observation target tissue with the light source light as it is as the observation light; and (b) a convex lens member that condenses light including reflected light of the observation light in the observation target tissue onto the camera lens for photographing. The mobile communication terminal device is equipped with a close-up photographing device including at least the above components. An imaging device that captures an image of the eye to be examined using the mobile communication terminal device, and estimates at least one of (i) the health state of the eye to be examined and (ii) information parameter values including at least one of distance, angle, and area in a part of the tissue of the eye to be examined, and a diagnostic support device that supports the diagnosis of the eye to be examined based on the captured image including one or more first frame images that can be used for the estimation, having, the diagnostic support device, an acquisition means for acquiring the captured image, a first storage means in which first knowledge for extracting the first frame image from the acquired captured image is stored in advance, a first extraction means for extracting the first frame image included in the acquired captured image based on the first knowledge, a second storage means in which second knowledge for estimating at least one of (i) the health state of the eye to be examined and (ii) the information parameter values is stored based on the captured image, an estimation means for estimating at least one of (i) the health state of the eye to be examined reflected in the acquired captured image and (ii) the information parameter values based on the extracted first frame image and the second knowledge, a generation means for generating diagnostic support information including at least one of the estimated health state of the eye to be examined and the information parameter values, a distribution means for distributing the generated diagnostic support information to an external device, A diagnostic support system characterized by having the above.

19. A mobile communication terminal device including a light source and a camera lens for photography, comprising: (a) an observation light irradiating member that irradiates an observation target tissue of an eye to be examined with any one of slit light, blue light, and linearly polarized light generated based on light from the light source as observation light, or irradiates the observation target tissue with the light from the light source as it is as the observation light; and (b) a convex lens member that condenses light including reflected light of the observation light in the observation target tissue onto the camera lens for photography. An image of the eye to be examined photographed by a mobile communication terminal device equipped with a close-up photography device including at least the above components, which functions as a diagnostic support device for assisting in the diagnosis of the eye to be examined based on a photographed image including one or more first frame images that can be used to estimate at least one of (i) the health state of the eye to be examined and (ii) information parameter values including any one or more of distance, angle, and area in a part of the tissue of the eye to be examined. A computer having: acquisition means for acquiring the photographed image; first extraction means for extracting the first frame image included in the acquired photographed image based on the first knowledge; estimation means for estimating (i) the health state of the eye to be examined reflected in the acquired photographed image and (ii) at least one of the information parameter values based on the extracted first frame image and the second knowledge; generation means for generating diagnostic support information including at least one of the estimated health state of the eye to be examined and the information parameter values; distribution means for distributing the generated diagnostic support information to an external device; A program characterized by causing the above to function.

Images