Ophthalmic image processing method, ophthalmic image processing device, and ophthalmic image processing program
The ophthalmological image processing method uses machine learning and edge detection to analyze subsection images, addressing the limitations of existing devices by enhancing precision and speed in evaluating dry eye conditions.
Patent Information
- Application Number
- JP2021567518
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-27
- Filing Date
- 2020-12-23
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-12-23
AI Technical Summary
Existing ophthalmological devices struggle to accurately evaluate the condition of the eye in detail, particularly for conditions like dry eye, as they often misidentify noise elements and require multiple images, lacking precision in identifying specific areas within the eye.
An ophthalmological image processing method using machine learning to analyze subsection images of the eye, setting result reflection areas, and applying edge detection to improve prediction accuracy and reduce noise interference, allowing for detailed evaluation of dry eye conditions.
The method enables precise, high-accuracy analysis of dry eye conditions by identifying specific areas within the eye, reducing noise interference, and improving prediction speed and resolution.
Smart Images

Figure 0007582972000001 
Figure 0007582972000002 
Figure 0007582972000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an ophthalmic image processing method, an ophthalmic image processing apparatus, and an ophthalmic image processing program for analyzing the condition of a subject's eye. [Background technology]
[0002] Conventionally, various methods have been proposed for photographing an eye to be examined and evaluating the condition of the eye by a computer using the photographed image. For example, Patent Document 1 proposes an ophthalmic apparatus for diagnosing dry eye.
[0003] Patent Document 1 discloses an ophthalmic device that captures a color image of an interference pattern formed in the tear film of a test eye by illumination light, decomposes the captured color image into a plurality of color components, determines the interference fringes appearing in each color component based on the signal for each decomposed color component, and calculates a value indicating the progression state of dry eye based on the number of interference fringes determined for each color component or a change in the signal level of the determined interference fringes. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2005-211173 A Summary of the Invention [Problem to be solved by the invention]
[0005] According to the ophthalmologic apparatus described in the above Patent Document 1, the progression state of dry eye can be quantified accurately and precisely, but in this Patent Document 1, a fairly wide area in the subject's eye is set as an analysis area, and interference fringes are judged based on the signal level of at least one analysis line in the analysis area. In other words, although it is possible to judge the state of interference fringes in the subject's eye as a whole, it is not possible to evaluate the state of each detail of the subject's eye.
[0006] Generally, the method of evaluating one entire ophthalmic image has the advantage of short analysis time because the result can be obtained by analyzing only one image, but it has the following disadvantages: (1) although the presence of the extraction target can be predicted, it is not known which part of the image it is present in, so the diagnostic support ability is low, (2) it is extremely difficult to extract the structure and characteristics of the target that exists in only a small part of the image, (3) it is expected that noise elements (high contrast areas of eyelashes and the outline of the pupil in the interference fringe image) will be mistakenly taken as the extraction target, and (4) many images must be prepared to improve the prediction accuracy. Therefore, there has been a demand for an ophthalmic image processing method that can solve these problems and precisely analyze the condition of the examined eye, such as the presence or absence of breaks, as well as the condition of the interference fringes.
[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an ophthalmic image processing method, an ophthalmic image processing device, and an ophthalmic image processing program that are capable of predicting the condition of a test eye with high accuracy and analyzing it in detail. [Means for solving the problem]
[0008] The ophthalmic image processing method of the present invention is an ophthalmic image processing method for evaluating the condition of a test eye from an ophthalmic image photographed of the test eye using machine learning, and includes a learning step of extracting a plurality of small area images from a training ophthalmic image, and using correct answer data regarding the state of each small area image, learning in advance to predict the condition of the test eye for each small area image by machine learning to obtain a trained model, an image acquisition step of acquiring a test ophthalmic image, an extraction step of extracting a plurality of small area images from the test ophthalmic image, and a prediction step of predicting the state of the test eye for each small area image using the trained model, and is characterized in that the small area images are extracted from the ophthalmic image so as to have an image size corresponding to the condition of the test eye to be evaluated.
[0009] In addition, the ophthalmologic image processing method of the present invention further includes a result reflection area setting step for setting a result reflection area having a predetermined size less than the size of the small partition image, in which the prediction result of the small partition image is reflected, and in the extraction step, an area that contains the result reflection area and has a predetermined size greater than or equal to the result reflection area is extracted as the small partition image in correspondence with each of the result reflection areas, and in the prediction step, the prediction result obtained for each small partition image is reflected in the corresponding result reflection area.
[0010] In addition, the ophthalmological image processing method of the present invention further includes an edge image generation step of extracting edges from the ophthalmological image to generate an edge image, and in the prediction step, when the brightness value in the edge image corresponding to the position of the result reflection area is equal to or greater than a predetermined threshold value, the small partition image corresponding to the result reflection area is set as the target for prediction processing.
[0011] In addition, the ophthalmologic image processing method of the present invention is further characterized in that, in the prediction step, an average brightness value is calculated for each small area image, and the small area images whose average brightness value is equal to or greater than a predetermined threshold value are set as targets for prediction processing.
[0012] In addition, the ophthalmologic image processing method according to the present invention is characterized in that it further comprises a prediction result display step of displaying an image showing the prediction result superimposed on the ophthalmologic image by a display means.
[0013] Furthermore, the ophthalmic image processing method of the present invention is further characterized in that the image acquisition step acquires multiple ophthalmic images arranged in chronological order, and the prediction step performs prediction processing on each of the multiple ophthalmic images, and the method further includes a change determination step for determining whether or not the temporal and / or spatial changes in the state of the test eye satisfy predetermined conditions for the prediction results obtained for each of the multiple ophthalmic images.
[0014] In the ophthalmologic image processing method according to the present invention, the condition of the eye to be examined is the presence or absence of interference fringes in a tear film.
[0015] In the ophthalmologic image processing method according to the present invention, the state of the subject's eye is the presence or absence of a break.
[0016] The ophthalmic image processing device of the present invention is an ophthalmic image processing device for evaluating the condition of a test eye from an ophthalmic image photographed of the test eye, and comprises an image acquisition unit for acquiring the ophthalmic image to be evaluated, an extraction unit for extracting multiple small area images from the ophthalmic image, and a prediction unit for extracting multiple small area images from a learning ophthalmic image and predicting the condition of the test eye for each small area image based on a trained model that has been pre-trained to predict the condition of the test eye for each small area image by machine learning using correct answer data regarding the state of each small area image, and is characterized in that the small area images are extracted from the ophthalmic image so as to have an image size corresponding to the condition of the test eye to be evaluated.
[0017] The ophthalmic image processing program of the present invention is an ophthalmic image processing program for causing a computer to realize a process of evaluating the condition of a test eye from an ophthalmic image photographed of the test eye, and is characterized in that the computer realizes an image acquisition function for acquiring the ophthalmic image to be evaluated, an extraction function for extracting multiple small area images from the ophthalmic image, and a prediction function for predicting the condition of the test eye for each small area image based on a trained model that has been pre-trained to predict the condition of the test eye for each small area image by machine learning using correct answer data regarding the state of each small area image, and the small area images are extracted from the ophthalmic image so as to have an image size corresponding to the condition of the test eye to be evaluated.
[0018] The ophthalmic image processing method of the present invention is an ophthalmic image processing method for evaluating the dry eye condition of a test eye from an ophthalmic image taken of the test eye using machine learning, and includes a learning step for extracting a plurality of small section images from a training ophthalmic image, and using correct answer data regarding the state of each small section image, learning in advance to predict by machine learning whether the dry eye condition of the test eye for each small section image corresponds to at least a healthy state, an aqueous-deficient dry eye state, or an evaporative dry eye state, to obtain a trained model; an image acquisition step for acquiring a test ophthalmic image; an extraction step for extracting a plurality of small section images from the test ophthalmic image; and a prediction step for predicting the dry eye condition of the test eye for each small section image using the trained model, and the small section images are extracted from the ophthalmic image so as to have an image size that enables the dry eye condition of the test eye to be appropriately evaluated.
[0019] In addition, the ophthalmologic image processing method of the present invention further includes a result reflection area setting step for setting a result reflection area having a predetermined size less than the size of the small partition image, in which the prediction result of the small partition image is reflected, and in the extraction step, an area that contains the result reflection area and has a predetermined size greater than or equal to the result reflection area is extracted as the small partition image in correspondence with each of the result reflection areas, and in the prediction step, the prediction result obtained for each small partition image is reflected in the corresponding result reflection area.
[0020] In addition, the ophthalmologic image processing method of the present invention is further characterized in that, in the prediction step, an average brightness value is calculated for each small area image, and the small area images whose average brightness value is equal to or greater than a predetermined threshold value are set as targets for prediction processing.
[0021] In addition, the ophthalmologic image processing method according to the present invention is characterized by further comprising a prediction result display step of displaying an image showing the prediction result superimposed on the ophthalmologic image by a display means.
[0022] Furthermore, the ophthalmic image processing method of the present invention is further characterized in that the image acquisition step acquires multiple ophthalmic images arranged in time series, and the prediction step performs prediction processing on each of the multiple ophthalmic images, and the method further includes a change determination step of determining whether or not the temporal and / or spatial change in the dry eye condition of the test eye satisfies a predetermined condition for the prediction results obtained for each of the multiple ophthalmic images.
[0023] The ophthalmic image processing device of the present invention is an ophthalmic image processing device for evaluating the dry eye condition of a test eye from an ophthalmic image taken of the test eye, and comprises an image acquisition unit that acquires the ophthalmic image to be evaluated, an extraction unit that extracts multiple small area images from the ophthalmic image, and a prediction unit that extracts multiple small area images from a learning ophthalmic image and predicts the dry eye condition of the test eye for each small area image based on a trained model that has been pre-trained by machine learning to predict whether the dry eye condition of the test eye for each small area image corresponds to at least a healthy state, an aqueous tear-deficient dry eye state, or an evaporative dry eye state, and is characterized in that the small area images are extracted from the ophthalmic image so as to have an image size that enables the dry eye condition of the test eye to be appropriately evaluated.
[0024] The ophthalmic image processing program of the present invention is an ophthalmic image processing program for causing a computer to realize a process of evaluating the dry eye condition of a test eye from an ophthalmic image photographed of the test eye, and the computer is provided with an image acquisition function for acquiring the ophthalmic image to be evaluated, an extraction function for extracting multiple small area images from the ophthalmic image, and a prediction function for extracting multiple small area images from a learning ophthalmic image and using correct answer data regarding the state of each small area image, predicting the dry eye condition of the test eye for each small area image based on a trained model that has been pre-trained by machine learning to predict whether the dry eye condition of the test eye for each small area image corresponds to at least a healthy state, an aqueous tear-deficient dry eye state, or an evaporative dry eye state, and the small area images are extracted from the ophthalmic image so as to have an image size that enables the dry eye condition of the test eye to be evaluated to be appropriately evaluated. Effect of the Invention
[0025] The embodiments of the present application address one or more of the deficiencies. [Brief description of the drawings]
[0026] [Figure 1] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing system corresponding to at least one of the embodiments of the present invention. [Diagram 2] 1 is a block diagram showing an example of a configuration of an ophthalmologic image-processing device and peripheral devices corresponding to at least one of the embodiments of the present invention. [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a configuration of an imaging device corresponding to at least one of the embodiments of the present invention. [Figure 4] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Diagram 5] FIG. 2 is a flowchart showing an example of the flow of a learning process corresponding to at least one of the embodiments of the present invention. [Figure 6]FIG. 2 is a flowchart illustrating an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. [Figure 7] FIG. 11 is an explanatory diagram illustrating an example of an ophthalmological image in which an ophthalmological image before prediction processing and a result of prediction processing are superimposed. [Figure 8] 11 is a table showing the relationship between the size of a small section image and the breakup detection rate. [Figure 9] 13 is a table showing the relationship between the size of a small section image and the detection rate of tear interference fringes. [Figure 10] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Figure 11] FIG. 2 is a flowchart illustrating an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. [Figure 12] 11 is an explanatory diagram for explaining an example of setting a result reflection area and a small section image. FIG. [Figure 13] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Figure 14] FIG. 2 is a flowchart illustrating an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. [Figure 15] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing system corresponding to at least one of the embodiments of the present invention. [Figure 16] 1 is a block diagram showing an example of a configuration of an ophthalmologic image-processing device and peripheral devices corresponding to at least one of the embodiments of the present invention. [Figure 17] FIG. 1 is an explanatory diagram showing an example of a configuration of an imaging device corresponding to at least one of the embodiments of the present invention. [Figure 18] FIG. 1 is an explanatory diagram showing a state of dry eye of a subject's eye as an evaluation target in an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Figure 19] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Figure 20]FIG. 2 is a flowchart showing an example of the flow of a learning process corresponding to at least one of the embodiments of the present invention. [Figure 21] FIG. 2 is a flowchart illustrating an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. [Figure 22] 11 is an explanatory diagram for explaining the relationship between the size of a small block image and whether or not a striped pattern can be detected; FIG. [Figure 23] FIG. 11 is an explanatory diagram illustrating an example of an ophthalmological image in which an ophthalmological image before prediction processing and a result of prediction processing are superimposed. [Figure 24] 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing apparatus corresponding to at least one of the embodiments of the present invention. [Diagram 25] FIG. 2 is a flowchart illustrating an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. [Figure 26] 11 is an explanatory diagram for explaining an example of setting a result reflection area and a small section image. FIG. [Figure 27] FIG. 11 is an explanatory diagram of a graph for plotting and representing the likelihood of a dry eye condition of a subject's eye. [Figure 28] FIG. 1 is an explanatory diagram showing an example of dynamic analysis of the dry eye state of a subject's eye. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that the various components in the examples of the embodiments described below can be appropriately combined to the extent that no inconsistencies or the like arise. In addition, the contents described as an example of one embodiment may be omitted in other embodiments. In addition, the contents of operations and processes that are not related to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes that make up the various flows described below is random to the extent that no inconsistencies or the like arise in the process contents.
[0028] [First embodiment] Hereinafter, an example of an ophthalmologic image processing system according to a first embodiment of the present invention will be described with reference to the drawings. As will be described later, the ophthalmologic image processing system of the present invention can also be realized as a standalone ophthalmologic image processing device that does not need to be connected to other devices via a communication network, so it does not necessarily have to be an ophthalmologic image processing system. The following description will be given on the assumption that it may function as an ophthalmologic image processing system.
[0029] Fig. 1 is a block diagram showing an example of a configuration of an ophthalmologic image processing system corresponding to at least one of the embodiments of the present invention. As shown in Fig. 1, the ophthalmologic image processing system 100 is configured such that a server device 10 and terminal devices 201 to 20n (hereinafter, these may be collectively referred to as terminal devices 20) can be connected to each other via a communication network 30. For example, the server device 10 in Fig. 1 may function as an ophthalmologic image processing device, and any one of the multiple terminal devices 201 to 20n may connect to the server device 10 functioning as the ophthalmologic image processing device via the communication network for use. In this case, a configuration may be used in which a program for using the ophthalmologic image processing device is installed in the terminal device 20, or a configuration may be used in which a program on the server is used via a browser.
[0030] In addition, all of the components of the ophthalmologic image processing device described below do not need to be included in the same device, and some of the components may be included in other devices. For example, some of the components may be included in either the server device 10 or the multiple terminal devices 201 to 20n that can be connected via a communication network, and the ophthalmologic image processing device may use the components included in the other devices while communicating with the ophthalmologic image processing device. The server device 10 is not limited to a single device, and multiple server devices may be used. The trained model described later may be stored in the device as the ophthalmologic image processing device, or may be distributed and included in the server device 10, the multiple terminal devices 201 to 20n, etc. as other devices, and may be connected to a device that includes the trained model to be used via a communication network each time. In other words, as long as the trained model stored in some storage means can be used, it does not matter whether the trained model storage means is included in the ophthalmologic image processing device itself or in another device.
[0031] 2 is a block diagram showing an example of the configuration of an ophthalmological image processing device and peripheral devices corresponding to at least one of the embodiments of the present invention. In the following description, in order to describe an example in which the terminal device 20 functions as an ophthalmological image processing device, the same reference numerals as those of the terminal device, such as the ophthalmological image processing device 20, are used, but the present invention is not limited to this.
[0032] The ophthalmological image processing device 20 may be a device designed as a dedicated machine, but is assumed to be realizable by a general computer. That is, as shown in FIG. 2, the ophthalmological image processing device 20 includes at least a CPU (Central Processing Unit) 2001, a GPU (Graphics Processing Unit) 2002, and a memory 2003, which would normally be included in a general computer. The ophthalmological image processing device 20 is also connected to an imaging device 40 and a display device 50 via an interface (I / F) 2003. The ophthalmological image processing device 20 may also be configured such that an input device such as a mouse or keyboard, an output device such as a printer, and a communication device for connecting to a communication network are connected via a bus. The processing in each part of the ophthalmological image processing device 20 is realized by reading a program for executing the processing in each part from the memory and executing it in the CPU or GPU that functions as a control circuit (processing circuit, processing circuitry). In other words, the processor (processing circuit) is configured to execute each processing of each device by executing the program. The photographing device 40 is used to photograph an ophthalmologic image, and the display device 50 is used, for example, to display an image showing a prediction result superimposed on the ophthalmologic image.
[0033] FIG. 3 is an explanatory diagram showing an example of the configuration of an imaging device corresponding to at least one of the embodiments of the present invention. The imaging device 40 may be any device capable of recording an image of the subject's eye as digital data, and may be any device known in the art. For example, as shown in FIG. 3, the imaging device 40 emits light from a light source 41 composed of a white LED or the like, and the light passing through an aperture passes through a lens 42, a splitter 43, and an objective lens 44, and is focused on an anterior segment 45 of the subject's eye. The reflected light from the anterior segment 45 passes through the objective lens 44 and the splitter 43, and is imaged on an imaging element 47 through an imaging lens 46. The imaging data imaged on the imaging element 47 is subjected to a predetermined process by an image processing engine, and is converted into image data and video image data. The ophthalmic image captured by the imaging device 40 is transmitted to an ophthalmic image processing device 20. In the following, there will be cases where the term "image" is used simply to refer to ophthalmic images or small area images, but this term is used to refer not only to the content of the image, but also to the data of that image.
[0034] The configuration of the ophthalmological image processing device 20 will be described as an ophthalmological image processing device 20A in the first embodiment with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of an ophthalmological image processing device corresponding to at least one of the embodiments of the present invention. As shown in Fig. 4, the ophthalmological image processing device 20A includes an image acquisition unit 21, an extraction unit 22, a prediction unit 23, and a storage unit 24.
[0035] The image acquisition unit 21 has a function of acquiring an ophthalmic image. Here, the ophthalmic image refers to an image obtained by photographing the subject's eye. The ophthalmic image is photographed with a photographing range and a photographing method that can evaluate the state of the subject's eye to be evaluated. The state of the subject's eye refers to a specific state that can be used to evaluate the subject's eye, such as the occurrence of interference fringes in the tear film and the presence or absence of breaks in the tear film. In addition, the ophthalmic image is not limited to a photographed image of the surface of the subject's eye, and images obtained by a fundus camera, SLO (scanning fundus camera), OCT (optical coherence tomography), etc. can also be adopted as the ophthalmic image of this example. The state of the subject's eye in the case of a fundus camera and SLO can be, for example, whether abnormalities in the course of blood vessels, pigmentation due to a history of bleeding, abnormalities in the shape of the optic nerve, accumulation of lipids, the presence of lacunae, etc. are normal or abnormal. In addition, in the case of OCT, the condition of the subject's eye can be considered to be the above-mentioned examples in the case of the fundus camera and SLO, as well as the presence of edema, abnormalities in the layer structure of the tomography, thickness of the photoreceptor part of the retina (thinning occurs when abnormal), etc. The ophthalmologic images taken by the photographing device 40 are acquired by this image acquisition unit 21.
[0036] The extraction unit 22 has a function of extracting a plurality of small section images from an ophthalmic image. Here, a small section image refers to an image consisting of an area smaller than the ophthalmic image and having a size that serves as a unit for performing prediction processing described later. The size of the small section image does not necessarily need to be a fixed size and can be set appropriately, but problems arise at both extreme sizes, such as an extremely small size making it impossible to detect the state of the subject's eye, and an extremely large size detecting an image that includes many areas other than the state of the subject's eye as the corresponding image. Therefore, it is preferable to set the size so that the state of the subject's eye can be appropriately detected.
[0037] The prediction unit 23 has a function of predicting the state of the subject's eye for each small section image based on a trained model that has been trained in advance. The training process of the trained model is performed by extracting a plurality of small section images from a training ophthalmic image and predicting the state of the subject's eye for each small section image by machine learning using correct answer data related to the state of each small section image. The trained model can be various models that perform training by machine learning, for example, training a neural network by deep learning. As another example, a convolutional neural network (CNN) can be adopted. The size of the small section image used for training and the test small section image extracted by the extraction unit 22 must be the same size. The prediction unit 23 inputs the test small section image extracted by the extraction unit 22 into the trained model, predicts whether or not it has a feature corresponding to the state of the subject's eye, and outputs the prediction result. Details of the training process will be described later.
[0038] The storage unit 24 has a function of storing information necessary for the processing of each unit in the ophthalmologic image-processing device 20A, and also storing various information generated by the processing of each unit. The storage unit 24 may also store a trained model 241. Note that the trained model may be stored in a server device 10 connectable via a communication network, and the server device 10 may have the function of the prediction unit 23.
[0039] Next, a flow of a learning process corresponding to at least one of the embodiments of the present invention will be described. FIG. 5 is a flow chart showing an example of a flow of a learning process corresponding to at least one of the embodiments of the present invention. In FIG. 5, the learning process is started by acquiring an ophthalmic image for learning in the ophthalmic image processing device 20A (step S101). It is preferable to acquire a plurality of ophthalmic images. Next, the ophthalmic image processing device 20A extracts a plurality of small section images from the acquired ophthalmic image (step S102). In addition, the ophthalmic image processing device 20A acquires correct answer data for each small section image (step S103). The small section images thus acquired and their correct answer data are stored as a set of teacher data sets. Learning is performed using a plurality of such teacher data sets. Next, the ophthalmic image processing device 20A causes a neural network to execute a process of predicting the state of the subject's eye based on the input small section image using a set of teacher data sets consisting of the small section image and its correct answer data, thereby obtaining a prediction result (step S104). Furthermore, the ophthalmologic image processing device 20A calculates the loss based on the loss function using the prediction result obtained by the prediction process and the correct answer data, and updates the parameters (weight, bias, etc.) of the neural network so that the loss is reduced (step S105). Then, the ophthalmologic image processing device 20A judges whether or not a predetermined condition is satisfied (step S106). The predetermined condition may be, for example, that the number of trials of the process of updating the parameters of the neural network reaches a predetermined number, or that the value of the loss of the loss function is observed to increase after recording a minimum. If the predetermined condition is not satisfied (S106-No), the ophthalmologic image processing device 20A selects the next set of teacher data sets (step S107), returns to step S104, and executes the processes of S104 and S105 using the next teacher data set. In this way, the ophthalmologic image processing device 20A repeats the processes of S104 and S105 while changing the teacher data set until the predetermined condition is satisfied, and ends the learning process when the predetermined condition is satisfied (S106-Yes).
[0040] It is possible to appropriately set how to use the multiple sets of teacher data sets prepared in advance for learning. For example, multiple sets of teacher data sets may be divided into multiple groups of a predetermined batch size, and mini-batch learning may be performed in which processing for each group is treated as a division of learning. In addition, by performing a learning process for all groups in mini-batch learning, all teacher data sets are used only once, but it is also possible to set the number of epochs to a predetermined size so that learning is performed using one set of teacher data sets multiple times (the number set as the number of epochs). In addition to learning a neural network from scratch, transfer learning may be performed by repurposing a trained model used for image recognition other than ophthalmological images.
[0041] Next, a flow of a prediction process corresponding to at least one of the embodiments of the present invention will be described. FIG. 6 is a flow chart diagram showing an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. In FIG. 6, the prediction process is started by acquiring a test ophthalmic image in the ophthalmic image processing device 20A (step S201). Next, the ophthalmic image processing device 20A extracts a plurality of small section images from the acquired ophthalmic image (step S202). Next, the ophthalmic image processing device 20A inputs the small section images to a trained model that has previously trained to predict the state of the subject eye, executes a prediction process, and obtains a prediction result (step S203). Then, the ophthalmic image processing device 20A judges whether the prediction process has been completed for all the small section images (step S204). If the prediction process has not been completed for all the small section images (S204-No), the ophthalmic image processing device 20A selects the next small section image (step S205) and returns to step S203. In this manner, prediction processing is sequentially performed on the small section images, and when prediction processing has been completed for all small section images (S204-Yes), the ophthalmic image processing device 20A superimposes an image showing the prediction result on the ophthalmic image (step S206) and terminates the prediction processing.
[0042] FIG. 7 is an explanatory diagram showing an example of an ophthalmological image in which an ophthalmological image before prediction processing and a prediction processing result are superimposed. FIG. 7(a) shows a case where tear break-up is learned as the state of the subject's eye to be predicted, and the left side shows an acquired ophthalmological image, and the right side shows an image showing the prediction result superimposed. The square-shaped areas are predicted areas where tear break-up has occurred. Each square is the size of a small section image. It can be seen that the tear break-up to be detected has been accurately detected. FIG. 7(b) shows a case where tear interference fringes are learned as the state of the subject's eye to be predicted, and the left side shows an acquired ophthalmological image, and the right side shows an image showing the prediction result superimposed. The square-shaped areas are predicted areas where tear interference fringes have occurred. Each square is the size of a small section image. It can be seen that the tear interference fringes to be detected have been accurately detected.
[0043] The size of the small-section image can be set to various sizes, but it is preferable that the size is such that the condition of the subject's eye can be detected with high accuracy. Therefore, the optimal size of the small-section image was verified. In this verification, as an example, an image with a size of 640 x 480 pixels (horizontal x vertical) that is a VGA type was used as the size of the ophthalmic image. First, the optimal size of the small-section image for break-up detection was verified. The sizes of the small-section images used to learn break-up detection were prepared at 10-pixel intervals from 20 pixels vertical x 20 pixels horizontal to 60 pixels vertical x 60 pixels horizontal, and learning and inference were performed using CNN (using a simple three-layer model). The lattice of the part with break-up was manually marked, and it was superimposed on the break-up part inferred with the template of each size, and the cell where the two matched was designated as TP, the part where only the manual mark was positive was designated as false negative: FN, and the part where only the inferred mark was positive was designated as false positive: FP, and the accuracy rate (R) was calculated as R = TP / (TP + FP + FN).
[0044] FIG. 8 is a table showing the relationship between the size of a small block image and the detection rate of breakup. As can be seen from FIG. 8, the graph showing the calculated accuracy rate for each small block image size clearly shows that there is an effective size range. On the graph, the highest detection rate was achieved for 40 pixels x 40 pixels, and the actual size in this case is 0.60 mm on a side. Although it is an issue to be appropriately set as to what level of accuracy rate is acceptable, for example, if an accuracy rate of 60% or more is a preferable condition, it is desirable to set the small block image to be in the range of 30 x 30 pixels to 55 x 55 pixels, that is, to have an actual size of one side of approximately 0.45 mm to 0.85 mm.
[0045] Next, we verified the optimal size of the small block image for detecting tear interference fringes. In this verification, as an example, we used an image of VGA type ophthalmic image size of 640 × 480 pixels (horizontal × vertical). The size of the small block image used to learn the detection of stripes was prepared at 16 pixel intervals, from 16 pixels (vertical × horizontal) to 80 pixels (vertical × horizontal) and 80 pixels (horizontal), and learning and inference were performed using CNN. Regarding the detection of stripes, the tear interference fringes of healthy subjects are grayish stripes, but the color (hue) of the interference fringes of healthy subjects and evaporative dry eye patients is very similar, and it is important that the CNN model can detect the stripes themselves to improve the accuracy of stripe detection. Therefore, in order to determine whether a template of a certain size is suitable for detecting stripes in an interference fringe image of a healthy subject, it is necessary to subtract the proportion of images of the same hue that are regarded as healthy subject images. Therefore, we prepared images of tear interference fringes of a healthy subject with clear stripes and images in which the stripes were obscured by blurring, calculated the apparent stripe detection rate for the original image and the blurred image for each template size, and calculated the true stripe detection rate based on these. Specifically, the true stripe detection rate (F) was calculated using the apparent stripe detection rate for the original image (f1) and the apparent stripe detection rate for the blurred image (f2), as F = (f1-f2) / f1.
[0046] Fig. 9 is a table showing the relationship between the size of the small section image and the detection rate of tear interference fringes. By increasing the template size, an increase in f1 and a decrease in f2 were observed, and as a result, as shown in Fig. 9, it was observed that the larger the size of the small section image, the larger the calculated F. Although it is an issue to be appropriately set as to what level of detection rate is acceptable, for example, if a correct answer rate of 60% or more is a preferable condition, it is desirable to set the small section image to a range of 25 x 25 pixels or more, that is, one side of the actual size is approximately 0.35 mm or more.
[0047] 8 and 9, when the size of the small section image is determined based on the prediction accuracy when predicting the state of the subject's eye, a determination method may be considered in which the relationship between the size of the small section image and the prediction accuracy rate is calculated in advance, and the size is set to a predetermined accuracy rate or higher. Also, a determination method may be considered in which the size of the small section image is set based on the actual size of the small section image relative to the surface area of the eyeball, based on the viewpoint of what size the state of the subject's eye should be observed at on the actual eyeball surface or fundus in order to accurately detect the target state.
[0048] As described above, one aspect of the first embodiment includes an image acquisition unit that acquires an ophthalmic image to be evaluated, an extraction unit that extracts multiple small section images from the ophthalmic image, and a prediction unit that extracts multiple small section images from a learning ophthalmic image and predicts the condition of the test eye for each small section image based on a trained model that has been pre-trained to predict the condition of the test eye for each small section image through machine learning using correct answer data regarding the state of each small section image.The small section images are extracted from the ophthalmic image so that they have an image size that corresponds to the state of the test eye to be evaluated, so that the condition of the test eye can be predicted with high accuracy, and since prediction results are obtained for each small section image, it is possible to analyze the condition of the test eye in detail.
[0049] In other words, by configuring the system to extract multiple small region images from an ophthalmic image and perform prediction processing for each small region image as in this example, the following advantages can be obtained: (1) even targets that exist in only a small portion of the image can be extracted with high sensitivity and accuracy; (2) it is possible to show which part of the image the target structure is located in, thereby providing high diagnostic support capabilities; (3) a single image is divided into small regions to obtain multiple small region images, making it easy to collect a large number of samples to improve prediction accuracy; and (4) a model that is highly robust against noise can be constructed by learning that eyelashes, contours, etc. are not targets to be extracted.
[0050] [Second embodiment] Hereinafter, an example of an ophthalmologic image processing system according to a second embodiment of the present invention will be described with reference to the drawings. As in the first embodiment, the ophthalmologic image processing system of the present invention can also be realized as a standalone ophthalmologic image processing device that does not need to be connected to other devices via a communication network, so in the following description, it will be described as an ophthalmologic image processing device. Note that the same components as those in the first embodiment will be designated by the same reference numerals and their description may be omitted.
[0051] 10 is a block diagram showing an example of a configuration of an ophthalmologic image processing device corresponding to at least one of the embodiments of the present invention. As shown in FIG. 10, an ophthalmologic image processing device 20B includes an image acquisition unit 21, a result reflection region setting unit 25, an extraction unit 22B, a prediction unit 23B, and a storage unit 24.
[0052] The result reflection area setting unit 25 has a function of setting a result reflection area having a predetermined size equal to or smaller than the size of the small section image, for reflecting the prediction result of the small section image. The result reflection area refers to an area for finally reflecting the prediction result regarding the state of the subject's eye with respect to the ophthalmic image. In the first embodiment, the range of the small section image used for the prediction process and the range for reflecting the prediction result are the same area, but in this second embodiment, a result reflection area having a size equal to or smaller than the size of the small section image used for the prediction process is set. That is, a relatively large image is used for the prediction process, but a relatively small result reflection area is used as the area for reflecting the result. The result reflection area may be set based on any regularity, but it is preferable to fill the ophthalmic image with multiple result reflection areas without gaps, for example. This result reflection area is an area set in a one-to-one correspondence with the small section image. The set result reflection area is stored in the storage unit 24.
[0053] The extraction unit 22B has a function of extracting an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area as a small-partition image, corresponding to each result reflection area. The positional relationship between the small-partition image and the result reflection area may be any as long as the result reflection area is included. For example, the small-partition image may be extracted so that the result reflection area is located at the center of the small-partition image. As an example, the result reflection area may be set to a size of 20 pixels vertical by 20 pixels horizontal, and the small-partition image may be extracted to a size of 48 pixels vertical by 48 pixels horizontal so that the result reflection area is located at the center. Since the small-partition image with a size equal to or larger than the result reflection area is extracted based on the set result reflection area, the two small-partition images corresponding to the two adjacent result reflection areas may include overlapping pixel areas. The extracted small-partition images are stored in the storage unit 24.
[0054] The prediction unit 23B has a function of predicting the state of the subject's eye for each small section image based on a trained model that has been trained in advance, and further has a function of reflecting the prediction result obtained for each small section image in the corresponding result reflection area. When multiple result reflection areas are set for one ophthalmic image, the same number of small section images as the result reflection areas are extracted, so that the prediction process is performed sequentially for the multiple small section images, and the obtained prediction results are reflected in the corresponding result reflection areas. The reflection results obtained for the result reflection areas are stored in the storage unit 24. Alternatively, an image showing the prediction result for each result reflection area may be superimposed on the ophthalmic image and output to the display device 50. Note that the learning process for obtaining the trained model used in the prediction unit 23B is the same as that in the first embodiment, so a description thereof will be omitted.
[0055] Next, a flow of a prediction process corresponding to at least one of the embodiments of the present invention will be described. FIG. 11 is a flow chart diagram showing an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. In FIG. 11, the prediction process is started by acquiring a test ophthalmic image in the ophthalmic image processing device 20B (step S301). Next, the ophthalmic image processing device 20B sets a plurality of result reflection areas for the acquired ophthalmic image (step S302). In addition, the ophthalmic image processing device 20B extracts small section images corresponding to each result reflection area (step S303). Next, the ophthalmic image processing device 20B inputs the small section images to a trained model that has previously trained to predict the state of the subject eye, and executes a prediction process to obtain a prediction result (step S304). In addition, the ophthalmic image processing device 20B reflects the prediction result obtained for the small section image in the corresponding result reflection area (step S305). Then, the ophthalmic image processing device 20B determines whether the prediction process has been completed for all small section images (step S306). If the prediction process has not been completed for all small section images (S306-No), the ophthalmologic image processing device 20B selects the next small section image (step S307) and returns to step S304. In this manner, the prediction process is sequentially performed for the small section images, and if the prediction process has been completed for all small section images (S306-Yes), the ophthalmologic image processing device 20B superimposes an image showing the prediction result on the ophthalmologic image (step S308) and ends the prediction process.
[0056] FIG. 12 is an explanatory diagram for explaining an example of setting a result reflection area and a small partition image. As an example, it is assumed that result reflection areas A, B, and C each having a size of 20 pixels vertical by 20 pixels horizontal are set adjacent to each other. In this situation, it is assumed that small partition images A, B, and C corresponding to the result reflection areas A, B, and C, respectively, are extracted with a size of 48 pixels vertical by 48 pixels horizontal so that the result reflection areas are located at the center. As shown in FIG. 12(a), it can be seen that the small partition image A is extracted so that the result reflection area A is located at the center. In addition, it can be seen that the small partition image B corresponding to the adjacent result reflection area B is extracted in a form including a pixel area overlapping with the small partition image A, as shown in FIG. 12(b).
[0057] As described above, as one aspect of the second embodiment, the display further includes a result reflection area setting unit that sets a result reflection area having a predetermined size equal to or smaller than the size of the small-partition image and reflects the prediction result of the small-partition image, and the extraction unit extracts an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area as a small-partition image in correspondence with each result reflection area, and the prediction unit reflects the prediction result obtained for each small-partition image in the corresponding result reflection area. Therefore, a large small-partition image size is set for the process of predicting the state of the subject's eye so as to improve the accuracy of prediction, and the result reflection area that reflects the prediction result is set to a small size, thereby obtaining a result image with high resolution while maintaining prediction accuracy. That is, when an image showing the prediction result is superimposed on an ophthalmic image, if the prediction result based on a large-sized small-partition image is superimposed as it is at the size of the small-partition image, a result display screen with low resolution will be obtained, but by reflecting the prediction result in a result reflection area smaller than the small-partition image as in this example, it is possible to obtain a result image with high resolution while maintaining prediction accuracy.
[0058] [Third embodiment] Hereinafter, an example of an ophthalmologic image processing system according to the third embodiment of the present invention will be described with reference to the drawings. As in the first and second embodiments, the ophthalmologic image processing system of the present invention can also be realized as a standalone ophthalmologic image processing device that does not need to be connected to other devices via a communication network, so in the following description, it will be described as an ophthalmologic image processing device. Note that the same reference numerals are used for the same configurations as the first and second embodiments, and the description may be omitted.
[0059] 13 is a block diagram showing an example of a configuration of an ophthalmologic image processing device corresponding to at least one of the embodiments of the present invention. As shown in FIG. 13, an ophthalmologic image processing device 20C includes an image acquisition unit 21, a result reflection region setting unit 25, an extraction unit 22C, an edge image generation unit 26, a prediction unit 23C, and a storage unit 24.
[0060] The extraction unit 22C has a function of extracting, in correspondence with each result reflection area, an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area as a small block image.
[0061] The edge image generating unit 26 has a function of extracting edges from an ophthalmic image to generate an edge image. Here, the edge image refers to an image in which a portion in an ophthalmic image where a change in luminance value is large is extracted as an edge. The edge image is an image that indicates which portion of the ophthalmic image has an edge, and therefore, in principle, has the same size as the ophthalmic image. The edge image is an image generated such that, for example, the portion extracted as an edge has a high luminance value (the stronger the edge, the higher the luminance value), and the portion that is not an edge has a low luminance value. In this example, the edge image is used to improve the efficiency of the prediction process on the premise that the portion extracted as an edge is a place that is highly likely to be an object to be extracted as the state of the subject's eye. Any method may be used as long as it is possible to generate an edge image, and for example, a known Sobel filter may be used in edge image generation. The generated edge image is stored in the storage unit 24.
[0062] The prediction unit 23C has a function of predicting the state of the subject's eye for each small section image based on a trained model that has been trained in advance, and a function of setting the small section image corresponding to the result reflection area as a target for prediction processing when the luminance value in the edge image corresponding to the position of the result reflection area is equal to or greater than a predetermined threshold, and further has a function of reflecting the prediction result obtained for each small section image in the corresponding result reflection area. That is, if the prediction processing is performed for all the result reflection areas set in the result reflection area setting unit 25, it takes a long processing time to complete the prediction processing for one ophthalmic image. In addition, the higher the resolution of the ophthalmic image, the longer the processing time. In order to shorten this processing time, it is determined that a portion that does not include an edge image in the result determination area is highly likely not to include a target to be extracted as the state of the subject's eye, and is excluded from the target of the prediction processing. That is, only the portion that includes an edge image in the result determination area is set as a target for the prediction processing. By performing such processing, it is possible to shorten the processing time required for the prediction processing.
[0063] Next, a flow of a prediction process corresponding to at least one of the embodiments of the present invention will be described. FIG. 14 is a flow chart showing an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. In FIG. 14, the prediction process is started by acquiring a test ophthalmic image in the ophthalmic image processing device 20C (step S401). Next, the ophthalmic image processing device 20C sets a plurality of result reflection areas for the acquired ophthalmic image (step S402). In addition, the ophthalmic image processing device 20C extracts small section images corresponding to each result reflection area (step S403). Next, the ophthalmic image processing device 20C generates an edge image from the ophthalmic image (step S404). Then, for those in the edge image corresponding to the position of the result reflection area whose brightness value is equal to or greater than a predetermined threshold, the ophthalmic image processing device 20C sets the small section image corresponding to the result reflection area as a target for prediction process (step S405). Next, the ophthalmological image processing device 20C inputs the small-partition image to be subjected to prediction processing into a trained model that has been trained in advance to predict the state of the subject's eye, and executes the prediction processing to obtain a prediction result (step S406). The ophthalmological image processing device 20C also reflects the prediction result obtained for the small-partition image in the corresponding result reflection area (step S407). Then, the ophthalmological image processing device 20C judges whether the prediction processing is completed for all the small-partition images to be subjected to prediction processing (step S408). If the prediction processing is not completed for all the small-partition images to be subjected to prediction processing (S408-No), the ophthalmological image processing device 20C selects the next small-partition image to be subjected to prediction processing (step S409) and returns to step S406. In this way, the prediction processing is sequentially executed for the small-partition images to be subjected to prediction processing, and if the prediction processing is completed for all the small-partition images to be subjected to prediction processing (S408-Yes), the ophthalmological image processing device 20C superimposes an image showing the prediction result on the ophthalmological image (step S410) and ends the prediction processing.
[0064] As described above, as one aspect of the third embodiment, the prediction unit further includes an edge image generation unit that extracts edges from an ophthalmic image and generates an edge image, and when the brightness value in the edge image corresponding to the position of the result reflection area is equal to or greater than a predetermined threshold, the small section image corresponding to the result reflection area is set as the target for prediction processing.Therefore, only result reflection areas that include edges with a predetermined brightness value or greater that are likely to contain the target to be extracted as the condition of the test eye are extracted, and only the small section images corresponding to the extracted result reflection areas can be set as the target for prediction processing, thereby shortening the processing time required for prediction processing.
[0065] [Fourth embodiment] In addition to the configurations of the first to third embodiments, the prediction unit 23 may calculate an average brightness value for each small section image, and set small section images whose average brightness value is equal to or greater than a predetermined threshold value as targets for prediction processing.
[0066] It can be said that the parts of the ophthalmic image that are not necessary for predicting the state of the subject's eye, such as the parts outside the field of view and the parts of the eyelashes, are black (dark) and have a very low brightness value. Since such parts obviously do not include the object to be extracted as the state of the subject's eye, if such parts can be excluded from the target of the prediction process, the processing time required for the prediction process can be shortened. Therefore, the prediction unit 23 calculates the average brightness value for each small section image, sets the small section image whose average brightness value is equal to or greater than a predetermined threshold value as the target of the prediction process, and excludes the small section image whose average brightness value is less than the predetermined threshold value from the target of the prediction process, thereby shortening the processing time required for the prediction process.
[0067] In the fourth embodiment, by setting small-partition images whose average brightness value is less than a predetermined threshold as excluded from the prediction process, areas outside the field of view and eyelashes are removed, and further, by training the trained model to predict that eyelashes, contours, etc. are excluded from extraction targets during the learning stage, it becomes possible to determine that small-partition images that are mainly composed of eyelashes or contours and that exist among small-partition images whose average brightness value is equal to or greater than a predetermined threshold are excluded from the target (predict to classify as "excluded" or "other") during prediction processing in prediction unit 23. In this way, by removing and excluding unnecessary areas such as areas outside the field of view and eyelashes by two means, it is expected that the prediction speed will be further increased and the prediction accuracy will be further improved.
[0068] [Fifth embodiment] In addition to the configurations of the first to third embodiments, an edge image may be generated from an ophthalmic image in the edge image generation unit, and the edge image may be used to determine whether or not to set a small section image as a target for prediction processing.
[0069] According to the present invention, by dividing an ophthalmological image that is the target of prediction processing into small sections, it is possible to identify and display the position of a structure of interest in the image. However, this has the problem that inference must be performed on many small section images, which takes a long time to analyze. On the other hand, many of the generated small section images do not require inference, and if these can be excluded from the inference target in advance, it is expected that the inference time will be significantly shortened. In many cases, the structure of interest in an image has a clear outline or is densely packed with complex structures, so if such parts can be extracted by some method, it can be determined whether an arbitrary small section image is a target for inference or not.
[0070] Therefore, an edge image is generated from an ophthalmologic image by adopting a configuration similar to that of the edge image generating unit 26 described in the third embodiment. Then, the prediction unit 23 judges whether or not the edge image corresponding to the position of each small block image includes an edge portion. Specifically, it is determined whether or not to set the edge image corresponding to the position of the small block image as a target for the measurement process depending on the intensity of brightness of the edge image. Various conditions can be set for the intensity of brightness of the edge image, such as the average value, maximum value, median value, and even standard deviation of the brightness value. In this way, if the small block image includes an edge, it is estimated that the image is a small block image having a noteworthy structure, and the prediction process is performed. If the small block image does not include an edge, the small block image is set as not being a target for the prediction process. This allows the number of time-consuming inferences to be significantly reduced, thereby shortening the analysis time.
[0071] In addition, by setting small-partition images that do not contain edges as excluded from the prediction process using the configuration of the fifth embodiment, areas outside the field of view and eyelashes are removed, and further, by training the trained model to predict that eyelashes, contours, etc. are excluded from extraction targets during the learning stage, it becomes possible to determine that small-partition images that are mainly composed of eyelashes or contours and that exist within small-partition images that contained edges are excluded from the prediction process (predict to classify as "excluded" or "other") during prediction processing in prediction unit 23. In this way, by removing and excluding unnecessary areas such as areas outside the field of view and eyelashes using two means, it is expected that the prediction speed will be further increased and the prediction accuracy will be further improved.
[0072] [Sixth embodiment] In the first to fifth embodiments, the case where the prediction process is performed on one ophthalmic image has been described, but the present invention is not limited thereto. That is, the image acquisition unit 21 may acquire a plurality of ophthalmic images arranged in time series, the prediction unit 23 may execute the prediction process on each of the plurality of ophthalmic images, and a change determination unit may be further provided that determines whether or not the temporal and / or spatial change in the state of the subject's eye satisfies a predetermined condition for the prediction result obtained for each of the plurality of ophthalmic images.
[0073] When evaluating the condition of the subject's eye, some information can be obtained by evaluating one ophthalmic image, but some information can be obtained by considering the temporal and spatial changes of the prediction results of multiple ophthalmic images arranged in a time series. For example, as an example of temporal changes, it is possible to evaluate the possibility of a specific disease by evaluating how much time has passed since blinking when the condition of the subject's eye is detected. In this case, it is possible to calculate the number of small section images predicted as the condition of the subject's eye for each ophthalmic image, and the ratio of the condition of the subject's eye in the entire ophthalmic image, and to evaluate the extent to which the number and ratio of the condition of the subject's eye exceed a predetermined threshold when tracking the temporal changes. It is also possible to detect that the gradient of the temporal change in the ratio of the condition of the subject's eye is equal to or greater than a predetermined condition. It is also possible to evaluate, as an example of spatial changes, how the location where the condition of the subject's eye is detected moves.
[0074] In this way, by determining whether the temporal and / or spatial changes in the condition of the test eye satisfy specified conditions, it is possible to evaluate information that cannot be evaluated based on only the prediction results for a single ophthalmic image.
[0075] [Seventh embodiment] Hereinafter, an example of an ophthalmologic image processing system according to a seventh embodiment of the present invention will be described with reference to the drawings. As will be described later, the ophthalmologic image processing system of the present invention can also be realized as a standalone ophthalmologic image processing device that does not need to be connected to other devices via a communication network, so it does not necessarily have to be an ophthalmologic image processing system. The following description will be given on the assumption that it may function as an ophthalmologic image processing system.
[0076] Fig. 15 is a block diagram showing an example of a configuration of an ophthalmologic image processing system corresponding to at least one of the embodiments of the present invention. As shown in Fig. 15, the ophthalmologic image processing system 100 is configured such that a server device 10 and terminal devices 201 to 20n (hereinafter, these may be collectively referred to as terminal devices 20) can be connected to each other via a communication network 30. For example, the server device 10 in Fig. 15 may function as an ophthalmologic image processing device, and any one of the multiple terminal devices 201 to 20n may connect to the server device 10 functioning as the ophthalmologic image processing device via the communication network for use. In this case, a configuration may be used in which a program for using the ophthalmologic image processing device is installed in the terminal device 20, or a configuration may be used in which a program on the server is used via a browser.
[0077] In addition, all of the components of the ophthalmologic image processing device described below do not need to be included in the same device, and some of the components may be included in other devices. For example, some of the components may be included in either the server device 10 or the multiple terminal devices 201 to 20n that can be connected via a communication network, and the ophthalmologic image processing device may use the components included in the other devices while communicating with the ophthalmologic image processing device. The server device 10 is not limited to a single device, and multiple server devices may be used. The trained model described later may be stored in the device as the ophthalmologic image processing device, or may be distributed and included in the server device 10, the multiple terminal devices 201 to 20n, etc. as other devices, and may be connected to a device that includes the trained model to be used via a communication network each time. In other words, as long as the trained model stored in some storage means can be used, it does not matter whether the trained model storage means is included in the ophthalmologic image processing device itself or in another device.
[0078] 16 is a block diagram showing an example of the configuration of an ophthalmological image processing device and peripheral devices corresponding to at least one of the embodiments of the present invention. In the following description, in order to describe an example in which the terminal device 20 functions as an ophthalmological image processing device, the same reference numerals as those of the terminal device, such as the ophthalmological image processing device 20, are used, but the present invention is not limited to this.
[0079] The ophthalmological image processing device 20 may be a device designed as a dedicated machine, but is assumed to be realizable by a general computer. That is, as shown in FIG. 16, the ophthalmological image processing device 20 is preferably equipped with at least a CPU (Central Processing Unit) 2001 and a memory 2003 that would normally be equipped in a general computer, and further includes a GPU (Graphics Processing Unit) 2002. The ophthalmological image processing device 20 is connected to an imaging device 40 and a display device 50 via an interface (I / F) 2003. The ophthalmological image processing device 20 may also be configured such that an input device such as a mouse or keyboard, an output device such as a printer, and a communication device for connecting to a communication network are connected via a bus. The processing in each part of the ophthalmological image processing device 20 is realized by reading a program for executing the processing in each part from the memory and executing it in the CPU or GPU that functions as a control circuit (processing circuit, processing circuitry). In other words, the processor (processing circuit) is configured to execute each processing of each device by executing the program. The photographing device 40 is used to photograph an ophthalmic image, and the display device 50 is used to display, for example, an image showing a prediction result superimposed on the ophthalmic image. The ophthalmic image processing device 20 can be any device capable of executing processing equivalent to that of a computer, and can be realized, for example, by a smartphone or a tablet terminal.
[0080] FIG. 17 is an explanatory diagram showing an example of the configuration of an imaging device corresponding to at least one of the embodiments of the present invention. The imaging device 40 may be any device capable of recording an image of the subject's eye as digital data, and may be any device known in the art. For example, as shown in FIG. 17, the imaging device 40 emits light from a light source 41 composed of a white LED or the like, and the light passing through an aperture passes through a lens 42, a splitter 43, and an objective lens 44, and is focused on an anterior segment 45 of the subject's eye. The reflected light from the anterior segment 45 passes through the objective lens 44 and the splitter 43, and is imaged on an imaging element 47 through an imaging lens 46. The imaging data imaged on the imaging element 47 is subjected to a predetermined process by an image processing engine, and is converted into image data and video image data. The ophthalmic image captured by the imaging device 40 is transmitted to an ophthalmic image processing device 20. In the following, there will be cases where the term "image" is used simply to refer to ophthalmic images or small area images, but this term is used to refer not only to the content of the image, but also to the data of that image.
[0081] FIG. 18 is an explanatory diagram showing the state of dry eye of a test eye as an evaluation target in an ophthalmologic image processing device corresponding to at least one of the embodiments of the present invention. As shown in FIG. 18, the ophthalmologic image processing device of the present invention is configured to be able to predict at least three states of dry eye of a test eye, namely (a) a state of a healthy person (a state without dry eye), (b) a state of evaporative dry eye, and (c) a state of aqueous tear deficient dry eye. Here, (a) the state of a healthy person is a healthy state in which the lipid layer and the liquid layer of tears are well balanced, and is characterized by the observation of a slightly dark gray interference fringe with low saturation. (b) The state of evaporative dry eye is a state in which the evaporation of tears is accelerated, and is characterized by the observation of a dark and smooth state. (c) The state of aqueous tear deficient dry eye is a state in which the amount of tears secreted is reduced, and is characterized by the observation of an interference fringe with high saturation. In addition, (a) the condition of a healthy person looks like a pearl, so it can also be described as P (Pearl) type, (b) the condition of evaporative dry eye looks like crystal, so it can be described as C (Crystal) type, and (c) the condition of aqueous-deficient dry eye looks like Jupiter, so it can also be described as J (Jupiter) type.
[0082] The configuration of the ophthalmological image processing device 20 will be described as an ophthalmological image processing device 20A in the seventh embodiment with reference to Fig. 19. Fig. 19 is a block diagram showing an example of the configuration of an ophthalmological image processing device corresponding to at least one of the embodiments of the present invention. As shown in Fig. 19, the ophthalmological image processing device 20A includes an image acquisition unit 21, an extraction unit 22, a prediction unit 23, and a storage unit 24.
[0083] The image acquisition unit 21 has a function of acquiring ophthalmic images. Here, the ophthalmic images refer to images acquired by photographing the subject's eye. The ophthalmic images are captured with a photographing range and photographing method that can evaluate the dry eye condition of the subject's eye to be evaluated. The dry eye condition of the subject's eye refers to a specific condition that can be used to evaluate the dry eye of the subject's eye, and specifically includes at least three conditions: (a) a healthy subject's condition (a condition that is not dry eye), (b) an evaporative dry eye condition, and (c) an aqueous deficient dry eye condition. The ophthalmic images captured by the photographing device 40 are acquired by the image acquisition unit 21.
[0084] The extraction unit 22 has a function of extracting a plurality of small section images from an ophthalmic image. Here, a small section image refers to an image consisting of an area smaller than an ophthalmic image and having a size that serves as a unit for performing prediction processing described later. The size of the small section image does not necessarily need to be a fixed size and can be set appropriately, but problems arise at both extreme sizes, such as an extremely small size making it impossible to detect the dry eye state of the subject's eye, and an extremely large size detecting an image that includes many areas other than the dry eye state of the subject's eye as the corresponding image. Therefore, it is preferable to set the size so that the dry eye state of the subject's eye can be appropriately detected.
[0085] The prediction unit 23 has a function of predicting the dry eye state of the subject's eye for each small section image based on a trained model that has been trained in advance. The training process of the trained model is performed by extracting a plurality of small section images from the training ophthalmic image and predicting the dry eye state of the subject's eye for each small section image by machine learning using correct answer data regarding the state of each small section image. The trained model can be various models that learn by machine learning, and for example, a neural network trained by deep learning is applicable. Furthermore, as one example, a convolutional neural network (CNN) can be adopted. The size of the small section image used for training and the test small section image extracted by the extraction unit 22 must be the same size. The prediction unit 23 inputs the test small section image extracted by the extraction unit 22 into the trained model, predicts whether or not the small section image has a feature corresponding to the dry eye state of the subject's eye, and outputs the prediction result. Details of the training process will be described later.
[0086] The storage unit 24 has a function of storing information necessary for the processing of each unit in the ophthalmologic image-processing device 20A, and also storing various information generated by the processing of each unit. The storage unit 24 may also store a trained model 241. Note that the trained model may be stored in a server device 10 connectable via a communication network, and the server device 10 may have the function of the prediction unit 23.
[0087] Next, a flow of a learning process corresponding to at least one of the embodiments of the present invention will be described. FIG. 20 is a flow chart diagram showing an example of a flow of a learning process corresponding to at least one of the embodiments of the present invention. In FIG. 20, the learning process is started by acquiring an ophthalmic image for learning in the ophthalmic image processing device 20A (step S101). It is preferable to acquire a plurality of ophthalmic images. Next, the ophthalmic image processing device 20A extracts a plurality of small section images from the acquired ophthalmic image (step S102). In addition, the ophthalmic image processing device 20A acquires correct answer data for each small section image (step S103). The small section images thus acquired and their correct answer data are stored as a set of teacher data sets. Learning is performed using a plurality of such teacher data sets. Next, the ophthalmic image processing device 20A causes a neural network to execute a process of predicting the state of dry eye of the subject's eye based on the input small section image using a set of teacher data sets consisting of the small section image and its correct answer data, and obtains a prediction result (step S104). Furthermore, the ophthalmologic image processing device 20A calculates the loss based on the loss function using the prediction result obtained by the prediction process and the correct answer data, and updates the parameters (weight, bias, etc.) of the neural network so that the loss is reduced (step S105). Then, the ophthalmologic image processing device 20A judges whether or not a predetermined condition is satisfied (step S106). The predetermined condition may be, for example, that the number of trials of the process of updating the parameters of the neural network reaches a predetermined number, or that the value of the loss of the loss function is observed to increase after recording a minimum. If the predetermined condition is not satisfied (S106-No), the ophthalmologic image processing device 20A selects the next set of teacher data sets (step S107), returns to step S104, and executes the processes of S104 and S105 using the next teacher data set. In this way, the ophthalmologic image processing device 20A repeats the processes of S104 and S105 while changing the teacher data set until the predetermined condition is satisfied, and ends the learning process when the predetermined condition is satisfied (S106-Yes).
[0088] It is possible to appropriately set how to use the multiple sets of teacher data sets prepared in advance for learning. For example, multiple sets of teacher data sets may be divided into multiple groups of a predetermined batch size, and mini-batch learning may be performed in which processing for each group is treated as a division of learning. In addition, by performing a learning process for all groups in mini-batch learning, all teacher data sets are used only once, but it is also possible to set the number of epochs to a predetermined size so that learning is performed using one set of teacher data sets multiple times (the number set as the number of epochs). In addition to learning a neural network from scratch, transfer learning may be performed by repurposing a trained model used for image recognition other than ophthalmological images.
[0089] Next, a flow of a prediction process corresponding to at least one of the embodiments of the present invention will be described. FIG. 21 is a flow chart diagram showing an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. In FIG. 21, the prediction process is started by acquiring a test ophthalmic image in the ophthalmic image processing device 20A (step S201). Next, the ophthalmic image processing device 20A extracts a plurality of small section images from the acquired ophthalmic image (step S202). Next, the ophthalmic image processing device 20A inputs the small section images to a trained model that has previously trained to predict the state of dry eye of the subject eye, executes a prediction process, and obtains a prediction result (step S203). Then, the ophthalmic image processing device 20A judges whether the prediction process has been completed for all the small section images (step S204). If the prediction process has not been completed for all the small section images (S204-No), the ophthalmic image processing device 20A selects the next small section image (step S205) and returns to step S203. In this manner, prediction processing is sequentially performed on the small section images, and when prediction processing has been completed for all small section images (S204-Yes), the ophthalmic image processing device 20A superimposes an image showing the prediction result on the ophthalmic image (step S206) and terminates the prediction processing.
[0090] FIG. 22 is an explanatory diagram for explaining the relationship between the size of the small section image and whether or not stripes can be detected. The size of the small section image can be set to various sizes, but it is preferable that the size is a size that allows accurate detection of the dry eye state of the subject's eye. Therefore, the optimum size of the small section image was verified by checking the detection accuracy while changing the size of the small section image in various ways. FIG. 22(a) shows a case where the size of the small section image is set to 8 pixels vertically by 8 pixels horizontally, and the prediction accuracy was low when this size of 8 pixels x 8 pixels was used. In other words, stripes of tear interference fringes and the like could not be detected. This state is presumably because, as shown in FIG. 22(a), the size of the small section image is too small compared to the size of the stripes, and therefore the characteristics of the dry eye state including the stripes could not be extracted. Next, FIG. 22(b) shows a case where the size of the small section image is set to 24 pixels vertically by 24 pixels horizontally, and although the prediction accuracy was improved when compared to the case of the size of 8 pixels x 8 pixels in FIG. 22(a), it was not sufficient prediction accuracy. In this state, as shown in FIG. 22(b), the size of the small section image may be such that the boundary of the stripe pattern cannot be detected, so that the stripe pattern may not be predicted, and therefore the prediction accuracy is presumed to be insufficient. FIG. 22(c) shows a case where the size of the small section image is set to 48 pixels vertically by 48 pixels horizontally, and good prediction accuracy was obtained with this size of 48 pixels by 48 pixels. In this state, as shown in FIG. 22(c), it is presumed that the size of the small section image is set to a sufficient size for the size of the stripe pattern, so that the dry eye condition including the stripe pattern is unlikely to be missed. For this reason, in the present invention in which it is necessary to detect a stripe pattern as a dry eye condition of the subject's eye, it is not preferable to set the size of the small section image to a size that is too small, and it is preferable to set the size of the small section image to a size that provides a desired prediction accuracy in terms of the balance with the prediction accuracy.
[0091] In addition to the example of Fig. 22, when the size of a small section image is determined based on the prediction accuracy when predicting the dry eye condition of the subject's eye, a determination method may be considered in which the relationship between the size of the small section image and the accuracy rate of the prediction is calculated in advance, and the size is set to a predetermined accuracy rate or higher. Also, a determination method may be considered in which the size of the small section image is set based on the actual size of the small section image relative to the surface area of the eyeball, based on the viewpoint of what size should be observed on the actual eyeball surface to accurately detect the dry eye condition of the subject's eye.
[0092] FIG. 23 is an explanatory diagram showing an example of an ophthalmological image in which an ophthalmological image before prediction processing and a prediction processing result are superimposed. FIG. 23(a) shows an ophthalmological image including many aqueous-deficient dry eye conditions as an ophthalmological image to be subjected to prediction processing, and FIG. 23(b) shows an image showing a prediction result superimposed on the ophthalmological image. As shown in FIGS. 23(a) and 23(b), there are many Jupiter points indicating the predicted area of the aqueous-deficient dry eye condition, and it can be seen that prediction is performed with high accuracy. In addition, although there are many Jupiter points as a whole, it can be seen that there are also Pearl points indicating the predicted area of the healthy subject condition and Crystal points indicating the predicted area of the evaporative dry eye condition in individual small section images.
[0093] As described above, one aspect of the seventh embodiment includes an image acquisition unit that acquires the ophthalmic image to be evaluated, an extraction unit that extracts multiple small section images from the ophthalmic image, and a prediction unit that extracts multiple small section images from the learning ophthalmic image and uses correct answer data regarding the state of each small section image to predict the dry eye state of the test eye for each small section image based on a trained model that has been pre-trained by machine learning to predict whether the dry eye state of the test eye for each small section image is at least a healthy state, an aqueous-deficient dry eye state, or an evaporative dry eye state.The small section images are extracted from the ophthalmic image to have an image size that allows the dry eye state of the test eye to be evaluated to be appropriately evaluated, so that the dry eye state of the test eye can be predicted with high accuracy, and since a prediction result is obtained for each small section image, it is possible to analyze the dry eye state of the test eye in detail.
[0094] In other words, by configuring the system to extract multiple small region images from an ophthalmic image and perform prediction processing for each small region image as in this example, the following advantages can be obtained: (1) even targets that exist in only a small portion of the image can be extracted with high sensitivity and accuracy; (2) it is possible to show which part of the image the target structure is located in, thereby providing high diagnostic support capabilities; (3) a single image is divided into small regions to obtain multiple small region images, making it easy to collect a large number of samples to improve prediction accuracy; and (4) a model that is highly robust against noise can be constructed by learning that eyelashes, contours, etc. are not targets to be extracted.
[0095] [Eighth embodiment] Hereinafter, an example of an ophthalmologic image processing system according to the eighth embodiment of the present invention will be described with reference to the drawings. As in the seventh embodiment, the ophthalmologic image processing system of the present invention can also be realized as a standalone ophthalmologic image processing device that does not need to be connected to other devices via a communication network, so in the following description, it will be described as an ophthalmologic image processing device. Note that the same components as those in the seventh embodiment will be designated by the same reference numerals and their description may be omitted.
[0096] Fig. 24 is a block diagram showing an example of a configuration of an ophthalmologic image-processing device corresponding to at least one of the embodiments of the present invention. As shown in Fig. 24, an ophthalmologic image-processing device 20B includes an image acquisition unit 21, a result reflection area setting unit 25, an extraction unit 22B, a prediction unit 23B, and a storage unit 24.
[0097] The result reflection area setting unit 25 has a function of setting a result reflection area having a predetermined size equal to or smaller than the size of the small section image, for reflecting the prediction result of the small section image. The result reflection area refers to an area for finally reflecting the prediction result regarding the dry eye state of the subject's eye for the ophthalmic image. In the seventh embodiment, the range of the small section image used for the prediction process and the range for reflecting the prediction result are the same area, but in this eighth embodiment, a result reflection area having a size equal to or smaller than the size of the small section image used for the prediction process is set. That is, a relatively large image is used for the prediction process, but a relatively small result reflection area is used as the area for reflecting the result. The result reflection area may be set based on any regularity, but it is preferable to fill the ophthalmic image with multiple result reflection areas without gaps, for example. This result reflection area is an area set in a one-to-one correspondence with the small section image. The set result reflection area is stored in the storage unit 24.
[0098] The extraction unit 22B has a function of extracting an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area as a small-partition image, corresponding to each result reflection area. The positional relationship between the small-partition image and the result reflection area may be any as long as the result reflection area is included. For example, the small-partition image may be extracted so that the result reflection area is located at the center of the small-partition image. As an example, the result reflection area may be set to a size of 20 pixels vertical by 20 pixels horizontal, and the small-partition image may be extracted to a size of 48 pixels vertical by 48 pixels horizontal so that the result reflection area is located at the center. Since the small-partition image with a size equal to or larger than the result reflection area is extracted based on the set result reflection area, the two small-partition images corresponding to the two adjacent result reflection areas may include overlapping pixel areas. The extracted small-partition images are stored in the storage unit 24.
[0099] The prediction unit 23B has a function of predicting the state of dry eye of the subject's eye for each small section image based on a trained model that has been trained in advance, and further has a function of reflecting the prediction result obtained for each small section image in the corresponding result reflection area. When multiple result reflection areas are set for one ophthalmic image, the same number of small section images as the result reflection areas are extracted, so that the prediction process is performed sequentially for the multiple small section images, and the obtained prediction results are reflected in the corresponding result reflection areas. The reflection results obtained for the result reflection areas are stored in the storage unit 24. Alternatively, an image showing the prediction result for each result reflection area may be superimposed on the ophthalmic image and output to the display device 50. Note that the learning process for obtaining the trained model used in the prediction unit 23B is the same as that in the seventh embodiment, so a description thereof will be omitted.
[0100] Next, a flow of a prediction process corresponding to at least one of the embodiments of the present invention will be described. FIG. 25 is a flow chart diagram showing an example of a flow of a prediction process corresponding to at least one of the embodiments of the present invention. In FIG. 25, the prediction process is started by acquiring a test ophthalmic image in the ophthalmic image processing device 20B (step S301). Next, the ophthalmic image processing device 20B sets a plurality of result reflection areas for the acquired ophthalmic image (step S302). In addition, the ophthalmic image processing device 20B extracts small section images corresponding to each result reflection area (step S303). Next, the ophthalmic image processing device 20B inputs the small section images to a trained model that has previously trained to predict the state of dry eye of the subject's eye, and executes a prediction process to obtain a prediction result (step S304). In addition, the ophthalmic image processing device 20B reflects the prediction result obtained for the small section image in the corresponding result reflection area (step S305). Then, the ophthalmic image processing device 20B determines whether the prediction process has been completed for all small section images (step S306). If the prediction process has not been completed for all small section images (S306-No), the ophthalmologic image processing device 20B selects the next small section image (step S307) and returns to step S304. In this manner, the prediction process is sequentially performed for the small section images, and if the prediction process has been completed for all small section images (S306-Yes), the ophthalmologic image processing device 20B superimposes an image showing the prediction result on the ophthalmologic image (step S308) and ends the prediction process.
[0101] FIG. 26 is an explanatory diagram for explaining an example of setting a result reflection area and a small partition image. As an example, it is assumed that result reflection areas A, B, and C each having a size of 20 pixels vertical by 20 pixels horizontal are set adjacent to each other. In this situation, it is assumed that small partition images A, B, and C corresponding to the result reflection areas A, B, and C, respectively, are extracted with a size of 48 pixels vertical by 48 pixels horizontal so that the result reflection areas are located at the center. As shown in FIG. 26(a), it can be seen that the small partition image A is extracted so that the result reflection area A is located at the center. In addition, it can be seen that the small partition image B corresponding to the adjacent result reflection area B is extracted in a form including a pixel area overlapping with the small partition image A, as shown in FIG. 26(b).
[0102] As described above, as one aspect of the eighth embodiment, the display further includes a result reflection area setting unit that sets a result reflection area having a predetermined size equal to or smaller than the size of the small-partition image and reflects the prediction result of the small-partition image, and the extracting unit extracts an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area as a small-partition image in correspondence with each result reflection area, and the predicting unit reflects the prediction result obtained for each small-partition image in the corresponding result reflection area. Therefore, a large small-partition image size is set for the process of predicting the dry eye state of the subject's eye so as to improve the accuracy of prediction, and the result reflection area that reflects the prediction result is set to a small size, thereby obtaining a result image with high resolution while maintaining prediction accuracy. That is, when an image showing the prediction result is superimposed on an ophthalmic image, if the prediction result based on a large-sized small-partition image is superimposed as it is at the size of the small-partition image, a result display screen with low resolution will be obtained, but by reflecting the prediction result in a result reflection area smaller than the small-partition image as in this example, it is possible to obtain a result image with high resolution while maintaining prediction accuracy.
[0103] [Ninth embodiment] In addition to the configurations of the seventh and eighth embodiments, the prediction unit 23 may calculate an average brightness value for each small section image, and set small section images whose average brightness value is equal to or greater than a predetermined threshold value as targets for prediction processing.
[0104] It can be said that the parts of the ophthalmic image that are not necessary for predicting the dry eye state of the subject's eye, such as the parts outside the visual field and the parts of the eyelashes, are black (dark) and have a very low brightness value. Since it is clear that such parts do not include the target to be extracted as the dry eye state of the subject's eye, if such parts can be excluded from the target of the prediction process, the processing time required for the prediction process can be shortened. Therefore, in the prediction unit 23, the average brightness value is calculated for each small section image, and small section images whose average brightness value is equal to or greater than a predetermined threshold are set as the target of the prediction process, and small section images whose average brightness value is less than the predetermined threshold are excluded from the target of the prediction process. This makes it possible to remove parts that are not necessary for analysis, and improves the inference accuracy by not having to infer unnecessary parts. In addition, the processing time required for the prediction process can be shortened.
[0105] In the ninth embodiment, by setting small-partition images whose average brightness value is less than a predetermined threshold value as excluded from the prediction process, areas outside the field of view and eyelashes are removed, and further, by training the trained model to predict that eyelashes, contours, etc. are excluded from extraction targets during the learning stage, it becomes possible to determine that small-partition images consisting mainly of eyelashes or contours that exist among small-partition images whose average brightness value is equal to or greater than a predetermined threshold value are excluded from the target (predict to classify as "excluded" or "other") during prediction processing in prediction unit 23. In this way, by removing and excluding unnecessary areas such as areas outside the field of view and eyelashes by two means, it is expected that the prediction speed will be further increased and the prediction accuracy will be further improved.
[0106] [Tenth embodiment] In the seventh to ninth embodiments, the prediction unit 23 has been described for predicting the state of dry eye of the subject's eye, but no reference has been made to the output contents of the prediction unit 23. The trained model used in the prediction process of the prediction unit 23 has three output nodes to output the likelihood of three states: P type (state of a healthy subject), C type (state of evaporative dry eye), and J type (state of aqueous deficiency dry eye). The likelihood of each type regarding the state of dry eye of the subject's eye is expressed by a numerical value ranging from 0 to 1.0 for each type, and is output so that the sum of the likelihoods of all types is 1.0. Therefore, the prediction result for each small section image is not output as a complete C type (output of C type is 1.0) or a complete J type (output of J type is 1.0), but is output as an output such as the most likely C type (output of C type is 0.8) or the most likely J type (output of J type is 0.7). Therefore, in some cases, it is possible to output an output that includes two elements, for example, an intermediate state between types P and C, or an intermediate state between types P and J. In this way, the numerical value of the likelihood of each type may be output between 0 and 1.0 for processing, or the type with the highest likelihood value may be set to 1.0 and the likelihood of other types may be set to 0, completely classifying the type into one of the types.
[0107] For the likelihood of each type obtained as a prediction result for each small segment image, the integrated value of the prediction results for all small segment images contained in one ophthalmic image is calculated, and the integrated value of each type is divided by the total number of small segment images, which can be used as an index for determining which type the image should be classified into when looking at the entire image. A large integrated value for type C indicates how water-deficient the ophthalmic image is (evaporative dry eye), and a large integrated value for type J indicates how oil-deficient the ophthalmic image is (aqueous-deficient dry eye), so they can be used as an index for dry eye treatment.
[0108] FIG. 27 is an explanatory diagram of a graph for plotting the likelihood of the dry eye state of the subject's eye. FIG. 27 shows the integrated values of the likelihood of each type of ophthalmic image calculated as described above, plotted on a graph. For example, by plotting the values before and after treatment on a graph, the effect of treatment can be visualized, which is very effective in making it easy to explain to patients. The axes of the two dry eye types other than the normal type may be perpendicular as shown in FIG. 27(a) or set at an appropriate angle. For example, the points of normal, aqueous-deficient dry eye, and evaporative dry eye may be arranged at the vertices of an equilateral triangle as shown in FIG. 27(b). The digitization of the plotted points using the integrated values may be performed from only one ophthalmic image, or may be performed using a series of multiple ophthalmic images or some of the discrete ophthalmic images.
[0109] [Eleventh embodiment] In the seventh to tenth embodiments, the case where the prediction process is performed on one ophthalmic image has been described, but the present invention is not limited thereto. That is, the image acquisition unit 21 may acquire a plurality of ophthalmic images arranged in time series, the prediction unit 23 may execute the prediction process on each of the plurality of ophthalmic images, and a change determination unit may be further provided that determines whether or not the temporal and / or spatial change in the state of the dry eye of the subject eye satisfies a predetermined condition for the prediction results obtained for each of the plurality of ophthalmic images.
[0110] When evaluating the dry eye condition of a subject's eye, some information can be obtained by evaluating one ophthalmic image, but some information can be obtained by considering the temporal and spatial changes of the prediction results of multiple ophthalmic images arranged in a time series. For example, as an example of temporal change, it is possible to evaluate how much time has passed since blinking that the dry eye condition of the subject's eye has been detected, or to calculate the number of small section images predicted as the dry eye condition of the subject's eye for each ophthalmic image, or the ratio of the dry eye condition of the subject's eye in the entire ophthalmic image, and to evaluate how much the number and ratio of the condition of the subject's eye have changed over time to exceed a predetermined threshold when tracking the temporal change. It is also possible to detect that the gradient of the temporal change in the ratio of the dry eye condition of the subject's eye is equal to or greater than a predetermined condition. As an example of spatial change, it is possible to evaluate how the location where the dry eye condition of the subject's eye is detected moves.
[0111] FIG. 28 is an explanatory diagram showing an example of dynamic analysis of the state of dry eye of a subject's eye. In FIG. 28, a plurality of ophthalmological images arranged in time series are acquired by a technique such as video recording, and the ratios (proportions) of three states, P type (state of a healthy subject), C type (state of evaporative dry eye), and J type (state of aqueous deficient dry eye), are calculated for each ophthalmological image, and the ratios of the three states are shown over time. In FIG. 28, the vertical axis shows the proportions of the healthy type (P), aqueous deficient type (J), and evaporative type (C), and the horizontal axis shows the time axis. By displaying the graph in this way, the dynamics of the tear state can be visualized. Of the four graphs shown in FIG. 28, the top two graphs show the dynamic analysis of a healthy subject (a person without dry eye), the third graph shows the dynamic analysis of a patient with evaporative dry eye (EDE), and the last graph shows the dynamic analysis of a patient with aqueous deficient dry eye (ADDE). As can be seen from the four graphs shown in FIG. 28, there are characteristics that can only be observed by dynamic analysis. For example, the increase in the proportion of P type after opening the eyelids indicates that the area in which the tear oil layer is in good condition spreads upward from the lower eyelid, and the slope A (dash-dotted line) on the graph is an index of the balance between the oil layer and the aqueous layer. In addition, the slope B (dashed line) indicating the process in which the proportion of P type reaches a plateau and decreases, and the slope B' (dashed line) indicating the subsequent steady state can be used as an index of tear stability. Similarly, in the diagnosis of aqueous-deficient dry eye, in which the water content is reduced but the oil secretion is high, the proportion of P type may be used, but the proportion of J type may also be focused on. By using these values, the balance between water and oil can be known, although it is not easy to distinguish them on the image, and the selection of a treatment method can be supported. When multiple ophthalmic images are obtained by video recording, analysis may be performed on all frames. However, since this takes a long time to perform inference, it is acceptable to thin out the frames appropriately within the range that does not interfere with the dynamic analysis, for example, to analyze every third frame.
[0112] In this way, by determining whether the temporal and / or spatial changes in the dry eye condition of the test eye satisfy specified conditions, it is possible to evaluate information that cannot be evaluated based on only the prediction results for a single ophthalmic image. [Explanation of symbols]
[0113] 100 Ophthalmic image processing system 10. Server equipment 20, 201 to 20n Terminal equipment 20, 20A-20C Ophthalmic image processing device 2001 CPU 2002 GPU 2003 Memory 2004 I / F 21 Image acquisition unit 22, 22B, 22C extraction section 23, 23B, 23C Prediction Section 24 Memory section 241 trained models 25 Result reflection area setting section 40 Imaging Equipment 41 Light source 42 Lens 43 Splitter 44 Objective Lens 45 Anterior segment 46 Imaging Lens 47 Image sensor 50 Display device
Claims
1. An ophthalmic image processing method for evaluating a condition of a subject's eye from an ophthalmic image obtained by photographing the subject's eye using machine learning, comprising: A learning step of extracting a plurality of small section images from the learning ophthalmic image, and learning in advance to predict the state of the subject's eye for each small section image by machine learning using correct answer data regarding the state of each small section image to obtain a trained model; an image acquisition step of acquiring a test ophthalmic image; Extracting a plurality of sub-region images from a test ophthalmic image; A prediction step of predicting a state of the subject's eye for each small segment image using the trained model, The small section image is extracted from the ophthalmologic image so as to have an image size according to the state of the subject's eye to be evaluated.
13. An ophthalmologic image processing method comprising:
2. The method further includes a result reflection area setting step of setting a result reflection area, which is an area having a predetermined size equal to or smaller than the size of the small partition image, for reflecting the prediction result of the small partition image, In the extraction step, an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area is extracted as the small partition image in correspondence with each of the result reflection areas; In the prediction step, the prediction result obtained for each small block image is reflected in the corresponding result reflection area.
2. An ophthalmologic image processing method according to claim 1.
3. The method further includes an edge image generating step of extracting edges from the ophthalmic image to generate an edge image, In the prediction step, when a luminance value in the edge image corresponding to the position of the result reflection area is equal to or greater than a predetermined threshold, the small partition image corresponding to the result reflection area is set as a target for prediction processing.
3. An ophthalmologic image processing method according to claim 2.
4. In the prediction step, an average brightness value is calculated for each of the small-partition images, and the small-partition images having an average brightness value equal to or greater than a predetermined threshold value are set as targets for prediction processing.
4. An ophthalmologic image processing method according to claim 1, wherein the first and second images are processed by the first and second imaging devices.
5. The method further includes a prediction result display step of displaying an image showing the prediction result superimposed on the ophthalmologic image by a display means.
5. An ophthalmologic image processing method according to claim 1, wherein the first and second images are processed by the first and second imaging devices.
6. In the image acquisition step, a plurality of ophthalmologic images arranged in time series are acquired, In the prediction step, a prediction process is performed for each of a plurality of ophthalmologic images; The method further includes a change determination step of determining whether or not a temporal change and / or a spatial change in the state of the subject's eye satisfies a predetermined condition for the prediction results obtained for each of the plurality of ophthalmologic images.
6. An ophthalmologic image processing method according to claim 1,
7. The condition of the subject's eye is the presence or absence of interference fringes in the tear film.
7. An ophthalmologic image processing method according to claim 1,
8. The condition of the subject's eye is whether or not there is a break 7. An ophthalmologic image processing method according to claim 1,
9. An ophthalmologic image processing device for evaluating a state of a subject's eye from an ophthalmologic image obtained by photographing the subject's eye, an image acquisition unit for acquiring the ophthalmologic image to be evaluated; an extracting unit for extracting a plurality of small section images from the ophthalmologic image; a prediction unit that extracts a plurality of small-area images from a learning ophthalmic image, and predicts the state of the subject's eye for each small-area image based on a trained model that has been trained in advance to predict the state of the subject's eye for each small-area image by machine learning using correct answer data relating to the state of each small-area image, The small section image is extracted from the ophthalmologic image so as to have an image size according to the state of the subject's eye to be evaluated.
1. An ophthalmologic image processing device comprising:
10. An ophthalmologic image processing program for causing a computer to realize a process for evaluating a state of a subject's eye from an ophthalmologic image obtained by photographing the subject's eye, The computer includes: an image acquisition function for acquiring the ophthalmic image to be evaluated; an extraction function for extracting a plurality of subdivision images from the ophthalmologic image; A prediction function is realized that predicts the state of the subject's eye for each small-area image based on a trained model that has been trained in advance to predict the state of the subject's eye for each small-area image by machine learning using correct answer data regarding the state of each small-area image, The small section image is extracted from the ophthalmologic image so as to have an image size according to the state of the subject's eye to be evaluated.
13. An ophthalmological image processing program comprising:
11. An ophthalmic image processing method for evaluating a dry eye condition of a test eye from an ophthalmic image obtained by photographing the test eye using machine learning, comprising: a learning step of extracting a plurality of small segment images from the ophthalmic image for training, and learning in advance to predict whether the dry eye state of the test eye for each small segment image corresponds to at least a healthy state, an aqueous-deficient dry eye state, or an evaporative dry eye state by machine learning using correct answer data regarding the state of each small segment image to obtain a trained model; an image acquisition step of acquiring a test ophthalmic image; Extracting a plurality of sub-region images from a test ophthalmic image; A prediction step of predicting a dry eye state of a subject's eye of each small segment image using the trained model, The small section image is extracted from the ophthalmologic image so as to have an image size that allows appropriate evaluation of the state of dry eye of the subject's eye to be evaluated.
13. An ophthalmologic image processing method comprising:
12. The method further includes a result reflection area setting step of setting a result reflection area, which is an area having a predetermined size equal to or smaller than the size of the small partition image, for reflecting the prediction result of the small partition image, In the extraction step, an area that includes the result reflection area and has a predetermined size equal to or larger than the result reflection area is extracted as the small partition image in correspondence with each of the result reflection areas; In the prediction step, the prediction result obtained for each small block image is reflected in the corresponding result reflection area. The ophthalmologic image processing method according to claim 11 .
13. In the prediction step, an average brightness value is calculated for each of the small-partition images, and the small-partition images having an average brightness value equal to or greater than a predetermined threshold value are set as targets for prediction processing.
13. An ophthalmologic image processing method according to claim 11 or 12.
14. The method further includes a prediction result display step of displaying an image showing the prediction result superimposed on the ophthalmologic image by a display means. The ophthalmologic image processing method according to any one of claims 11 to 13.
15. In the image acquisition step, a plurality of ophthalmologic images arranged in time series are acquired, In the prediction step, a prediction process is performed for each of a plurality of ophthalmologic images; The method further includes a change determination step of determining whether or not a temporal change and / or a spatial change in the dry eye state of the subject eye satisfies a predetermined condition for the prediction results obtained for each of the plurality of ophthalmologic images.
15. An ophthalmologic image processing method according to any one of claims 11 to 14.
16. An ophthalmologic image processing device for evaluating a state of dry eye of a subject's eye from an ophthalmologic image obtained by photographing the subject's eye, comprising: an image acquisition unit for acquiring the ophthalmologic image to be evaluated; an extracting unit for extracting a plurality of small section images from the ophthalmologic image; a prediction unit that extracts a plurality of small segment images from a training ophthalmic image, and predicts the dry eye state of the test eye for each small segment image based on a trained model that has been trained in advance to predict whether the dry eye state of the test eye for each small segment image is at least a healthy state, an aqueous-deficient dry eye state, or an evaporative dry eye state by machine learning using correct answer data relating to the state of each small segment image, The small section image is extracted from the ophthalmologic image so as to have an image size that allows appropriate evaluation of the state of dry eye of the subject's eye to be evaluated.
1. An ophthalmologic image processing device comprising:
17. An ophthalmologic image processing program for causing a computer to execute a process for evaluating a state of dry eye of a subject's eye from an ophthalmologic image obtained by photographing the subject's eye, comprising: The computer includes: an image acquisition function for acquiring the ophthalmic image to be evaluated; an extraction function for extracting a plurality of subdivision images from the ophthalmologic image; A prediction function is realized in which a plurality of small segment images are extracted from a training ophthalmic image, and correct answer data regarding the state of each small segment image is used to predict, based on a trained model that has been trained in advance to predict, by machine learning, whether the dry eye state of the test eye for each small segment image is at least a healthy state, an aqueous-deficient dry eye state, or an evaporative dry eye state, and the dry eye state of the test eye for each small segment image is predicted based on the trained model; The small section image is extracted from the ophthalmologic image so as to have an image size that allows appropriate evaluation of the state of dry eye of the subject's eye to be evaluated.
13. An ophthalmological image processing program comprising:
Citation Information
Patent Citations
Ophtalmic instrument
JP2005211173A
Organism image processing device, output image production method, learning result production method and program
JP2018089301A
Image processing device and image processing program
JP2018121886A
Artificial intelligence-based system and methods for corneal diagnosis
US10468142B1
Retinal Vessel Image Enhancement Method and System
US20190122350A1
