Information processing device, information processing method, information processing system, and program
The information processing device improves disease risk determination by analyzing fundus images and biological data with a trained model and correction coefficients, addressing the accuracy issues of existing methods.
Patent Information
- Application Number
- JP2020132816
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-08-30
- Filing Date
- 2020-08-05
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2040-08-05
AI Technical Summary
Existing methods for disease risk determination using biological images from ophthalmological examination devices suffer from insufficient accuracy.
An information processing device that estimates disease risk using a trained model to analyze fundus images and biological information, applying weighting coefficients for correction, thereby improving determination accuracy.
Enhances the accuracy of disease risk determination by integrating fundus image analysis with biological information, allowing for precise risk estimation and intuitive user understanding.
Smart Images

Figure 0007596092000001 
Figure 0007596092000002 
Figure 0007596092000003
Abstract
Description
[Technical field]
[0001] The disclosure of this specification relates to an information processing device, an information processing method, an information processing system, and a program. [Background technology]
[0002] A variety of imaging diagnostic devices, including conventional X-ray imaging, X-ray computed tomography, nuclear magnetic resonance imaging, ultrasound, positron emission tomography, and single-photon emission tomography, are used to image various parts of the body for disease screening and diagnosis.
[0003] In particular, the eye is the only part of the body where blood vessels can be directly observed from outside the body, and eye diseases such as diabetic retinopathy and age-related macular degeneration are diagnosed through eye examinations. In addition to eye diseases, applications are being considered for early diagnosis of lifestyle-related diseases such as arteriosclerosis and diabetes, as well as screening for various diseases, such as risk assessment for cerebral infarction and dementia. For example, a technology for assessing the risk of disease using an ophthalmic examination device is known. Patent Document 1 discloses generating risk information indicating the risk of a specific disease by analyzing examination data obtained from an ophthalmic examination device. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-386 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, it has been difficult to obtain sufficient risk assessment accuracy using only information obtained from biological images taken using diagnostic imaging devices or ophthalmic examination devices.
[0006] In view of the above problems, one of the objects of the disclosure of the present specification is to improve the accuracy of disease risk assessment.
[0007] In addition to the above-mentioned objective, the achievement of effects derived from each configuration shown in the description of the embodiment of the invention described below, which cannot be obtained by conventional techniques, can also be positioned as another objective of the disclosure of this specification. [Means for solving the problem]
[0008] The information processing device disclosed in this specification comprises: an estimation means for estimating a risk of a subject developing a disease by using a trained model that has learned a relationship between a feature amount acquired from a fundus image and a risk of developing a disease evaluated from the feature amount; The estimated risk of developing a disease is calculated by calculating the biological information of the subject. Using weighting coefficients determined for each and a correction means for correcting the difference. Effect of the Invention
[0009] According to the disclosure of the present specification, it is possible to improve the accuracy of disease risk assessment. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram showing an example of an overall configuration of an inspection system according to a first embodiment; [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Diagram 3] FIG. 1 is a flowchart showing an example of a processing procedure of an information processing device according to a first embodiment. [Figure 4] FIG. 13 is a diagram showing a display example of a disease risk estimation result according to the first embodiment; [Diagram 5] FIG. 1 is a diagram showing an example of the configuration of a medical appointment system according to a modified example of the first embodiment. [Figure 6] FIG. 13 is a diagram showing a flow of risk estimation according to the second embodiment. [Figure 7] FIG. 1 is a diagram showing an example of an overall configuration of an inspection system according to a first embodiment; [Figure 8]FIG. 1 is a diagram showing an example of an overall configuration of an inspection system according to a first embodiment; [Figure 9] FIG. 13 is a diagram showing an example of the configuration of a neural network used as a machine learning model according to Modification 4. [Figure 10] FIG. 23 is a diagram showing an example of the configuration of a neural network used as a machine learning model according to Modification 8. [Figure 11] FIG. 23 is a diagram showing an example of the configuration of a neural network used as a machine learning model according to Modification 8. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] The information processing device according to this embodiment is characterized by estimating the risk of disease based on biological images (medical images of a subject) taken by an image diagnostic device or an ophthalmic examination device, and biological information obtained from other tests, etc.
[0012] A preferred embodiment of the information processing device disclosed in this specification will be described in detail below with reference to the attached drawings. However, the components described in this embodiment are merely examples, and the technical scope of the information processing device disclosed in this specification is determined by the scope of the claims, and is not limited by the individual embodiments below. Furthermore, the disclosure of this specification is not limited to the following embodiments, and various modifications (including organic combinations of each embodiment) are possible based on the spirit of the disclosure of this specification, and are not excluded from the scope of the disclosure of this specification. In other words, configurations that combine each of the embodiments described below and their modifications are all included in the embodiments disclosed in this specification.
[0013] In the following embodiment, a case where a fundus camera is used as an imaging device for capturing a biological image used for estimating the risk of a disease will be described as a representative example, but the present invention is not limited to this, and other image diagnostic devices and ophthalmic examination devices can also be suitably applied. For example, other ophthalmic examination devices such as an optical coherence tomography (OCT) device capable of capturing images of the fundus or anterior part of the eye may be used depending on the disease for which risk estimation is performed. In addition, image diagnostic devices used for screening and diagnosing diseases, such as general X-ray photography, X-ray computed tomography, nuclear magnetic resonance imaging, ultrasound examination, positron emission tomography, and single photon emission computed tomography, may also be used.
[0014] [First embodiment] FIG. 1 is a diagram showing the overall configuration of an information processing system 100 including an information processing device according to this embodiment.
[0015] The information processing system 100 includes a fundus camera 101 , a biological information testing device 102 , a cloud server 103 , and an information processing device 104 .
[0016] The fundus camera 101 captures a fundus image, which is a biological image used for estimating the risk of disease.
[0017] For example, near-infrared light may be used for photographing with the fundus camera 101. In general, there are factors that degrade the quality of a still image during photographing, such as the position of the subject's eyes, body movement, blinking, and the inclusion of foreign matter in the image. Therefore, if the image captured at the time of photographing is not suitable for calculating image features, it is necessary to retake the image. However, flash photography using visible light used in conventional fundus cameras causes miosis, and a certain amount of time is required before re-photographing. Therefore, by using near-infrared light, miosis can be avoided, and continuous repeated photographing becomes possible. Furthermore, by using near-infrared light, it becomes easy to capture moving images without miosis.
[0018] Alternatively, weak visible light may be used for photographing with the fundus camera 101. As described above, generally, there are factors that degrade the quality of a still image during photographing, such as the subject's eye position, body movement, blinking, and contamination of the image with foreign matter, and if the image captured is not suitable for calculating image features, it is necessary to retake the image. However, in flash photography using visible light used in conventional fundus cameras, miosis occurs, and a certain amount of time is required before re-photographing. Therefore, by using weak visible light, miosis can be avoided, and continuous repeated photographing becomes possible. In addition, by using weak visible light, moving images can be captured without miosis.
[0019] Alternatively, the same light source or a light source with the same illuminance as the observation light used for locating the position of the anterior segment may be used for photographing with the fundus camera 101 .
[0020] It should be noted that the above is merely an example, and the photographing method is not limited to the above as long as a fundus image can be obtained.
[0021] In this embodiment, a fundus image captured by the fundus camera 101 is once transmitted to the biometric information testing device 102. The transmission method is a wired method such as USB or a wireless communication method such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark). The fundus image may be transmitted to the cloud server 103 without passing through the biometric information testing device 102, or may be transmitted to the information processing device 104.
[0022] The biological information testing device 102 acquires biological information used for estimating the risk of disease. The biological information testing device 102 can measure biological information such as the subject's height, weight, body fat percentage, systolic blood pressure, diastolic blood pressure, irregular pulse wave, heart rate, or body temperature. Note that biological information does not necessarily need to be acquired from an examination or measurement, and may be acquired by input from a user, such as the presence or absence of a smoking habit or a medical history. Furthermore, for example, the device may be provided with a blood test function for measuring blood glucose level, red blood cell count, hemoglobin, uric acid, etc. from the subject's blood, and a urine test function for testing the subject's urine. Note that in the case of a test using a specimen derived from a living body, such as a blood test or a urine test, a biological information testing kit 108 is provided to the subject by mail, as shown in FIG. 8. Then, the subject may store the results of the test in the cloud server 103 via a personal computer 107 or a mobile information terminal 106. In addition, for example, data such as blood pressure and weight measured by a blood pressure monitor 109 or a weight scale 110 that does not use a specimen may also be stored in the cloud server 103 by the subject in a similar manner. Note that the above types of biometric testing methods and flows of data transmission and reception are merely examples, and are not limiting, and it is sufficient if necessary biometric information can be transmitted to the information processing device 104.
[0023] The cloud server 103 stores and manages data such as images captured by the fundus camera 101 and biological information acquired by the biological information inspection device 102 .
[0024] As shown in FIG. 2, the information processing device 104 has, as its functional configuration, a communication IF (Interface) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a storage unit 114, an operation unit 115, a display unit 116, and a control unit 117.
[0025] The communication IF 111 is realized by a LAN card or the like, and controls communication between an external device (for example, the cloud server 103) and the information processing device 104. The ROM 112 is realized by a non-volatile memory or the like, and stores various programs and the like. The RAM 113 is realized by a volatile memory or the like, and temporarily stores various information. The storage unit 114 is an example of a computer-readable storage medium, and is realized by a large-capacity information storage device represented by a hard disk drive (HDD) or a solid-state drive (SSD), and stores various information. The operation unit 115 is realized by a keyboard, a mouse, or the like, and inputs instructions from a user to the device. The display unit 116 is realized by a display or the like, and displays various information to the user. The control unit 117 is realized by a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), or the like, and controls each process in the information processing device 104.
[0026] The control unit 117 includes, as its functional components, an acquisition unit 118, an estimation unit 119, a correction unit 120, and a display control unit 121.
[0027] The acquisition unit 118 reads and acquires data such as a fundus image of the subject captured by the fundus camera 101 and biological information of the subject acquired by the biological information testing device 102 from the cloud server 103. Note that the data does not necessarily have to be acquired from the cloud server 103, and for example, data transmitted directly from the fundus camera 101 or the biological information testing device 102 may be acquired.
[0028] The estimation unit 119 estimates the risk of a disease of the subject from the fundus image of the subject acquired by the acquisition unit 118. In this embodiment, the risk of a disease indicates the probability of developing a disease within a certain period of time. The probability of developing a disease may be expressed quantitatively, such as by a percentage, or may be expressed qualitatively, such as by indicating that the risk is high or low.
[0029] The correction unit 120 corrects the estimation result of the disease risk estimated by the estimation unit 119 and calculates the final disease risk. More specifically, the correction unit 120 corrects the estimation result of the disease risk estimated by the trained model using a weighting coefficient of the probability of developing each disease that is preset for each of the multiple pieces of biometric information obtained by the biometric information testing device 102. That is, the correction unit 120 corresponds to an example of a correction means that corrects the risk of developing a disease using a predetermined weighting coefficient determined for each piece of biometric information.
[0030] The display control unit 121 causes the display unit 116 to display the final disease risk estimation result corrected by the correction unit 120.
[0031] Next, a process procedure for estimating a risk of a disease performed by the information processing device 104 according to this embodiment will be described with reference to the flowchart of FIG.
[0032] (S3000) (Reading fundus image) In S3000, the acquisition unit 118 reads and acquires a fundus image captured by the fundus camera 101 and stored in the cloud server 103. Alternatively, the acquisition unit 118 acquires a fundus image directly transmitted from the fundus camera 101.
[0033] (S3010) (Calculation of image features) In S3010, the estimation unit 119 detects a feature amount for estimating a risk of a disease from the fundus image acquired from the cloud server 103.
[0034] Alternatively, a moving image captured by the fundus camera 101 may be obtained from the cloud server 103, and desired image features may be detected from the moving image. Generally, there are factors that degrade the quality of a still image during imaging, such as the subject's eye position, body movement, blinking, and the inclusion of foreign matter in the image. Therefore, by calculating image features from a portion of the moving image that has no image quality problems and is suitable for calculating image features, the risk of disease can be appropriately estimated. The optimum portion in the moving image can be selected using the brightness, contrast, and sharpness of the image, matching with a pre-registered pattern, and the like.
[0035] The detected feature amount may be, for example, the morphological features and color information of the defective portion, such as the diameter, curvature, angle, or branching of the blood vessel. Alternatively, it may be, for example, abnormal shape, image contrast, or color information of a portion other than the blood vessel, such as bleeding or white spots. Note that the feature amount used for risk estimation is not limited to these, and various feature amounts may be used.
[0036] (S3020) (Estimating the risk of disease) In S3020, the estimation unit 119 estimates the risk of a disease by inputting the feature amount detected in S3010 into the trained model. That is, the estimation unit 119 corresponds to an example of an estimation means that estimates the risk of a subject developing a disease by using a trained model that has learned the relationship between the feature amount acquired from a fundus image and the risk of developing a disease evaluated from the feature amount. More specifically, it corresponds to an example of an estimation means that estimates the probability that the subject will develop a disease by inputting the feature amount obtained from the fundus image of the subject into the trained model.
[0037] Here, the trained model refers to a machine learning model that follows a machine learning algorithm such as a support vector machine and that has been trained in advance using appropriate training data. Note that the trained model is not one that does not undergo further training, and can also undergo additional training.
[0038] The learning data is composed of one or more pairs of input data and output data (correct answer data). The trained model according to this embodiment trains output data (data on risk of disease) for input data (data on multiple feature amounts detected from a biological image such as a fundus image) as training data according to an arbitrary training algorithm. Specifically, for example, the model learns the correlation between an abnormal shape other than a blood vessel portion due to bleeding or white spot detected from a fundus image and the probability of developing diabetic retinopathy. Alternatively, for example, the model learns the correlation between a feature amount related to the vascular morphology indicating an arterial diameter, a venous diameter, a ratio between an arterial diameter and a venous diameter, a branching angle of a blood vessel, an asymmetry of the branching, an arterial venous stenosis or a twisting of a blood vessel, and a risk of developing a cardiovascular disease, a stroke, or the like evaluated from the feature amount. As described above, the risk of developing a disease may be output as a percentage, or may be classified into multiple qualitative classes such as high risk and low risk and output. Furthermore, the correlation between the input data and the output data to be trained is not limited to the above combinations, and various correlations related to disease risk estimation can be learned.
[0039] In addition, the trained model can be repeatedly trained based on a data set including input data and output data.
[0040] In this embodiment, the trained model for estimating disease risk may be generated by the information processing device 104, or may be a model generated by an information processing device different from the information processing device 104.
[0041] In the case where the information processing device 104 also generates a trained model for estimating a disease risk, the information processing device 104 further includes a generation unit (not shown).
[0042] As described above, the generating unit learns output data for input data as teacher data according to an arbitrary learning algorithm, and generates a trained model. Specific algorithms for machine learning include nearest neighbor, naive Bayes, decision tree, and support vector machine. In addition, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, is also included. When performing deep learning using a neural network, a trained model is obtained by learning a fundus image and a risk of developing a disease evaluated from the fundus image as a pair. For example, since people with thin blood vessels called retinal arterioles have a higher risk of developing high blood pressure than people with thick blood vessels, a high risk is paired as training data for a fundus image in which the retinal arterioles are evaluated as being thin as described above.
[0043] In other words, the estimation unit 119 corresponds to an example of an estimation means that estimates the risk of the subject developing a disease by inputting the fundus image of the subject acquired by the acquisition means into a learned model that has undergone deep learning of the relationship between the fundus image and the risk of developing the disease evaluated from the fundus image.
[0044] The relationship between the input data and the output data to be learned is not limited to the above combination, and various correlations related to disease risk estimation can be learned. Any of the above algorithms that can be used can be appropriately applied to this embodiment.
[0045] In addition, multiple trained models may be generated depending on the disease for which the risk is to be estimated, or one trained model may be generated so that it can estimate the risk of multiple diseases.
[0046] In other words, the estimation unit 119 corresponds to an example of an estimation means that estimates the risk of the subject developing the first disease and the second disease using a trained model that has learned the relationship between features obtained from a fundus image and the risk of developing a first disease evaluated from the features, and the relationship between features obtained from a fundus image and the risk of developing a second disease evaluated from the features.
[0047] Alternatively, the estimation unit 119 corresponds to an example of an estimation means that estimates the risk of a subject developing the first disease and the second disease by using a first trained model that has learned the relationship between features obtained from a fundus image and the risk of developing a first disease evaluated from the features, and a second trained model that has learned the relationship between features obtained from a fundus image and the risk of developing a second disease evaluated from the features.
[0048] (S3030) (Saving the estimation results) In S3030, the information processing device 104 stores the estimated disease risk in the storage unit 114. Alternatively, the information processing device 104 transfers the estimated disease risk to the cloud server 103 via the communication IF 111 and stores the risk in the cloud server 103. The information processing device 104 may store the estimated disease risk in both the storage unit 114 and the cloud server 104.
[0049] (S3040) (Reading biometric information) In S3040, the information processing device 104 reads the biological information obtained from the biological information testing device 102 and stored in the cloud server 103. Alternatively, the information processing device 104 acquires the biological information directly transmitted from the biological information testing device 102.
[0050] (S3050) (Reading the estimation results) In S3050, the acquisition unit 118 reads the estimated result of the disease risk stored in the storage unit 114 or the cloud server 103.
[0051] (S3060) (Correction of Estimated Results) In S3060, the correction unit 120 corrects the estimation result of the disease risk estimated by the estimation unit 119, and calculates the final disease risk. More specifically, the probability of developing each disease is corrected using a weighting coefficient of the probability of developing each disease that is preset for each of a plurality of pieces of biological information measured and examined using the biological information inspection device 102. For example, since the probability of developing a disease varies depending on the subject's blood pressure and body mass index (BMI) calculated from the subject's height and weight, the probability calculated based on the feature amount obtained from the fundus image is corrected based on the biological information. The correction amount uses a value calculated from the feature amount obtained from the fundus image of subjects suffering from a disease and healthy subjects, and the evaluation result of the biological information.
[0052] Thereafter, the corrected probability is calculated as the final risk estimation result. Note that multiple thresholds may be set for the probability of onset, and multiple risk stage categories may be classified using the thresholds, and the classified categories may be used as the final risk estimation result. For example, the probability of onset of a disease may be divided into three stages, namely, "0 to 33%, 34% to 66%, and 67% to 100%," and each of the stages may be classified into the categories of "low risk, medium risk, and high risk". Note that the classification method is not limited to the above, and may be divided into, for example, two stages or four or more stages. Also, the threshold value is an example, and is not limited thereto. That is, the output method of the output estimation result is not limited to the above, and it is sufficient that the output is made so that the subject can recognize the degree of risk of developing the disease. Furthermore, although the corrected estimation result is classified into categories in the above, when the trained model is trained, the output data may be trained by classifying it into multiple classes such as "low risk, medium risk, and high risk" in advance, and the estimation result may be output in a classified form at the stage when the disease risk is estimated in S3020.
[0053] (S3070) (Saving corrected estimation results) In S3070, the information processing device 104 stores the corrected disease risk in the storage unit 114. Alternatively, the information processing device 104 transfers the corrected disease risk to the cloud server 103 via the communication IF 111 and stores the corrected disease risk in the cloud server 103. Note that the information processing device 104 may store the corrected disease risk in both the storage unit 114 and the cloud server 104.
[0054] (S3080) (Display and print results) In S3080, the display control unit 121 causes the display unit 116 to display the estimated final disease risk, or transmits it to a separate printer for output.
[0055] FIG. 4 shows an example of display of the disease risk estimation result. FIG. 4(a) shows a state in which the disease risk estimation result calculated as the disease risk stage is classified into three stages, high risk, medium risk, and low risk, and displayed. That is, the display control means 121 classifies the corrected disease risk into multiple classes and displays them on the display unit. Furthermore, an explanation of what state each category indicates is displayed. For example, when the estimated category is "low risk", an explanation such as "The risk of developing is low. Please maintain a healthy lifestyle" is displayed. Note that the above explanation is an example and is not limited to the above. Furthermore, it is not necessary to include an explanation, and a tab for providing a supplementary explanation may be provided only when the subject wants to know more about the disease, such as when the subject is estimated to be at high risk. Alternatively, only a simple explanation may be included, and a tab for providing a supplementary explanation may be provided when the subject wants to know more about the disease.
[0056] The risk estimation results of a disease may be displayed in a graph, and the risk estimation results of a plurality of diseases may be displayed using a radar chart as in Fig. 4(b), or the risk estimation results of a plurality of diseases may be displayed using a bar graph as in Fig. 4(c). Note that the above is only an example, and the form is not limited as long as the risk estimation results of a disease are visually represented in two dimensions.
[0057] 4(d) to (f) show a state in which a fundus image photographed by the fundus camera 101 is displayed, and for example, the disease risk estimation result and the fundus image may be displayed in parallel with the results of FIG. 4(a) to (c) so that the disease risk estimation result and the fundus image can be viewed at the same time. That is, the display control unit 121 may display the corrected disease risk estimation result in parallel with the fundus image of the subject.
[0058] Specifically, Fig. 4(d) shows a state in which images of the left and right eyes are displayed side by side. Fig. 4(e) shows a state in which two images (a current image and a past image) captured at different times are displayed side by side. Fig. 4(f) shows an example of the main image parts related to risk estimation, with explanations added. That is, the parts of the fundus image that are highly correlated with diseases are highlighted and displayed.
[0059] The display method is not limited to these, and various display methods of risk estimation results, and display using not only captured images but also biometric information, past information thereof, and progress over time, etc. may be used. Furthermore, the estimation results may be displayed not only on the display unit 116 of the information processing device 104 but also on a display unit provided in the biometric information testing device 102. Alternatively, as shown in FIG. 7, application software dedicated to the testing system may be installed in the mobile information terminal 106. This may allow the received disease risk estimation results and information related to other testing systems to be displayed on the mobile information terminal 106 via the application software. In other words, the device that displays the estimation results does not necessarily have to be the information processing device 104.
[0060] Furthermore, rather than merely displaying the results of the disease risk estimation on the display unit, the results may be sent to a separate printer and output.
[0061] In this manner, the processing of the information processing device 104 is carried out.
[0062] According to the above, when estimating the disease risk of a subject, the estimation result estimated by the feature amount acquired from the fundus image is corrected by using a weighting coefficient of the probability of developing each disease preset for each of the plurality of biological information, thereby improving the accuracy of the estimation result. In addition, since the disease risk estimation does not require expertise, the disease risk can be easily estimated regardless of the user. Furthermore, by classifying the disease risk into a plurality of stages and displaying the estimation result in a display form that is easy for the subject to understand, the risk of the disease can be intuitively recognized. In addition, by displaying the biological image such as the fundus image that was the subject of the examination in parallel with the estimation result, the location of the disease can be easily recognized.
[0063] (Variation 1) In the first embodiment, the risk of disease is estimated and the results are displayed or output as a printout, etc. In this modification, the estimation results are sent to a medical institution or the like so that the subject can receive additional tests, visit an appropriate medical institution or department, or receive consultation on lifestyle habit guidance, etc.
[0064] FIG. 5 shows the configuration of a system that cooperates with a medical institution 130 and provides advice to subjects.
[0065] In the testing system of this embodiment, recommended medical institutions 130 and departments are output together with the risk estimation results according to the type of disease and the risk estimation results from a plurality of pre-registered medical institutions 130. The subject can check the above output results, for example, on the display unit or printout of the biological information testing device 102, or on the mobile information terminal 106 or personal computer 107.
[0066] Medical institutions 102 and relevant departments according to the type of disease and the degree of risk of the disease are stored in the cloud server 103. Then, by reading out this information from the cloud server 103 according to the risk estimation result, the medical institutions 130 and departments can be sent to the biometric information testing device 102, the mobile information terminal 106, or the personal computer 107 and presented to the subject.
[0067] Furthermore, a reservation at the medical institution 130 can be made on the biological information testing device 102, the mobile information terminal 106, or the personal computer 107. Application software for reservation use to make a reservation at the medical institution is installed in the biological information testing device 102.
[0068] When making a reservation using the portable information terminal 106 or personal computer 107, application software for reservation is downloaded and installed.
[0069] The reservation application software communicates with the medical appointment system 132 of the medical institution 130 to inquire about the subject's ID and the desired appointment date and time. If the reservation is possible based on the latest reservation status stored in the medical appointment system, the medical appointment system 132 registers the reservation using the ID. When the reservation is completed, the captured image, biometric information, and risk estimation results can be transferred to the medical institution 130 in conjunction with the examination. The transfer of the captured image, biometric information, and risk estimation results is selected on the reservation application software. With this operation, information such as the captured image, biometric information, and risk estimation results is read from the cloud server 103, securely transferred, and stored in the examinee information storage unit 133 in the medical appointment system 132 of the medical institution 130.
[0070] In addition, in the examination system of this embodiment, a doctor's diagnosis or consultation can be received on the system according to the disease risk obtained by the examination system. For example, if the risk of a fundus disease is high as a result of risk estimation using an image from the fundus camera 101, the fundus image is transferred to a doctor. Then, based on that, a face-to-face diagnosis or consultation can be received from a doctor via a video communication function (a system capable of video communication) built into the biometric information examination device 102, or a portable information terminal 106 or a personal computer 107 owned by the subject.
[0071] Similarly, if you are at high risk for other diseases, you may be able to receive additional testing, visits to appropriate medical institutions or departments, and consultations regarding lifestyle advice.
[0072] Furthermore, in the testing system of this embodiment, based on the risk estimation results and results obtained by the biological information system 102, the testing system can directly suggest improvements in lifestyle habits, intake of supplements and over-the-counter medicines, and the like.
[0073] At this time, the cloud server 103 stores the type of disease, lifestyle improvement instructions according to the degree of risk of the disease, recommended supplements, and general medicines.
[0074] Then, by reading out this information from the cloud server 103 in accordance with the risk estimation result, the above-mentioned proposal can be sent to the biological information testing device 102, the mobile information terminal 106 or the personal computer 107 and presented to the subject.
[0075] These services are not limited to those mentioned above, and various services can be provided depending on the results of the risk estimation.
[0076] [Second embodiment] In the first embodiment, the accuracy of disease risk estimation is improved by inputting multiple features obtained from a biometric image into a trained model and correcting the disease risk estimated using a weighting coefficient that is preset for each piece of biometric information obtained by a biometric test.
[0077] On the other hand, in this embodiment, the risk of disease is estimated using a trained model that has been trained on image features obtained from an image captured by a fundus camera 101 and the probability of developing a disease, and on biometric information obtained by a biometric information testing device 102 and the probability of developing a disease.
[0078] The overall configuration of the information processing system in this embodiment is the same as that in the first embodiment. The fundus camera 101, the biological information testing device 102, and the cloud server 103 used in this embodiment are the same as those in the first embodiment.
[0079] The process steps of this embodiment will be described below with reference to Fig. 6. Note that steps S6000 to S6040 and S6070 are the same as those in the first embodiment, and therefore the description thereof will be omitted.
[0080] (S6050: Estimating (determining) risk of disease) In S6050, first, the acquisition unit 118 acquires feature amounts obtained from a fundus image of the subject stored in the cloud server 103 and biometric information measured and examined using the biometric information examination device 102. Then, the estimation unit 119 inputs the feature amounts and biometric information acquired by the acquisition unit 118 to a trained model generated in advance, thereby estimating the risk of the subject developing a disease.
[0081] Specifically, the input data are features obtained from fundus images, such as arterial diameter, venous diameter, ratio of arterial diameter to venous diameter, branching angle of blood vessels, asymmetry of the branching, arterial venous stenosis or vascular tortuosity, and biometric information, such as blood pressure, BMI index, age, sex, medical history, or smoking habit, and the correlation is learned between the features and the probability of developing diseases such as cardiovascular disease and cerebrovascular disease evaluated from the biometric information.
[0082] The correlations between the input data and the output data to be learned are not limited to the above combinations, and various correlations related to disease risk estimation can be learned.
[0083] In addition, in this embodiment, the trained model for estimating disease risk may be generated by the information processing device 104, or may be a model generated by an information processing device different from the information processing device 104.
[0084] (S6060: Saving estimation results) In S6060, the information processing device 104 stores the estimated result of the disease risk in the storage unit 114. Alternatively, the information processing device 104 transfers the estimated result of the disease risk to the cloud server 103 via the communication IF 111 and stores the estimated result in the cloud server 103. The estimated result may be stored in both the storage unit 114 and the cloud server 104.
[0085] In this manner, the processing of the information processing device 104 is carried out.
[0086] Based on the above, when estimating a subject's risk of disease, the accuracy of estimating the risk of disease can be improved by using a trained model that has learned the correlation between features obtained from a fundus image and the probability of developing the disease evaluated from biometric information obtained by a biopsy.
[0087] (Variation 2) In the various embodiments and modifications described above, a trained model (trained model for estimation) used in the estimation process regarding the subject's disease may be adjusted (tuned) for each subject, and a trained model dedicated to the subject may be generated. For example, a transfer learning of a general-purpose trained model for estimating the subject's disease may be performed using a tomographic image acquired in a past examination of the subject, and a trained model dedicated to the subject may be generated. By linking the trained model dedicated to the subject with the ID of the subject and storing it in an external device such as the storage unit 114 or a server, the control unit 117 can identify and use the trained model dedicated to the subject based on the ID of the subject when performing a current examination of the subject. By using the trained model dedicated to the subject, the estimation accuracy regarding the disease of each subject can be improved.
[0088] (Variation 3) In the various embodiments and modifications described above, the control unit 117 may perform various image processing using an image acquired by shooting, etc. For example, the control unit 117 may generate a high-quality image by improving the image quality of the image acquired by shooting using a trained model for image quality improvement (image quality improvement model). Here, improving the image quality includes reducing noise, converting the color or gradation to make it easier to observe the subject, improving the resolution or spatial resolution, and enlarging the image size while suppressing a decrease in resolution.
[0089] As the machine learning model for improving image quality, for example, CNN or the like can be used. As the learning data of the high image quality model, various images such as anterior eye images and SLO images are used as input data, and high image quality images corresponding to the input images, for example, images that have been subjected to high image quality processing, are used as output data. Here, the high image quality processing includes aligning images captured multiple times at the same spatial position, and averaging the aligned images. Note that the high image quality processing is not limited to averaging, and may be, for example, processing using a smoothing filter, maximum a posteriori probability estimation processing (MAP estimation processing), tone conversion processing, etc. Furthermore, the high image quality processed image may be, for example, an image that has been subjected to filter processing such as noise removal and edge emphasis, or an image whose contrast has been adjusted to change a low brightness image to a high brightness image. Furthermore, since the output data of the learning data related to the high image quality model only needs to be a high image quality image, it may be an image captured using an OCT device with higher performance than the OCT device used to capture the tomographic image that is the input data, or an image captured under high load settings.
[0090] However, when machine learning is performed using images that have not been appropriately processed for image quality improvement as output data of training data, images obtained using a trained model trained using the training data may also be images that have not been appropriately processed for image quality improvement. Therefore, by removing pairs that include such images from the training data, it is possible to reduce the possibility that an inappropriate image will be generated using the trained model.
[0091] The control unit 117 performs image quality improvement processing using such an image quality improvement model, thereby making it possible to obtain an image with high image quality and high precision at a higher speed.
[0092] The high image quality model may be prepared for each type of image, which is input data. For example, a high image quality model for anterior eye images, a high image quality model for SLO images, a high image quality model for tomographic images, a high image quality model for OCTA front images, etc. may be prepared. For OCTA front images and En-Face images, a high image quality model may be prepared for each depth range for generating images. For example, a high image quality model for superficial layers, a high image quality model for deep layers, etc. may be prepared. Furthermore, the high image quality model may be learned for images for each imaging site (for example, the center of the macula, the center of the optic disc), or may be learned regardless of the imaging site.
[0093] At this time, for example, the fundus OCTA front image may be improved in quality using a high-quality model obtained by learning the fundus OCTA front image as learning data, and the anterior OCTA front image may be improved in quality using a high-quality model obtained by learning the anterior OCTA front image as learning data. The high-quality model may be learned regardless of the imaging site. Here, for example, the fundus OCTA front image and the anterior OCTA front image may have relatively similar distributions of blood vessels as imaging targets. In this way, in a plurality of types of medical images in which the imaging targets are relatively similar to each other, the feature amounts of each image may be relatively similar. Therefore, for example, the fundus OCTA front image may be configured to be able to improve not only the quality of the fundus OCTA front image but also the quality of the anterior OCTA front image using a high-quality model obtained by learning the fundus OCTA front image as learning data. For example, the anterior OCTA front image may be configured to be able to improve not only the quality of the anterior OCTA front image but also the quality of the fundus OCTA front image using a high-quality model obtained by learning the anterior OCTA front image as learning data. In other words, the image quality may be improved by using a high image quality model obtained by learning at least one type of front image, a fundus OCTA front image and an anterior eye OCTA front image, as learning data.
[0094] Here, a case is considered in which an anterior eye can also be photographed in an OCT device capable of photographing a fundus. In this case, for example, a fundus OCTA front image may be applied to the En-Face image of OCTA in the fundus photographing mode, and an anterior OCTA front image may be applied to the En-Face image of OCTA in the anterior segment photographing mode. In this case, when the high image quality button is pressed, for example, in the fundus photographing mode, in the display area of the En-Face image of OCTA, the display of one of a low-image quality fundus OCTA front image and a high-image quality fundus OCTA front image may be changed to the display of the other. In addition, when the high image quality button is pressed, for example, in the anterior segment photographing mode, in the display area of the En-Face image of OCTA, the display of one of a low-image quality anterior OCTA front image and a high-image quality anterior OCTA front image may be changed to the display of the other.
[0095] In addition, in an OCT device capable of photographing the fundus, when the anterior eye can also be photographed, an anterior eye adapter may be attached. Also, the optical system of the OCT device may be configured to be movable a distance approximately equal to the axial length of the subject's eye without using an anterior eye adapter. In this case, the focus position of the OCT device may be configured to be largely changeable toward the normal vision side to the extent that an image is formed on the anterior eye.
[0096] In addition, for example, in the fundus photography mode, a fundus OCT tomographic image may be applied to the tomographic image, and in the anterior segment photography mode, an anterior OCT tomographic image may be applied. In addition, the image quality improvement process of the fundus OCTA front image and the anterior segment OCTA front image described above can also be applied as image quality improvement process of the fundus OCT tomographic image and the anterior segment OCT tomographic image. At this time, when the image quality improvement button is pressed, for example, in the fundus photography mode, the display of one of the low-quality fundus OCT tomographic image and the high-quality fundus OCT tomographic image may be changed to the other in the display area of the tomographic image. In addition, when the image quality improvement button is pressed, for example, in the anterior segment photography mode, the display of one of the low-quality anterior OCT tomographic image and the high-quality anterior OCT tomographic image may be changed to the other in the display area of the tomographic image.
[0097] In addition, for example, in the fundus imaging mode, a fundus OCTA tomographic image may be applied to the tomographic image, and in the anterior segment imaging mode, an anterior OCTA tomographic image may be applied. In addition, the image quality improvement processing of the fundus OCTA front image and the anterior OCTA front image described above can also be applied as image quality improvement processing of the fundus OCTA tomographic image and the anterior OCTA tomographic image. At this time, for example, in the fundus imaging mode, information indicating a blood vessel region (for example, motion contrast data equal to or greater than a threshold) in the fundus OCTA tomographic image may be configured to be displayed superimposed on the fundus OCT tomographic image at a corresponding position in the display area of the tomographic image. In addition, for example, in the anterior segment imaging mode, information indicating a blood vessel region in the anterior OCTA tomographic image may be displayed superimposed on the anterior OCT tomographic image at a corresponding position in the display area of the tomographic image.
[0098] In this way, for example, when the feature quantities (appearance of the photographed subject) of a plurality of types of medical images are considered to be relatively similar to each other, at least one type of medical image of the plurality of types of medical images may be configured to be able to be improved in image quality using an image quality improvement model obtained by learning at least one type of medical image of the plurality of types of medical images as learning data. This makes it possible to configure the plurality of types of medical images to be able to be improved in image quality using, for example, a common learned model (common image quality improvement model).
[0099] The display screen of the fundus photography mode and the display screen of the anterior eye photography mode may have the same display layout or may have a display layout corresponding to each photography mode. Various conditions such as photography conditions and analysis conditions may be the same or different between the fundus photography mode and the anterior eye photography mode.
[0100] Here, the target image of the high image quality processing may be, for example, a plurality of OCTA front images (OCTA En-Face image, motion contrast En-Face image) (corresponding to a plurality of depth ranges). Also, the target image of the high image quality processing may be, for example, one OCTA front image corresponding to one depth range. Also, instead of the OCTA front image, the target image of the high image quality processing may be, for example, a luminance front image (luminance En-Face image), or an OCT tomographic image or a motion contrast data tomographic image (OCTA tomographic image) which is a B-scan image. Also, the target image of the high image quality processing may be not only an OCTA front image, but also various medical images such as a luminance front image and an OCT tomographic image or a motion contrast data tomographic image (OCTA tomographic image) which is a B-scan image. That is, the target image of the high image quality processing may be, for example, at least one of various medical images displayed on the display screen of the display unit 116. At this time, for example, since the image feature amount may differ for each type of image, a learned model for image quality improvement corresponding to each type of the target image of the high image quality processing may be used. For example, when the image quality improvement button is pressed in response to an instruction from the examiner, not only the image quality of the OCTA front image is improved using a trained model for image quality improvement corresponding to the OCTA front image, but also the image quality of the OCT tomographic image is improved using a trained model for image quality improvement corresponding to the OCT tomographic image. Also, for example, when the image quality improvement button is pressed in response to an instruction from the examiner, not only the display is changed to a high-quality OCTA front image generated using a trained model for image quality improvement corresponding to the OCTA front image, but also the display is changed to a high-quality OCT tomographic image generated using a trained model for image quality improvement corresponding to the OCT tomographic image. At this time, a line indicating the position of the OCT tomographic image may be superimposed on the OCTA front image. Also, the above line may be configured to be movable on the OCTA front image in response to an instruction from the examiner.Also, when the display of the high image quality button is in an active state, after the line is moved, the display may be changed to a high image quality OCT tomographic image obtained by performing image quality improvement processing on the OCT tomographic image corresponding to the current position of the line. Also, the high image quality button may be displayed for each target image for image quality improvement processing, so that image quality improvement processing can be performed independently for each image.
[0101] Also, information indicating a blood vessel region in an OCTA tomographic image (for example, motion contrast data equal to or greater than a threshold) may be displayed superimposed on an OCT tomographic image, which is a B-scan image at a corresponding position. At this time, for example, when the OCT tomographic image is improved in image quality, the OCTA tomographic image at the corresponding position may be improved in image quality. Then, information indicating a blood vessel region in an OCTA tomographic image obtained by improving image quality may be displayed superimposed on the OCT tomographic image obtained by improving image quality. Note that the information indicating a blood vessel region may be any identifiable information such as a color. Also, the information indicating a blood vessel region may be configured to be changeable between superimposed display and non-display according to an instruction from an examiner. Also, when a line indicating the position of an OCT tomographic image is moved on an OCTA front image, the display of the OCT tomographic image may be updated according to the position of the line. At this time, the OCTA tomographic image at the corresponding position is also updated, so that the superimposed display of information indicating a blood vessel region obtained from the OCTA tomographic image may be updated. Thereby, for example, at an arbitrary position, it is possible to effectively check the three-dimensional distribution and state of a blood vessel region while easily checking the positional relationship between the blood vessel region and the region of interest. Moreover, the image quality of the OCTA tomographic image may be improved by, instead of using a trained model for improving image quality, an image quality improvement process such as an averaging process of a plurality of OCTA tomographic images acquired at corresponding positions. The OCT tomographic image may be a pseudo OCT tomographic image reconstructed as a cross section at an arbitrary position in the OCT volume data. The OCTA tomographic image may be a pseudo OCTA tomographic image reconstructed as a cross section at an arbitrary position in the OCTA volume data. The arbitrary position may be at least one arbitrary position, and may be configured to be changeable according to an instruction from the examiner. In this case, a plurality of pseudo tomographic images corresponding to a plurality of positions may be configured to be reconstructed.
[0102] In addition, only one tomographic image (for example, an OCT tomographic image or an OCTA tomographic image) may be displayed, or multiple tomographic images may be displayed. When multiple tomographic images are displayed, tomographic images acquired at different positions in the sub-scanning direction may be displayed, or when multiple tomographic images acquired by, for example, a cross scan or the like are enhanced in image quality and displayed, images in different scanning directions may be displayed. When multiple tomographic images acquired by, for example, a radial scan or the like are enhanced in image quality and displayed, a partially selected multiple tomographic images (for example, two tomographic images at positions symmetrical to each other with respect to a reference line) may be displayed. Furthermore, multiple tomographic images may be displayed on a display screen for follow-up observation (a display screen for follow-up), and instructions for enhancing image quality and analysis results (for example, the thickness of a specific layer, etc.) may be displayed by a method similar to the above-mentioned method. At this time, the multiple tomographic images displayed may be multiple tomographic images acquired at different dates and times of a specific part of the eye to be examined, or multiple tomographic images acquired at different times on the same examination day. Also, image quality enhancement processing may be performed on the tomographic images based on information stored in a database by a method similar to the above-mentioned method.
[0103] Similarly, when the SLO image is displayed with high image quality, for example, the SLO image displayed on the same display screen may be displayed with high image quality. Furthermore, when the luminance front image is displayed with high image quality, for example, the luminance front image displayed on the same display screen may be displayed with high image quality. Furthermore, a plurality of SLO images and luminance front images may be displayed on a display screen for follow-up observation, and an instruction for image quality improvement and an analysis result (for example, the thickness of a specific layer, etc.) may be displayed by a method similar to the above-mentioned method. Furthermore, image quality improvement processing may be performed on the SLO image and the luminance front image based on information stored in a database by a method similar to the above-mentioned method. Note that the display of the tomographic image, the SLO image, and the luminance front image is an example, and these images may be displayed in any manner according to a desired configuration. Furthermore, at least two or more of the OCTA front image, the tomographic image, the SLO image, and the luminance front image may be displayed with high image quality by a single instruction.
[0104] With this configuration, the display control unit 121 can display the high-quality image obtained by the image quality improvement process on the display unit 116. When at least one of a plurality of conditions related to the display of the high-quality image, the display of the analysis result, the depth range of the displayed front image, etc. is selected, the selected condition may be maintained even if the display screen is changed. The display control unit 121 may control the display of various high-quality images, the above-mentioned lines, information indicating the blood vessel region, etc.
[0105] The high image quality model may be used for at least one frame of the live video on the preview screen displayed on the display unit 116 by the display control unit 121. In this case, when a plurality of live videos of different parts or different types are displayed on the preview screen, a trained model corresponding to each live video may be used. For example, an image with high image quality using a high image quality model for an anterior eye image may be used for an anterior eye image used in the alignment process. Similarly, an image with high image quality using a high image quality model for each image may be used for various images used in the detection process of a predetermined area in various images.
[0106] At this time, for example, when the high image quality button is pressed in response to an instruction from the examiner, the display of a plurality of different types of live moving images (for example, an anterior eye image, an SLO image, a tomographic image) may be changed (simultaneously) to a display of a high image quality moving image obtained by performing high image quality processing on each of them. At this time, the display of the high image quality moving image may be a continuous display of high image quality images obtained by performing high image quality processing on each frame. Also, for example, since the image feature amount may differ for each type of image, a learned model for high image quality corresponding to each type of target image for high image quality processing may be used. For example, when the high image quality button is pressed in response to an instruction from the examiner, not only the anterior eye image is subjected to high image quality processing using a high image quality model corresponding to the anterior eye image, but also the SLO image is subjected to high image quality processing using a high image quality model corresponding to the SLO image. Also, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the display of a high image quality anterior eye image generated using a high image quality model corresponding to the anterior eye image is changed to the display of a high image quality SLO image generated using a high image quality model corresponding to the SLO image. Also, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the SLO image is processed to have high image quality using a high image quality model corresponding to the SLO image, but also the tomographic image is processed to have high image quality using a high image quality model corresponding to the tomographic image. Also, for example, when the high image quality button is pressed in response to an instruction from the examiner, not only the display is changed to a high-image quality SLO image generated using a high image quality model corresponding to the SLO image, but also the display is changed to a high-image quality tomographic image generated using a high image quality model corresponding to the tomographic image. At this time, a line indicating the position of the tomographic image may be superimposed on the SLO image. Also, the above line may be configured to be movable on the SLO image in response to an instruction from the examiner. Also, when the display of the high image quality button is in an active state, after the above line is moved, the display may be changed to a high-image quality tomographic image obtained by processing the tomographic image corresponding to the current position of the line.Alternatively, a high image quality button may be displayed for each image that is to be subjected to high image quality processing, thereby enabling high image quality processing to be performed independently for each image.
[0107] As a result, even for live moving images, the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. Therefore, for example, when the operator corrects the alignment position while checking the preview screen, the failure of re-imaging can be reduced, so that the accuracy and efficiency of diagnosis can be improved. In addition, the control unit 117 may drive and control the above-mentioned scanning means in response to an instruction regarding the start of imaging so that a partial area such as an artifact area obtained by segmentation processing or the like is re-imaging (rescanned) during or at the end of imaging. Note that, depending on the state of the movement of the subject's eye, it may not be possible to capture a good image with one rescan, so the control unit 117 may drive and control so that rescanning is repeated a predetermined number of times. At this time, even during the predetermined number of rescans, the rescanning may be terminated in response to an instruction from the operator (for example, after the cancel imaging button is pressed). At this time, the imaging data until the rescanning is terminated in response to an instruction from the operator may be saved. Note that, for example, a confirmation dialog may be displayed after the cancel imaging button is pressed, and the imaging data may be selected to be saved or discarded in response to an instruction from the operator. Also, for example, after pressing the cancel shooting button, the next rescan is not executed (although the current rescan is executed until completion), and the system may be configured to wait until an instruction (input) from the operator is received in the confirmation dialog. Also, for example, when information indicating the reliability of the object recognition result for the target site (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, start of shooting, etc. may be automatically performed. Also, for example, when information indicating the reliability of the object recognition result for the target site (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, start of shooting, etc. may be changed to an executable state (cancellation of the execution prohibition state) in response to an instruction from the examiner.
[0108] Here, during auto-alignment, there is a possibility that the image of the subject, such as the retina of the subject's eye E, has not yet been captured well. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to a large difference between the medical image input to the trained model and the medical image used as training data. Therefore, when an evaluation value such as an image quality evaluation of a tomographic image (B-scan image) exceeds a threshold, the display of a high-quality moving image (continuous display of high-quality frames) may be automatically started. In addition, when an evaluation value such as an image quality evaluation of a tomographic image exceeds a threshold, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner. The image quality improvement button is a button for specifying the execution of image quality improvement processing. Of course, the image quality improvement button may be a button for instructing the display of a high-quality image.
[0109] Also, a different image quality improvement model may be prepared for each imaging mode having a different scan pattern, etc., and a trained model for image quality improvement corresponding to the selected imaging mode may be selected. Also, one image quality improvement model obtained by training learning data including various medical images obtained in different imaging modes may be used.
[0110] Here, in an ophthalmic device, for example, an OCT device, the scan pattern of the light beam used for measurement and the imaging site are different for each imaging mode. Therefore, for a trained model using a tomographic image as input data, a trained model may be prepared for each imaging mode, and a trained model corresponding to the imaging mode selected according to an instruction from an operator may be selected. In this case, the imaging modes may include, for example, a retina imaging mode, an anterior eye imaging mode, a vitreous body imaging mode, a macular imaging mode, an optic disc imaging mode, an OCTA imaging mode, etc. In addition, the scan patterns may include 3D scan, radial scan, cross scan, circle scan, raster scan, and Lissajous scan (scanning along a Lissajous curve), etc. In addition, in the OCTA imaging mode, a drive control unit (not shown) controls the above-mentioned scanning means so that the same region (same position) of the subject's eye is scanned multiple times with the measurement light. In the OCTA imaging mode, for example, a raster scan, a radial scan, a cross scan, a circle scan, a Lissajous scan, etc. can be set as the scan pattern. In addition, for a trained model that uses a tomographic image as input data, training can be performed using tomographic images corresponding to cross sections in different directions as training data. For example, training may be performed using tomographic images of cross sections in the xz direction and tomographic images of cross sections in the yz direction as training data.
[0111] The determination of whether or not to execute the image quality improvement process using the image quality improvement model (or display the high image quality image obtained by the image quality improvement process) may be performed according to an instruction from an operator using an image quality improvement button provided on the display screen, or may be performed according to settings previously stored in the storage unit 114. The fact that the image quality improvement process is performed using a trained model (image quality improvement model) may be displayed by displaying an active state of the image quality improvement button, or a message to that effect may be displayed on the display screen. The image quality improvement process may be performed in the execution state at the time of the previous startup of the ophthalmic apparatus, or may be performed in the execution state at the time of the previous examination for each subject.
[0112] In addition, the moving image to which various learned models such as the high image quality model can be applied is not limited to a live moving image, and may be, for example, a moving image stored (saved) in the storage unit 114. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit 114 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, a reference frame may be selected based on a condition that the vitreous body is present on the frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high image quality image (high image quality frame) generated sequentially by the high image quality engine for at least one frame of the moving image may be continuously displayed.
[0113] As for the above-mentioned method of inter-frame alignment, the method of alignment in the X direction and the method of alignment in the Z direction (depth direction) may be the same or different methods may be applied. The alignment in the same direction may be performed multiple times using different methods. For example, after performing a rough alignment, a precise alignment may be performed. As the alignment method, for example, there are (rough Z direction) alignment using a retinal layer boundary obtained by segmenting a tomographic image (B-scan image), (precise X-direction and Z-direction) alignment using correlation information (similarity) between a reference image and a plurality of regions obtained by dividing a tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. Also, it may be configured such that rough alignment is performed in pixel units and then precise alignment is performed in subpixel units.
[0114] The high image quality model may be updated by additional learning using the ratio value set (changed) in response to an instruction from the examiner as learning data. For example, if the examiner tends to set the ratio of the input image to the high image quality image high when the input image is relatively dark, the learned model will be additionally learned to have such a tendency. This allows, for example, customization as a learned model that can obtain a synthesis ratio that suits the examiner's preference. At this time, a button for determining whether or not to use the set (changed) ratio value as learning data for additional learning in response to an instruction from the examiner may be displayed on the display screen. Also, the ratio determined using the learned model may be set as a default value, and the ratio value may be changed from the default value in response to an instruction from the examiner. Also, the high image quality model may be a learned model obtained by additional learning of learning data including at least one high image quality image generated using the high image quality model. At this time, it may be configured to be selectable whether or not to use the high image quality image as learning data for additional learning in response to an instruction from the examiner.
[0115] (Variation 4) In the various embodiments and modifications described above, the control unit 117 may generate a labeled image for an image acquired by shooting using a trained model for image segmentation, and perform image segmentation processing. Here, the labeled image refers to a labeled image in which a region label is attached to each pixel of the tomographic image. Specifically, it is an image in which an arbitrary region is divided by a group of pixel values (hereinafter, label values) that can identify an arbitrary region among a group of regions depicted in the acquired image. Here, the arbitrary region that is identified includes a region of interest, a volume of interest (VOI), etc.
[0116] By identifying the coordinate group of pixels having an arbitrary label value from an image, it is possible to identify the coordinate group of pixels depicting a corresponding area such as a retinal layer in the image. Specifically, for example, if the label value indicating the ganglion cell layer constituting the retina is 1, a coordinate group having a pixel value of 1 among the pixels of the image is identified, and a pixel group corresponding to the coordinate group is extracted from the image. This makes it possible to identify the area of the ganglion cell layer in the image.
[0117] The image segmentation process may include a process of performing a reduction or enlargement process on the label image. In this case, the image completion process used for reducing or enlarging the label image uses a nearest neighbor method or the like so as not to erroneously generate undefined label values or label values that should not exist at the corresponding coordinates.
[0118] Image segmentation processing is processing for identifying regions called ROI (Region Of Interest) or VOI, such as organs or lesions depicted in an image, for use in image diagnosis or image analysis. For example, image segmentation processing can identify a group of regions of layers constituting the retina from an image acquired by OCT imaging of the posterior segment. If no region to be identified is depicted in the image, the number of identified regions is 0. If multiple region groups to be identified are depicted in the image, the number of identified regions may be multiple, or it may be one region surrounding the group of regions.
[0119] The identified region group is output as information that can be used in other processes. Specifically, for example, a coordinate group of a pixel group constituting each of the identified region groups can be output as a numerical data group. Also, for example, a coordinate group indicating a rectangular region, an elliptical region, a rectangular parallelepiped region, an ellipsoid region, or the like including each of the identified region groups can be output as a numerical data group. Furthermore, for example, a coordinate group indicating a straight line, a curve, a plane, or a curved surface that is on the boundary of the identified region group can be output as a numerical data group. Also, for example, a label image indicating the identified region group can be output.
[0120] Here, as the machine learning model for image segmentation, for example, a convolutional neural network (CNN) can be used. Here, with reference to FIG. 9, an example of configuring the machine learning model according to this modification using CNN will be described. FIG. 9 shows an example of the configuration of a trained model for image segmentation. In this example of the trained model, for example, when a tomographic image 1301 is input, a label image 1302 indicating a specified region group can be output.
[0121] The machine learning model shown in Fig. 9 is composed of a plurality of layers that process and output a group of input values. The types of layers included in the configuration of the machine learning model include a convolution layer, a downsampling layer, an upsampling layer, and a merger layer.
[0122] The convolution layer is a layer that performs convolution processing on a group of input values according to parameters such as the set filter kernel size, the number of filters, the stride value, the dilation value, etc. The number of dimensions of the filter kernel size may also be changed according to the number of dimensions of the input image.
[0123] The downsampling layer is a layer that performs processing to reduce the number of output value groups to be less than the number of input value groups by thinning out or combining input value groups. Specifically, such processing includes, for example, Max Pooling processing.
[0124] The upsampling layer is a layer that performs a process to make the number of output values larger than the number of input values by duplicating the input values or adding values interpolated from the input values, for example, a linear interpolation process.
[0125] A synthesis layer is a layer that inputs a group of values, such as a group of output values of a layer or a group of pixel values that make up an image, from multiple sources and performs processing to synthesize them by concatenating or adding them.
[0126] Note that different parameter settings for the layers and nodes that make up the neural network may affect the degree to which the trends trained from the teacher data can be reproduced in the output data. In other words, in many cases, appropriate parameters differ depending on the implementation form, so they can be changed to preferred values as necessary.
[0127] In addition to changing the parameters as described above, CNN may be able to obtain better characteristics by changing the configuration of the CNN, such as outputting more accurate alignment position information, shortening processing time, or shortening the time required to train a machine learning model.
[0128] The CNN used in this modification is a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers. The U-net type machine learning model is configured (for example, by using skip connections) so that the position information (spatial information) obscured in the multiple layers configured as the encoder can be used in the same dimensional layers (layers corresponding to each other) in the multiple layers configured as the decoder.
[0129] Although not shown, as an example of a modified CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer. Through these steps of the CNN, the features of the captured image can be extracted.
[0130] As the machine learning model according to this modification, for example, CNN (U-net type machine learning model) as shown in FIG. 9, a model combining CNN and LSTM, FCN (Fully Convolutional Network), SegNet, etc. can be used. Also, a machine learning model that performs object recognition, etc. can be used according to a desired configuration. As the machine learning model that performs object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN can be used. Furthermore, a machine learning model that performs object recognition in regions can be used. As the machine learning model that performs object recognition in regions, YOLO (You Only Look Once), SSD (Single Shot Detector, or SingleShot MultiBox Detector) can be used.
[0131] In addition, the learning data of the machine learning model for image segmentation uses a tomographic image acquired by OCT as input data, and uses a labeled image in which a region is labeled for each pixel of the tomographic image as output data. As the labeled image, for example, an image labeled with the inner limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner-outer segment junction (ISOS), retinal pigment epithelium layer (RPE), Bruch's membrane (BM), and choroid can be used. Note that, as other regions, for example, images labeled with the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, iris, and crystalline lens may be used.
[0132] In addition, the input data of the machine learning model for image segmentation is not limited to a tomographic image. It may be an anterior eye image, an SLO image, an OCTA image, or the like. In this case, the learning data may be various images as input data, and labeled images in which the pixel of each image is labeled with a region name or the like as output data. For example, when the input data of the learning data is an SLO image, the output data may be an image labeled with the peripheral part of the optic disc, Disc, Cup, and the like.
[0133] The labeled image used as output data may be an image in which a doctor or the like has labeled each region in a tomographic image, or an image in which a label has been added to each region by a rule-based region detection process. However, if machine learning is performed using a labeled image that has not been properly labeled as output data of the training data, the image obtained using a trained model trained using the training data may also be a labeled image that has not been properly labeled. Therefore, by removing pairs including such labeled images from the training data, it is possible to reduce the possibility that an inappropriate labeled image is generated using the trained model. Here, the rule-based region detection process refers to a detection process that utilizes a known regularity, such as the regularity of the shape of the retina.
[0134] The control unit 117 is expected to perform image segmentation processing using such a trained model for image segmentation, thereby quickly and accurately detecting specific regions in various images. Note that a trained model for image segmentation may also be prepared for each type of image that is input data. For OCTA front images and En-Face images, a trained model may also be prepared for each depth range for generating an image. Furthermore, the trained model for image segmentation may also be trained for images for each imaging site (e.g., center of the macula, center of the optic disc), or may be trained regardless of imaging site.
[0135] In addition, for the trained model for image segmentation, additional training may be performed using data manually corrected according to the operator's instructions as training data. In addition, the need for additional training and the need to transmit data to the server may be determined in a similar manner. In these cases, it is expected that the accuracy of each process can be improved and that processing can be performed according to the examiner's preferences.
[0136] Furthermore, when the control unit 117 detects partial regions (e.g., a region of interest, an artifact region, an abnormal region, etc.) of the subject's eye E using the learned model, it can also perform a predetermined image processing for each detected partial region. As an example, a case will be described where at least two partial regions of the vitreous region, the retina region, and the choroid region are detected. In this case, when performing image processing such as contrast adjustment on at least two detected partial regions, adjustments suitable for each region can be performed by using different image processing parameters. By displaying an image in which adjustments suitable for each region have been performed, the operator can more appropriately diagnose diseases, etc. for each partial region. Note that the configuration of using different image processing parameters for each detected partial region may be similarly applied to partial regions of the subject's eye E obtained by detecting the partial regions of the subject's eye E without using the learned model.
[0137] (Variation 5) The display control unit 121 in the above-mentioned various embodiments and modifications may display analysis results such as the thickness of a desired layer and various blood vessel densities on a report screen of the display screen after capturing a tomographic image. In addition, the display control unit 121 may display, as analysis results, parameter values (distributions) related to a region of interest including at least one of the optic disc, macular region, blood vessel region, capillary region, arterial region, venous region, nerve fiber bundle, vitreous region, macular region, choroid region, sclera region, lamina cribrosa region, retinal layer boundary, retinal layer boundary edge, photoreceptor cells, blood cells, blood vessel wall, blood vessel inner wall boundary, blood vessel outer boundary, ganglion cells, corneal region, iridocorneal region, Schlemm's canal, etc. Here, the region of interest may be, for example, a vortex vein, which is an outlet to the outside of the eye of a blood vessel in the Haller layer (an example of a blood vessel in a partial depth range of the choroid region). In this case, the parameters related to the site of interest may be, for example, the number of vortex veins (for example, the number per region), the distance from the optic disc to each vortex vein, the angle at which each vortex vein is located with the optic disc as the center, etc. This makes it possible to accurately diagnose, for example, various diseases related to Pachychoroid (thickened choroid) (for example, choroidal neovascularization). In addition, for example, by analyzing medical images to which various artifact reduction processes have been applied, the above-mentioned various analysis results can be displayed as accurate analysis results. Note that the artifacts may be, for example, false image regions caused by light absorption by blood vessel regions, projection artifacts, band-like artifacts in front images caused in the main scanning direction of the measurement light due to the state of the subject's eye (movement, blinking, etc.), etc. In addition, the artifacts may be any image loss regions that randomly occur on a medical image of a specific part of the subject for each shooting. The display control unit 121 may also cause the display unit 116 to display, as an analysis result, parameter values (distribution) for a region including at least one of the various artifacts (missing regions) described above. Also, the display unit 116 may display, as an analysis result, parameter values (distribution) for a region including at least one of abnormal sites such as drusen, neovascularization, vitiligo (hard exudate), and pseudodrusen. Also, a comparison result obtained by comparing the analysis result with a standard value or standard range obtained using a standard database may be displayed.
[0138] The analysis results may be displayed as an analysis map, sectors showing statistics corresponding to each divided region, etc. The analysis results may be generated using a trained model (analysis result generation engine, trained model for generating analysis results) obtained by learning the analysis results of medical images as training data. In this case, the trained model may be obtained by learning using training data including a medical image and the analysis results of the medical image, or training data including a medical image and the analysis results of a medical image of a type different from the medical image.
[0139] Furthermore, the learning data for performing image analysis may include a label image generated using a trained model for image segmentation processing and an analysis result of a medical image using the label image. In this case, the control unit 117 can function as an example of an analysis result generating unit that generates an analysis result of a tomographic image from the result of image segmentation processing using a trained model for generating analysis results. Furthermore, the trained model may be obtained by learning using training data including input data that is a set of multiple medical images of different types of a predetermined part, such as an En-Face image and a motion contrast front image (En-Face image of OCTA) described later.
[0140] The analysis result obtained using the high-quality image generated using the high-quality model may be displayed. In this case, the input data included in the learning data may be a high-quality image generated using a trained model for image quality improvement, or a set of a low-quality image and a high-quality image. The learning data may be an image in which at least a part of an image has been manually or automatically corrected with respect to an image whose quality has been improved using the trained model.
[0141] The learning data may be data obtained by labeling (annotating) the input data with information including at least one of analytical values (e.g., average values, medians, etc.) obtained by analyzing the analytical region, a table including analytical values, an analytical map, and the position of the analytical region such as a sector in an image, as correct answer data (in supervised learning).The analysis results obtained using the trained model for generating analysis results may be displayed in response to an instruction from an operator.
[0142] In addition, the estimation unit 119 in the above-mentioned embodiment and modified example can output a highly accurate estimation result by using, for example, an image to which various artifact reduction processes as described above have been applied for the estimation process. In addition, the display control unit 121 may display the position of the identified abnormal part or the like on the image as the estimation result, or may display the state of the abnormal part or the like by characters or the like. Furthermore, the display control unit 121 may display the classification result of the abnormal part or the like (for example, Curtin classification) as a diagnosis result separately from the estimation result about the disease. In addition, as the classification result, for example, information indicating the certainty of each abnormal part (for example, a numerical value indicating a ratio) may be displayed. In addition, information necessary for a doctor to confirm the diagnosis may be displayed as a diagnosis result. As the necessary information, for example, advice such as additional imaging may be considered. For example, when an abnormal part is detected in a blood vessel region in an OCTA image, it may be displayed that an additional fluorescent imaging using a contrast agent capable of observing blood vessels in more detail than OCTA will be performed. In addition, the diagnosis result may be information regarding the subject's future medical treatment policy, etc. The diagnosis result may include at least one of the following: diagnosis name, type and condition (degree) of the lesion (abnormal site), position of the lesion in the image, position of the lesion relative to the region of interest, findings (image findings, etc.), grounds for the diagnosis name (positive medical support information, etc.), and grounds for denying the diagnosis name (negative medical support information). In this case, for example, a diagnosis result that is more likely than the diagnosis result such as the diagnosis name input in response to an instruction from the examiner may be displayed as medical support information. In addition, when multiple types of medical images are used, for example, a type of medical image that may be the grounds for the diagnosis result may be displayed in an identifiable manner. In addition, the grounds for the diagnosis result may be a map (attention map, activation map) that visualizes the features extracted by the trained model, for example, a color map (heat map) that shows the features in color. In this case, for example, the heat map may be superimposed on the medical image that is the input data.The heat map can be obtained, for example, using Gradient-weighted Class Activation Mapping (Grad-CAM) or Guided Grad-CAM, which are techniques for visualizing areas that contribute greatly to the output value of a predicted (estimated) class (areas with large gradients).
[0143] The diagnosis result may be generated using a trained model (a diagnosis result generation engine, a trained model for generating a diagnosis result) obtained by learning using a diagnosis result of a medical image as learning data. The trained model may be obtained by learning using training data including a medical image and a diagnosis result of the medical image, or training data including a medical image and a diagnosis result of a medical image of a different type from the medical image.
[0144] The learning data may include a labeled image generated using a trained model for image segmentation processing and a diagnosis result of a medical image using the labeled image. In this case, the control unit 117 can function as an example of a diagnosis result generating unit that generates a diagnosis result of a tomographic image from the result of the image segmentation processing using, for example, a trained model for generating a diagnosis result.
[0145] Furthermore, the diagnostic result obtained using the high-quality image generated using the trained model for image quality improvement may be displayed. In this case, the input data included in the training data may be a high-quality image generated using the trained model for image quality improvement, or a set of a low-quality image and a high-quality image. Note that the training data may be an image in which at least a part of an image has been manually or automatically corrected with respect to an image whose image quality has been improved using the trained model.
[0146] The learning data may be data obtained by labeling (annotating) the input data with information including at least one of the following as correct answer data (in supervised learning): diagnosis name, type and condition (degree) of lesion (abnormal site), position of lesion in image, position of lesion relative to the region of interest, findings (image findings, etc.), grounds for diagnosis name (positive medical support information, etc.), grounds for denying diagnosis name (negative medical support information), etc. The diagnosis result obtained using the trained model for generating diagnosis result may be displayed in response to an instruction from the examiner.
[0147] A trained model may be prepared for each piece of information or each type of information used as input data, and a diagnosis result may be obtained using the trained model. In this case, a statistical process may be performed on the information output from each trained model to determine a final diagnosis result. For example, the proportion of information output from each trained model may be added for each type of information, and information having a higher total proportion than other information may be determined as a final diagnosis result. The statistical process is not limited to calculating the sum, and may be calculating an average value or a median value. For example, a diagnosis result may be determined using information having a higher proportion (the information with the highest proportion) than other information among the information output from each trained model. Similarly, a diagnosis result may be determined using information having a proportion equal to or greater than a threshold among the information output from each trained model.
[0148] Also, the system may be configured to be capable of judging (approving) whether the determined diagnostic result is good or bad according to an instruction (selection) of the operator. Also, the diagnostic result may be determined from the information output from each trained model according to an instruction (selection) of the operator. At this time, for example, the display control unit 121 may display the information output from each trained model and its ratio side by side on the display unit 116. Then, the system may be configured such that the operator selects, for example, information having a higher ratio than other information, and the selected information is determined as the diagnostic result. Furthermore, the diagnostic result may be determined using a machine learning model from the information output from each trained model. In this case, the machine learning algorithm may be a different type of machine learning algorithm from the machine learning algorithm used to generate the diagnostic result, and for example, a neural network, a support vector machine, an adaboost, a Bayesian network, or a random forest may be used.
[0149] The learning of the various trained models described above may be not only supervised learning (learning with labeled training data) but also semi-supervised learning. Semi-supervised learning is a method in which, for example, after a plurality of classifiers (classifiers) perform supervised learning, unlabeled training data is classified (classified), and the classification results (classification results) are automatically labeled (annotated) according to the reliability of the classification results (classification results) (for example, classification results with a probability equal to or higher than a threshold), and learning is performed with the labeled training data. Semi-supervised learning may be, for example, co-training (Co-Training or Multiview). In this case, the trained model for generating a diagnosis result may be a trained model obtained by semi-supervised learning (for example, co-training) using, for example, a first classifier that classifies medical images of normal subjects and a second classifier that classifies medical images including a specific lesion. Note that the purpose is not limited to diagnosis, and may be, for example, imaging support. In this case, the second classifier may be, for example, a classifier that classifies medical images including partial regions such as a site of interest or an artifact region.
[0150] Furthermore, the display control unit 121 according to the above-described various embodiments and modifications may display the object recognition results (object detection results) and segmentation results of partial regions such as the above-described site of interest, artifact region, and abnormal site on the report screen of the display screen. At this time, for example, a rectangular frame or the like may be superimposed around the object on the image. Also, for example, a color or the like may be superimposed on the object in the image. Note that the object recognition results and segmentation results may be generated using a trained model (object recognition engine, trained model for object recognition, segmentation engine, trained model for segmentation) obtained by learning learning data in which information indicating object recognition and segmentation is used as correct answer data to label (annotate) a medical image. Note that the above-described analysis result generation and diagnosis result generation may be obtained by using the above-described object recognition result and segmentation result. For example, the analysis result generation and diagnosis result generation process may be performed on the site of interest obtained by the object recognition and segmentation process.
[0151] Furthermore, when detecting an abnormal region, the control unit 117 may use a Generative Adversarial Network (GAN) or a Variational Auto-Encoder (VAE). For example, a Deep Convolutional GAN (DCGAN) consisting of a generator obtained by learning to generate medical images and a classifier obtained by learning to distinguish between new medical images generated by the generator and real medical images can be used as a machine learning model.
[0152] When DCGAN is used, for example, a discriminator encodes an input medical image into a latent variable, and a generator generates a new medical image based on the latent variable. Then, the difference between the input medical image and the new medical image generated can be extracted (detected) as an abnormal area. When VAE is used, for example, an input medical image is encoded by an encoder into a latent variable, and a new medical image is generated by decoding the latent variable by a decoder. Then, the difference between the input medical image and the new medical image generated can be extracted as an abnormal area.
[0153] Furthermore, the control unit 117 may detect an abnormal area using a convolutional auto-encoder (CAE). When using a CAE, the same medical image is made to learn as input data and output data during learning. As a result, when a medical image having an abnormal area is input to the CAE during estimation, a medical image having no abnormal area is output according to the learning tendency. After that, the difference between the medical image input to the CAE and the medical image output from the CAE can be extracted as the abnormal area.
[0154] In these cases, the control unit 117 can generate information on the difference between a medical image obtained using a generative adversarial network or an autoencoder and a medical image input to the generative adversarial network or an autoencoder as information on an abnormal part. This is expected to enable the control unit 117 to detect an abnormal part quickly and accurately. For example, even if it is difficult to collect a large number of medical images including an abnormal part as training data in order to improve the detection accuracy of an abnormal part, medical images of normal subjects, which are relatively easy to collect in large numbers, can be used as training data. Therefore, for example, learning for detecting an abnormal part with high accuracy can be efficiently performed. Here, the autoencoder includes VAE, CAE, and the like. In addition, at least a part of the generation unit of the generative adversarial network may be composed of VAE. This makes it possible to generate a relatively clear image while reducing the phenomenon of generating similar data, for example. For example, the control unit 117 can generate information on the difference between a medical image obtained from various medical images using a generative adversarial network or an autoencoder and a medical image input to the generative adversarial network or the autoencoder as information on an abnormal part. Furthermore, for example, the display control unit 121 can cause the display unit 116 to display information regarding the difference between a medical image obtained from various medical images using a generative adversarial network or an autoencoder and the medical image input to the generative adversarial network or the autoencoder as information regarding abnormal areas.
[0155] In particular, the trained model for generating a diagnosis result may be a trained model obtained by training with training data including input data that is a set of multiple medical images of a specific part of a subject. In this case, the input data included in the training data may be, for example, input data that is a set of a motion contrast front image and a luminance front image (or a luminance tomographic image) of the fundus. In addition, the input data included in the training data may be, for example, input data that is a set of a tomographic image (B-scan image) of the fundus and a color fundus image (or a fluorescent fundus image). In addition, the multiple medical images of different types may be anything obtained by different modalities, different optical systems, or different principles.
[0156] In particular, the trained model for generating a diagnosis result may be a trained model obtained by training with training data including input data consisting of a set of multiple medical images of different parts of a subject. In this case, the input data included in the training data may be, for example, input data consisting of a set of a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment. In addition, the input data included in the training data may be, for example, input data consisting of a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic disc of the fundus.
[0157] The input data included in the learning data may be a plurality of medical images of different parts and different types of the subject. In this case, the input data included in the learning data may be, for example, input data including a set of a tomographic image of the anterior eye and a color fundus image. The above-mentioned learned model may be a learned model obtained by learning with learning data including input data including a set of a plurality of medical images of a specific part of the subject with different shooting angles of view. The input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a specific part into a plurality of regions, such as a panoramic image. In this case, by using a wide-angle image such as a panoramic image as the learning data, there is a possibility that the feature amount of the image can be accurately acquired because it has a larger amount of information than a narrow-angle image, and therefore the result of the processing can be improved. The input data included in the learning data may be a set of a plurality of medical images of a specific part of the subject at different dates and times.
[0158] In addition, the display screen on which at least one of the estimation result, analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to a report screen. Such a display screen may be displayed on at least one of a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (a display screen on which various live moving images are displayed), for example. For example, by displaying at least one of the results obtained using the trained model on the shooting confirmation screen, the operator can confirm the highly accurate result even immediately after shooting.
[0159] Also, for example, when a specific object is recognized, a frame surrounding the recognized object may be superimposed on the live video image. At this time, when information indicating the certainty of the object recognition result (for example, a numerical value indicating a ratio) exceeds a threshold, the frame surrounding the object may be highlighted, for example, by changing the color. This allows the examiner to easily identify the object on the live video image.
[0160] In addition, a trained model for generating correct answer data for generating correct answer data such as labeling (annotation) may be used to generate correct answer data used for training the various trained models described above. In this case, the trained model for generating correct answer data may be obtained by additionally learning (sequentially) correct answer data obtained by labeling (annotation) by the examiner. In other words, the trained model for generating correct answer data may be obtained by additionally learning training data in which data before labeling is used as input data and data after labeling is used as output data. In addition, in a plurality of consecutive frames such as a moving image, the result of a frame determined to have low accuracy of the result may be corrected taking into consideration the results of object recognition, segmentation, etc. of the previous and following frames. In this case, the trained model may be configured to additionally learn the corrected result as correct answer data in response to an instruction from the examiner. In addition, for example, for a medical image with low accuracy of the result, the trained model may be configured to additionally learn an image labeled (annotated) as input data while the examiner checks a color map (heat map) showing the feature amounts in color, which is an example of a map (attention map, activation map) that visualizes the feature amounts extracted by the trained model on the medical image. For example, if the location to be noted in a heat map on a layer immediately before outputting the results of the trained model is different from the examiner's intention, additional learning may be performed on a medical image in which the location that the examiner thinks should be noted is labeled (annotated). This allows the trained model to additionally train the feature amounts of a partial region on a medical image that has a relatively large influence on the output result of the trained model by prioritizing (weighting) it over other regions.
[0161] Here, the above-mentioned various trained models can be obtained by machine learning using training data. Machine learning includes, for example, deep learning consisting of a multi-layered neural network. For at least a part of the multi-layered neural network, for example, a convolutional neural network can be used. For at least a part of the multi-layered neural network, a technology related to an autoencoder (autoencoder) may be used. For learning, a technology related to backpropagation (backpropagation method) may be used. For learning, a method (dropout) of randomly inactivating each unit (each neuron or each node) may be used. For learning, a method (batch normalization) of normalizing data transmitted to each layer of a multi-layered neural network before an activation function (for example, a ReLu function) is applied may be used. However, the machine learning is not limited to deep learning, and any learning using a model that can extract (express) the feature amount of training data such as an image by itself through learning may be used. Here, the machine learning model refers to a learning model by a machine learning algorithm such as deep learning. A trained model is a model that has been trained (learned) in advance using appropriate training data for a machine learning model based on an arbitrary machine learning algorithm. However, a trained model is not one that does not undergo further learning, but rather is capable of additional learning. Training data is composed of a pair of input data and output data (correct answer data). Here, training data may be called teacher data, or correct answer data may be called teacher data.
[0162] In addition, the GPU can perform efficient calculations by processing more data in parallel. Therefore, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing with the GPU. Therefore, in this modified example, a GPU is used in addition to a CPU for processing by the control unit 117, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, the CPU and the GPU work together to perform calculations to perform learning. In addition, the processing of the learning unit may be performed only by the CPU or the GPU. In addition, a processing unit (estimation unit 119) that executes processing using the above-mentioned various learned models may also use a GPU like the learning unit. In addition, the learning unit may include an error detection unit and an update unit, not shown. The error detection unit obtains an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the correct answer data. The error detection unit may calculate the error between the output data from the neural network and the correct answer data using a loss function. Furthermore, the update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. The update unit updates the connection weighting coefficients using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.
[0163] As a machine learning model used for the above-mentioned object recognition, segmentation, image quality improvement, etc., a U-net type machine learning model having an encoder function consisting of multiple layers including multiple downsampling layers and a decoder function consisting of multiple layers including multiple upsampling layers can be applied. In the U-net type machine learning model, the position information (spatial information) made ambiguous in the multiple layers configured as the encoder can be used in the same dimensional layers (layers corresponding to each other) in the multiple layers configured as the decoder (for example, by using skip connections).
[0164] In addition, as a machine learning model used for the above-mentioned object recognition, segmentation, image quality improvement, etc., for example, FCN (Fully Convolutional Network), SegNet, etc. may be used. Also, a machine learning model that performs object recognition on a region-by-region basis may be used according to a desired configuration. As a machine learning model that performs object recognition, for example, RCNN (Region CNN), fastRCNN, or fasterRCNN may be used. Furthermore, as a machine learning model that performs object recognition on a region-by-region basis, YOLO (You Only Look Once), SSD (Single Shot Detector, or Single Shot MultiBox Detector) may be used.
[0165] The machine learning model may be, for example, a capsule network (CapsNet). Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image is reduced. This allows learning to be performed in such a way that, for example, the effects of local distortion and translation of an image are reduced. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is maintained. This allows learning to be performed in such a way that, for example, the spatial positional relationship between features in an image is taken into consideration.
[0166] (Variation 6) In the preview screen in the various embodiments and modifications described above, the various trained models described above may be used for at least one frame of the live video. In this case, when multiple live videos of different parts or different types are displayed on the preview screen, the trained models corresponding to each live video may be used. This allows, for example, the processing time to be shortened even for live videos, so that the examiner can obtain highly accurate information before the start of imaging. Therefore, for example, failures in re-imaging can be reduced, and the accuracy and efficiency of diagnosis can be improved.
[0167] The multiple live video images may be, for example, video images of the anterior eye for alignment in the XYZ directions, and front video images of the fundus for focus adjustment of the fundus observation optical system and OCT focus adjustment. The multiple live video images may also be, for example, tomographic video images of the fundus for coherence gate adjustment of OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length). When such a preview image is displayed, the control unit 117 may be configured to perform the above-mentioned various adjustments so that the area detected using the above-mentioned trained model for object recognition or trained model for segmentation satisfies a predetermined condition. For example, the control unit 117 may be configured to perform various adjustments such as OCT focus adjustment so that a value (for example, a contrast value or an intensity value) related to a vitreous area or a predetermined retinal layer such as RPE detected using the trained model for object recognition or trained model for segmentation exceeds a threshold value (or becomes a peak value). In addition, for example, the OCT coherence gate may be adjusted so that a specific retinal layer such as the vitreous region or RPE detected using a trained model for object recognition or a trained model for segmentation is located at a specific position in the depth direction.
[0168] In these cases, the control unit 117 can use the learned model to perform image quality improvement processing on the moving image to generate a high-quality moving image. In addition, the drive control unit (not shown) can drive and control an optical member for changing the shooting range, such as a reference mirror, so that a partial area, such as a site of interest obtained by segmentation processing, etc., is located at a predetermined position in the display area while a high-quality moving image is displayed. In such a case, the drive control unit can automatically perform alignment processing so that a desired area is located at a predetermined position in the display area based on highly accurate information. Note that the optical member for changing the shooting range may be, for example, an optical member for adjusting a coherence gate position, and specifically, may be a reference mirror that reflects a reference light. In addition, the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member may be, for example, a mirror for changing the optical path length of the measurement light (not shown). Note that the optical member for changing the shooting range may be, for example, a stage unit (not shown). The drive control unit may drive and control the scanning means in response to an instruction to start imaging so that a partial area such as an artifact area obtained by segmentation processing or the like is imaged again (rescanned) during or at the end of imaging. Also, for example, when information indicating the likelihood of an object recognition result for a site of interest (for example, a numerical value indicating a ratio) exceeds a threshold, various adjustments, imaging start, etc. may be automatically performed. Also, for example, when information indicating the likelihood of an object recognition result for a site of interest (for example, a numerical value indicating a ratio) exceeds a threshold, each adjustment, imaging start, etc. may be changed to an executable state (cancellation of the execution prohibition state) in response to an instruction from an examiner.
[0169] In addition, the moving image to which the various learned models described above can be applied is not limited to a live moving image, and may be, for example, a moving image stored (saved) in the storage unit 114. At this time, for example, a moving image obtained by aligning at least one frame of the tomographic moving image of the fundus stored (saved) in the storage unit 114 may be displayed on the display screen. For example, when it is desired to preferably observe the vitreous body, a reference frame may be selected based on a condition that the vitreous body is present on the frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Then, a moving image in which other frames are aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a high-quality image (high-quality frame) generated sequentially by the learned model for image quality improvement for at least one frame of the moving image may be configured to be continuously displayed.
[0170] As for the above-mentioned method of inter-frame alignment, the method of alignment in the X direction and the method of alignment in the Z direction (depth direction) may be the same or different methods may be applied. The alignment in the same direction may be performed multiple times using different methods. For example, after performing a rough alignment, a precise alignment may be performed. As the alignment method, for example, there are (rough Z direction) alignment using a retinal layer boundary obtained by segmenting a tomographic image (B-scan image), (precise X-direction and Z-direction) alignment using correlation information (similarity) between a reference image and a plurality of regions obtained by dividing a tomographic image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), (X direction) alignment using a two-dimensional front image, etc. Also, it may be configured such that rough alignment is performed in pixel units and then precise alignment is performed in subpixel units.
[0171] Here, during various adjustments, it is possible that the subject, such as the retina of the subject's eye, has not yet been imaged properly. For this reason, there is a possibility that a high-quality image cannot be obtained with high accuracy due to a large difference between the medical image input to the trained model and the medical image used as training data. Therefore, when an evaluation value such as an image quality evaluation of a tomographic image (B-scan) exceeds a threshold, the display of a high-quality moving image (continuous display of high-quality frames) may be automatically started. In addition, when an evaluation value such as an image quality evaluation of a tomographic image (B-scan) exceeds a threshold, the image quality improvement button may be changed to a state (active state) that can be specified by the examiner.
[0172] Also, for example, a trained model for improving image quality may be prepared for each imaging mode having a different scan pattern, etc., and a trained model for improving image quality corresponding to a selected imaging mode may be selected. Also, one trained model for improving image quality obtained by training learning data including various medical images obtained in different imaging modes may be used.
[0173] (Variation 7) In the above-described embodiment and modified example, when various trained models are undergoing additional learning, it may be difficult to output (inference / prediction) using the trained model itself undergoing additional learning. For this reason, it is preferable to configure the system so that input of medical images other than the training data to the trained model undergoing additional learning is prohibited. In addition, another trained model that is the same as the trained model before the additional learning is performed may be prepared as a spare trained model. In this case, it is preferable to configure the system so that input of medical images other than the training data to the spare trained model is possible during the additional learning. Then, after the additional learning is completed, the trained model after the additional learning is evaluated, and if there is no problem, the spare trained model may be replaced with the trained model after the additional learning is performed. In addition, if there is a problem, the spare trained model may be used.
[0174] In addition, for the evaluation of the trained model after the additional learning is performed, for example, a trained model for classification for classifying high-quality images obtained by the trained model for image quality improvement from other types of images may be used. The trained model for classification may be, for example, a trained model obtained by learning training data including a plurality of images including a high-quality image and a low-quality image obtained by the trained model for image quality improvement as input data and data in which the types of these images are labeled (annotated) as correct answer data. In this case, the type of image in the input data at the time of estimation (prediction) may be displayed together with information indicating the likelihood of each type of image included in the correct answer data at the time of learning (for example, a numerical value indicating a ratio). In addition to the above images, the input data of the trained model for classification may include high-quality images that have been subjected to high contrast, noise reduction, etc. by superimposing a plurality of low-quality images (for example, averaging a plurality of low-quality images obtained by positioning), etc. In addition, the trained model after the additional learning may be evaluated by, for example, comparing multiple high-quality images obtained from the same image using the trained model after the additional learning and the trained model before the additional learning (backup trained model), or comparing the analysis results of the multiple high-quality images. At this time, for example, it may be determined whether the comparison result of the multiple high-quality images (an example of a change due to additional learning) or the comparison result of the analysis results of the multiple high-quality images (an example of a change due to additional learning) is within a predetermined range, and the determination result may be displayed.
[0175] Also, a trained model obtained by training for each imaging site may be selectively used. Specifically, a plurality of trained models including a first trained model obtained using training data including a first imaging site (e.g., an anterior eye part, a posterior eye part, etc.) and a second trained model obtained using training data including a second imaging site different from the first imaging site may be prepared. The control unit 117 may have a selection means for selecting one of the plurality of trained models. In this case, the control unit 117 may have a control means for executing additional training for the selected trained model. In response to an instruction from the examiner, the control means may search for data in which an imaging site corresponding to the selected trained model and a photographed image of the imaging site are paired, and perform training using the searched data as training data as additional training for the selected trained model. The imaging site corresponding to the selected trained model may be obtained from information in the header of the data or may be manually input by the examiner. Also, the search for data may be performed via a network from a server or the like of an external facility such as a hospital or a research institute. This enables efficient additional learning for each imaging part using images of the imaging part corresponding to the trained model.
[0176] The selection means and the control means may be configured by a software module executed by a processor such as a CPU or MPU of the control unit 117. The selection means and the control means may also be configured by a circuit performing a specific function such as an ASIC, an independent device, or the like.
[0177] In addition, when acquiring the learning data for additional learning via a network from a server or the like of an external facility such as a hospital or research institute, it is useful to reduce a decrease in reliability due to tampering or a system trouble during additional learning. Therefore, the validity of the learning data for additional learning may be detected by checking the consistency by digital signature or hashing. This makes it possible to protect the learning data for additional learning. In this case, if the validity of the learning data for additional learning cannot be detected as a result of checking the consistency by digital signature or hashing, a warning to that effect is issued and additional learning using the learning data is not performed. The server may be installed anywhere and may be in any form, such as a cloud server, a fog server, an edge server, or the like.
[0178] In addition, the protection of data by confirming the consistency as described above is not limited to learning data for additional learning, but can be applied to data including medical images. The image management system may be configured so that transactions of data including medical images between servers of multiple facilities are managed by a distributed network. The image management system may be configured so that multiple blocks in which a transaction history and a hash value of a previous block are recorded together are linked in chronological order. Note that, as a technology for performing the confirmation of consistency, encryption (e.g., lattice encryption, quantum encryption by quantum key distribution, etc.) that is difficult to calculate even using a quantum computer such as a quantum gate method may be used. Here, the image management system may be a device and a system that receives and stores images captured by an imaging device or images that have been processed. In addition, the image management system can transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system may include, for example, a picture archiving and communication system (PACS). In addition, the image management system includes a database that can store various information such as information on subjects associated with received images and imaging times. In addition, the image management system is connected to a network and can send and receive images, convert images, and send and receive various information associated with stored images in response to requests from other devices.
[0179] When additional learning is performed on various trained models, a GPU can be used for high-speed processing. Since a GPU can perform efficient calculations by processing a large amount of data in parallel, it is effective to use a GPU for processing when performing learning multiple times using a learning model such as deep learning. The additional learning process may be performed by a GPU working in cooperation with a CPU, etc.
[0180] (Variation 8) In the various embodiments and modified examples described above, the examiner's instructions may be instructions by voice or the like, in addition to manual instructions (e.g., instructions using a user interface or the like). In this case, for example, a machine learning model including a voice recognition model (voice recognition engine, trained model for voice recognition) obtained by machine learning may be used. The manual instructions may also be instructions by character input or the like using a keyboard, touch panel, or the like. In this case, for example, a machine learning model including a character recognition model (character recognition engine, trained model for character recognition) obtained by machine learning may be used. The examiner's instructions may also be instructions by gestures or the like. In this case, a machine learning model including a gesture recognition model (gesture recognition engine, trained model for gesture recognition) obtained by machine learning may be used.
[0181] Furthermore, the instructions from the examiner may be a gaze detection result of the examiner on the display screen of the display unit 116. The gaze detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing the periphery of the display screen of the display unit 116. In this case, the pupil detection from the moving image may use an object recognition engine as described above. Furthermore, the instructions from the examiner may be instructions based on brain waves, weak electrical signals flowing through the body, or the like.
[0182] In such a case, for example, the learning data may be learning data in which the input data is character data or voice data (waveform data) indicating an instruction to display the results of the processing of the various learned models as described above, and the correct answer data is an execution command for actually displaying the results of the processing of the various learned models on the display unit 116. In addition, the learning data may be learning data in which the correct answer data is, for example, an execution command for whether or not to automatically set the shooting parameters and an execution command for changing the button for the command to an active state. Note that, as the learning data, any data may be used as long as the instruction contents indicated by the character data or voice data correspond to the execution command contents. In addition, the voice data may be converted into character data using an acoustic model or a language model. In addition, a process of reducing noise data superimposed on the voice data may be performed using waveform data obtained by multiple microphones. In addition, the device may be configured to be able to select between instructions by characters or voice and instructions by a mouse or touch panel according to an instruction from an examiner. In addition, the device may be configured to be able to select whether instructions by characters or voice are on or off according to an instruction from an examiner.
[0183] Here, the machine learning includes deep learning as described above, and for example, a recurrent neural network (RNN) can be used for at least a part of a multi-layered neural network. Here, as an example of a machine learning model according to this modification, an RNN, which is a neural network that handles time-series information, will be described with reference to FIGS. 10(a) and (b). In addition, a long short-term memory (hereinafter, LSTM), which is a type of RNN, will be described with reference to FIGS. 11(a) and (b).
[0184] FIG. 10(a) shows the structure of an RNN, a machine learning model. The RNN 3520 has a loop structure in the network, and at time t, data x t Enter 3510 and enter data h tThe RNN 3520 outputs 3530. Since the RNN 3520 has a loop function in the network, it is possible to pass the current state to the next state, so it can handle time-series information. Figure 10(b) shows an example of input and output of a parameter vector at time t. Data x t 3510 contains N pieces of data (Params1 to ParamsN). In addition, the data h output from the RNN 3520 t 3530 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.
[0185] However, since RNN cannot handle long-term information during backpropagation, LSTM is sometimes used. LSTM can learn long-term information by equipping it with a forget gate, an input gate, and an output gate. Here, the structure of LSTM is shown in Figure 11(a). In LSTM3540, the information that the network takes over at the next time t is the internal state of the network called a cell, c t-1 and output data h t-1 The lowercase letters (c, h, x) in the figure represent vectors.
[0186] Next, the details of LSTM3540 are shown in Figure 11(b). In Figure 11(b), FG is the forget gate network, IG is the input gate network, and OG is the output gate network, each of which is a sigmoid layer. Therefore, each element outputs a vector with a value between 0 and 1. The forget gate network FG decides how much past information to retain, and the input gate network IG decides which value to update. CU is the cell update candidate network, an activation function tanh layer. It creates a vector of new candidate values to be added to the cell. The output gate network OG selects the cell candidate elements and selects how much information to convey at the next time.
[0187] The above-mentioned LSTM model is a basic form, and is not limited to the network shown here. The connection between networks may be changed. A QRNN (Quasi Recurrent Neural Network) may be used instead of LSTM. Furthermore, the machine learning model is not limited to a neural network, and boosting, a support vector machine, etc. may be used. In addition, when the instructions from the examiner are input by characters or voice, etc., a technology related to natural language processing (e.g., Sequence to Sequence) may be applied. In this case, as the technology related to natural language processing, for example, a model that is output for each input sentence may be applied. In addition, the above-mentioned various trained models may be applied to outputs to the examiner, not limited to instructions from the examiner. In addition, a dialogue engine (dialogue model, trained model for dialogue) that responds to the examiner by output by characters or voice, etc. may be applied.
[0188] As a technique for natural language processing, a trained model obtained by pre-training document data by unsupervised learning may be used. As a technique for natural language processing, a trained model obtained by further performing transfer learning (or fine tuning) on a trained model obtained by pre-training according to a purpose may be used. As a technique for natural language processing, for example, BERT (Bidirectional Encoder Representations from Transformers) may be applied. As a technique for natural language processing, a model capable of extracting (representing) a context (feature amount) by predicting a specific word in a sentence from both the left and right contexts may be applied. As a technique for natural language processing, a model capable of determining the relationship (continuity) of two sequences (sentences) in input time series data may be applied. As a technique for natural language processing, a model in which a Transformer Encoder is used in a hidden layer and a sequence of vectors is input and output may be applied.
[0189] Here, the instructions from the examiner to which this modification is applicable may be any instructions related to at least one of changing the display of various images and analysis results as described in the various embodiments and modifications described above, selecting a depth range for generating an En-Face image, selecting whether to use as learning data for additional learning, selecting a learned model, outputting (displaying, transmitting, etc.) or saving the results obtained using various learned models, etc. Furthermore, the instructions from the examiner to which this modification is applicable may be instructions not only after shooting but also before shooting, and may be, for example, instructions related to various adjustments, instructions related to setting various shooting conditions, and instructions related to starting shooting. Furthermore, the instructions from the examiner to which this modification is applicable may be instructions related to changing the display screen (screen transition).
[0190] The machine learning model may be a machine learning model that combines a machine learning model related to images such as CNN and a machine learning model related to time-series data such as RNN. In such a machine learning model, for example, the relationship between the feature amount related to the image and the feature amount related to the time-series data can be learned. When the input layer side of the machine learning model is CNN and the output layer side is RNN, for example, learning may be performed using learning data in which medical images are input data and text related to the medical images (for example, the presence or absence of a lesion, the type of lesion, the recommendation for the next examination, etc.) is output data. As a result, for example, medical information related to the medical images is automatically explained in text, so that even an examiner with little medical experience can easily understand the medical information related to the medical images. In addition, when the input layer side of the machine learning model is RNN and the output layer side is CNN, for example, learning may be performed using learning data in which medical text such as lesions, findings, and diagnoses is input data and medical images corresponding to the medical text are output data. As a result, for example, medical images related to a case that the examiner wants to confirm can be easily searched.
[0191] In addition, a machine translation engine (machine translation model, trained model for machine translation) that machine translates text such as characters and voice into an arbitrary language may be used for instructions from the examiner or output to the examiner. The arbitrary language may be configured to be selectable according to an instruction from the examiner. The arbitrary language may be configured to be automatically selected by using a trained model that automatically recognizes the type of language. The automatically selected language type may be configured to be correctable according to an instruction from the examiner. For example, the above-mentioned technology related to natural language processing (e.g., Sequence to Sequence) may be applied to the machine translation engine. For example, after a text input to the machine translation engine is machine-translated, the machine-translated text may be input to a character recognition engine or the like. For example, a text output from the above-mentioned various trained models may be input to the machine translation engine, and the text output from the machine translation engine may be output.
[0192] Also, the above-mentioned various trained models may be used in combination. For example, the configuration may be such that characters corresponding to an instruction from an examiner are input to a character recognition engine, and voice obtained from the input characters is input to another type of machine learning engine (for example, a machine translation engine, etc.). Also, for example, the configuration may be such that characters output from another type of machine learning engine are input to a character recognition engine, and voice obtained from the input characters is output. Also, for example, the configuration may be such that voice corresponding to an instruction from an examiner is input to a voice recognition engine, and characters obtained from the input voice are input to another type of machine learning engine (for example, a machine translation engine, etc.). Also, for example, the configuration may be such that voice output from another type of machine learning engine is input to a voice recognition engine, and characters obtained from the input voice are displayed on the display unit 116. At this time, it may be configured so that output by characters or output by voice can be selected according to an instruction from the examiner. Also, it may be configured so that input by characters or input by voice can be selected according to an instruction from the examiner. Also, the above-mentioned various configurations may be adopted according to the selection according to an instruction from the examiner.
[0193] (Variation 9) The label image and the high-quality image related to the image acquired by the main shooting may be stored in the storage unit 114 in response to an instruction from the operator. At this time, for example, after an instruction from the operator to save the high-quality image, when registering the file name, a file name including information (e.g., characters) indicating that the image was generated by processing (image quality improvement processing) using a trained model for image quality improvement may be displayed in an editable state in response to an instruction from the operator as a recommended file name in any part of the file name (e.g., the first part or the last part). Similarly, for the label image, etc., a file name including information that the image was generated by processing using a trained model may be displayed.
[0194] In addition, when a high-quality image is displayed on the display unit 116 on various display screens such as a report screen, a display indicating that the displayed image is a high-quality image generated by processing using a high-quality model may be displayed together with the high-quality image. In this case, the operator can easily identify that the displayed high-quality image is not the image itself acquired by shooting, and therefore it is possible to reduce misdiagnosis and improve diagnostic efficiency. The display indicating that the high-quality image is generated by processing using a high-quality model may be in any form as long as it is a display that can distinguish the input image from the high-quality image generated by the processing. In addition, for processing using various trained models as described above as well as processing using a high-quality model, a display indicating that the result is generated by processing using the trained model of that type may be displayed together with the result. For example, when displaying an analysis result of a segmentation result using a trained model for image segmentation processing, a display indicating that the analysis result is based on the result using a trained model for image segmentation may be displayed together with the analysis result.
[0195] At this time, the display screen such as the report screen may be stored as image data in the storage unit 114 in response to an instruction from the operator. For example, the report screen may be stored in the storage unit 114 as a single image in which high-quality images and an indication that these images are images generated by processing using a trained model are arranged side by side.
[0196] In addition, regarding the display indicating that the high quality image is generated by processing using the high quality model, a display indicating what kind of learning data the high quality model has been trained on may be displayed on the display unit 116. The display may include an explanation of the types of input data and correct answer data of the training data, or any display regarding correct answer data such as the imaging site included in the input data and correct answer data. Note that, for processing using the various trained models described above, such as image segmentation processing, for example, a display indicating what kind of learning data the trained model of that type has been trained on may be displayed on the display unit 116.
[0197] Also, information (e.g., text) indicating that the image was generated by processing using a trained model may be displayed or saved in a state where it is superimposed on the image, etc. In this case, the location to be superimposed on the image may be any area (e.g., the edge of the image) that does not overlap with an area in which a site of interest to be photographed is displayed. Also, a non-overlapping area may be determined, and the information may be superimposed on the determined area. Note that images obtained by processing using the various trained models described above, such as image segmentation processing, as well as processing using the high image quality model, may be processed in the same manner.
[0198] In addition, when the default setting is such that the high image quality processing button or the like is in an active state (high image quality processing is on) as the initial display screen of the report screen, a report image corresponding to the report screen including a high image quality image or the like may be transmitted to the server in response to an instruction from the examiner. In addition, when the default setting is such that the button is in an active state, a report image corresponding to the report screen including a high image quality image or the like may be (automatically) transmitted to the server at the end of the examination (for example, when the shooting confirmation screen or the preview screen is changed to the report screen in response to an instruction from the examiner). At this time, a report image generated based on various settings in the default setting (for example, settings related to at least one of the depth range for generating an En-Face image in the initial display screen of the report screen, whether or not the analysis map is superimposed, whether or not the image is a high image quality image, whether or not the display screen is for follow-up observation, etc.) may be transmitted to the server. Note that the same processing may be performed when the button represents switching of the image segmentation processing.
[0199] (Variation 10) In the above-described embodiment and modified example, among the various trained models as described above, an image obtained by a first type of trained model (e.g., a high-quality image, an image showing an analysis result such as an analysis map, an image showing a predetermined area detection result, an image showing a segmentation result) may be input to a second type of trained model different from the first type. At this time, a configuration may be made such that a result (e.g., an estimation result, an analysis result, a diagnosis result, a predetermined area detection result, a segmentation result) is generated by processing the second type of trained model.
[0200] Furthermore, among the various trained models described above, a result of processing a first type of trained model (e.g., an estimation result, an analysis result, a diagnosis result, a predetermined area detection result, a segmentation result) may be used to generate an image to be input to a second type of trained model different from the first type from an image input to the first type of trained model. In this case, the generated image is likely to be an image suitable for processing using the second type of trained model. For this reason, the accuracy of an image (e.g., a high-quality image, an image showing an analysis result such as an analysis map, an image showing a predetermined area detection result, an image showing a segmentation result) obtained by inputting the generated image into the second type of trained model can be improved.
[0201] In addition, a common image may be input to the first type of trained model and the second type of trained model, and each processing result using these trained models may be generated (or displayed). At this time, for example, the generation (or display) of each processing result using these trained models may be performed collectively (in conjunction) in response to an instruction from an examiner. Also, the type of image to be input (e.g., high-quality image, object recognition result, segmentation result, similar case image), the type of processing result to be generated (or displayed) (e.g., high-quality image, estimation result, diagnosis result, analysis result, object recognition result, segmentation result, similar case image), the type of input, the type of output (e.g., text, voice, language), etc. may be selected in response to an instruction from the examiner. Also, the type of input may be automatically selected by using a trained model that automatically recognizes the type of input. Also, the type of output may be automatically selected so as to correspond to the type of input (e.g., to be the same type). Also, the automatically selected type may be corrected in response to an instruction from the examiner. At this time, at least one trained model may be selected according to the selected type. In this case, when multiple trained models are selected, the method of combining the multiple trained models (for example, the order in which data is input) may be determined according to the selected type. For example, the type of image to be input and the type of processing result to be generated (or displayed) may be configured to be selectable to be different from each other, or, if they are the same, information prompting the examiner to select differently may be output to the examiner. In addition, each trained model may be executed in any location. For example, a part of the multiple trained models may be used in a cloud server, and the others may be used in another server such as a fog server or an edge server. In addition, when a network is configured to be capable of wireless communication within a facility, a site including a facility, or an area including multiple facilities, the reliability of the network may be improved by configuring the network to use radio waves in a dedicated wavelength band allocated exclusively to the facility, site, area, etc.The network may also be configured by wireless communication capable of high speed, large capacity, low latency, and multiple simultaneous connections. With these, for example, surgery on the vitreous body, cataract, glaucoma, corneal refractive correction, external eye, etc., and treatment such as laser photocoagulation can be supported in real time even if remotely. At this time, for example, a fog server or edge server that wirelessly receives at least one of various medical images obtained by the apparatus related to these surgery or treatment may be configured to wirelessly transmit information obtained using at least one of various trained models to the apparatus related to surgery or treatment. Also, for example, the information wirelessly received by the apparatus related to surgery or treatment may be the movement amount (vector) of the optical system or optical member as described above, and in this case, the apparatus related to surgery or treatment may be configured to be automatically controlled. Also, for example, for the purpose of supporting the operation by the examiner, it may be configured as automatic control (semi-automatic control) with the permission of the examiner.
[0202] In addition, a similar case image search may be performed using an external database stored in a server or the like, using an analysis result or a diagnosis result by processing the trained model as described above as a search key. In addition, a similar case image search may be performed using an external database stored in a server or the like, using an object recognition result or a segmentation result by processing various trained models as described above as a search key. In addition, in a case where a plurality of medical images stored in a database are managed in a state in which the feature amounts of each of the plurality of medical images are already added as auxiliary information by machine learning or the like, a similar case image search engine (similar case image search model, trained model for similar case image search) using the medical image itself as a search key may be used. For example, the control unit 117 can use a trained model for similar case image search (different from the trained model for image quality improvement) to search for similar case images related to the medical image from various medical images. In addition, for example, the display control unit 121 can display on the display unit 116 similar case images obtained from various medical images using the trained model for similar case image search. At this time, the similar case image is, for example, an image of a feature similar to the feature of the medical image input to the trained model. In addition, when a partial region such as an abnormal part is included in the medical image input to the trained model, the similar case image is, for example, an image of a feature similar to the feature of the partial region such as an abnormal part. Therefore, not only can learning for accurately searching for a similar case image be efficiently performed, but also, when an abnormal part is included in the medical image, the examiner can efficiently diagnose the abnormal part. In addition, a plurality of similar case images may be searched, and the plurality of similar case images may be displayed so that the order of similarity of the feature amount can be identified. In addition, the trained model for similar case image search may be configured to additionally train using training data including an image selected from the plurality of similar case images in response to an instruction from the examiner and the feature amount of the image.
[0203] Furthermore, the learning data for various trained models is not limited to data obtained using the ophthalmic device itself that actually takes the photographs, but may be data obtained using the same type of ophthalmic device or data obtained using the same kind of ophthalmic device, depending on the desired configuration.
[0204] The various trained models according to the above-described embodiments and modified examples can be provided in the control unit 117. The trained models may be configured, for example, by software modules executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured by a circuit that performs a specific function such as an ASIC. These trained models may also be provided in another server device connected to the control unit 117. In this case, the control unit 117 can use the trained models by connecting to a server or the like that has the trained models via any network such as the Internet. Here, the server that has the trained models may be, for example, a cloud server, a fog server, or an edge server. When configuring a network within a facility, a site that includes a facility, or an area that includes multiple facilities to be capable of wireless communication, the reliability of the network may be improved by configuring the network to use radio waves in a dedicated wavelength band that is limited to the facility, site, area, or the like. The network may also be configured by wireless communication that is capable of high speed, large capacity, low latency, and multiple simultaneous connections.
[0205] (Variation 11) The medical images processed by the control unit 117 according to the various embodiments and modifications described above include images acquired using any modality (imaging device, imaging method). The medical images to be processed may include medical images acquired by any imaging device or the like, or images created by a medical image processing device or a medical image processing method.
[0206] Furthermore, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. The medical image may also be an image that represents the structure (shape) of the predetermined part, or an image that represents its function. The image that represents the function includes, for example, an image that represents blood flow dynamics (blood flow rate, blood flow velocity, etc.) such as an OCTA image, a Doppler OCT image, an fMRI image, and an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be photographed, and includes any part such as the human eye (subject eye), the brain, the lungs, the intestines, the heart, the pancreas, the kidneys, and the liver, the head, the chest, the legs, and the arms. In particular, in the various embodiments and modifications described above, a medical image related to the subject eye is used for the estimation process. In this regard, the subject in the medical image used in the estimation process in the various embodiments and modified examples described above is not limited to the subject's eye, but may be any subject having symmetry in the left-right direction, the up-down direction, or the left-right and up-down direction, and may be, for example, another organ such as the lung. However, the subject in the various embodiments and modified examples described above does not need to be limited to a subject having symmetry. In addition, when the subject is an organ such as the lung, the imaging device may have a configuration such as an endoscope.
[0207] The medical image may be a tomographic image of the subject or a front image. The front image includes, for example, an SLO image of the fundus or anterior part of the eye, a fundus image photographed by fluorescence, and an En-Face image generated by using data acquired by OCT (three-dimensional OCT data) of at least a partial range in the depth direction of the subject. The En-Face image may be an OCTA En-Face image (motion contrast front image) generated by using three-dimensional OCTA data (three-dimensional motion contrast data) of at least a partial range in the depth direction of the subject. The three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.
[0208] Here, the motion contrast data is data showing a change between a plurality of volume data obtained by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, by obtaining data showing a change between a plurality of tomographic images obtained at approximately the same position at each of the different positions, the motion contrast data can be obtained as volume data. The motion contrast front image is also called an OCTA front image (OCTA En-Face image) related to OCT angiography (OCTA) that measures the movement of blood flow, and the motion contrast data is also called OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value) between two tomographic images or the interference signals corresponding thereto, and may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to scan the same region (same position) of the subject's eye a plurality of times. In addition, when controlling the scanning means so that the measurement light scans substantially the same position multiple times, the time interval between one scan (one B-scan) and the next scan (next B-scan) may be changed (determined). This allows, for example, the blood vessel region to be visualized with high accuracy even if the blood flow velocity varies depending on the state of the blood vessel. At this time, for example, the time interval may be changed according to an instruction from the examiner. Also, for example, according to an instruction from the examiner, any one of a plurality of motion contrast images corresponding to a plurality of time intervals set in advance may be selected. Also, for example, the time interval when the motion contrast data was acquired may be associated with the motion contrast data and stored in the storage unit 114. Also, for example, the display control unit 121 may cause the display unit 116 to display the time interval when the motion contrast data was acquired and the motion contrast image corresponding to the motion contrast data.Also, for example, the time interval may be automatically determined, or at least one candidate for the time interval may be determined. In this case, for example, a machine learning model may be used to determine (output) the time interval from the motion contrast image. Such a machine learning model can be obtained by learning learning data in which a plurality of motion contrast images corresponding to a plurality of time intervals are input data, and the difference between the plurality of time intervals and the time interval when a desired motion contrast image is obtained is used as correct answer data.
[0209] Moreover, the En-Face image is, for example, a front image generated by projecting data in a range between two layer boundaries in the XY direction. At this time, the front image is generated by projecting or integrating data corresponding to a depth range determined based on two reference planes, which is at least a part of the depth range of volume data (three-dimensional tomographic image) obtained using optical interference, onto a two-dimensional plane. The En-Face image is a front image generated by projecting data corresponding to a depth range determined based on a detected retinal layer, out of the volume data, onto a two-dimensional plane. Note that, as a method for projecting data corresponding to a depth range determined based on two reference planes onto a two-dimensional plane, for example, a method of setting a representative value of data within the depth range as a pixel value on the two-dimensional plane can be used. Here, the representative value can include values such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of the region surrounded by the two reference planes. Furthermore, the depth range related to the En-Face image may be, for example, a range including a predetermined number of pixels in a deeper or shallower direction based on one of two layer boundaries related to the detected retinal layer. In addition, the depth range for the En-Face image may be, for example, a range that is changed (offset) from the range between two layer boundaries for the detected retinal layers in response to an operator's instruction.
[0210] Moreover, the imaging device is a device for capturing an image used in diagnosis. The imaging device includes, for example, a device for obtaining an image of a predetermined part of a subject by irradiating the predetermined part with light, radiation such as X-rays, electromagnetic waves, or ultrasound, or a device for obtaining an image of the predetermined part by detecting radiation emitted from a subject. More specifically, the imaging device according to the above-mentioned various embodiments and modifications includes at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, an endoscope, and the like. Note that the configuration according to each of the above-mentioned embodiments and modifications can be applied to these imaging devices. In this case, the movement of the subject corresponding to the movement of the subject's eye to be predicted may be, for example, a face or body movement, a heart movement (heartbeat), or the like.
[0211] The OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The OCT device may include a line-OCT device (or an SS-line-OCT device) using line light. The OCT device may include a full field-OCT device (or an SS-full field-OCT device) using area light. The OCT device may include a Doppler-OCT device. The SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device or a wavefront compensation OCT (AO-OCT) device using wavefront adaptive optics. The SLO device or OCT device may include a polarized SLO (PS-SLO) device or a polarized OCT (PS-OCT) device for visualizing information related to polarization phase difference and depolarization. The SLO device and the OCT device may include a pathology microscope SLO device and a pathology microscope OCT device. The SLO device and the OCT device may include a handheld SLO device and a handheld OCT device. The SLO device and the OCT device may include a catheter SLO device and a catheter OCT device. The SLO device and the OCT device may include a head-mounted SLO device and a head-mounted OCT device. The SLO device and the OCT device may include a binocular SLO device and a binocular OCT device. The SLO device and the OCT device may be configured to change the imaging angle of view by an optically variable magnification. The SLO device may be configured to receive light in a time-division manner using each of the RGB light sources, or to receive light simultaneously using a plurality of light receiving elements, to obtain a color image or a fluorescent image.
[0212] In the above-described embodiment and modified examples, the control unit 117 may be configured as a part of the OCT device, or the control unit 117 may be configured as a separate unit from the OCT device. In this case, the control unit 117 may be connected to an imaging device such as the OCT device via the Internet or the like. The configuration of the OCT device is not limited to the above-described configuration, and some of the components included in the OCT device, such as an SLO imaging unit, may be configured as a separate unit from the OCT device.
[0213] In addition, in the trained models for voice recognition, character recognition, gesture recognition, etc. according to the above-mentioned modified examples, since training is performed using time-series data, it is considered that the slope between the input consecutive time-series data values is extracted as part of the feature and used in the estimation process. It is expected that such trained models can perform accurate estimation by using the influence of the time change of a specific numerical value in the estimation process. In addition, in the trained models for estimation process, image quality improvement, segmentation process, image analysis, and diagnosis result generation according to the above-mentioned embodiment and modified examples, it is considered that the magnitude of the luminance value of the tomographic image, the order and slope of the bright and dark areas, the position, distribution, continuity, etc. are extracted as part of the feature and used in the estimation process.
[0214] [Other embodiments] Furthermore, the technology disclosed in this specification can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the technology may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.
[0215] Needless to say, the object of the technology disclosed in this specification can be achieved by the following: That is, a recording medium (or storage medium) on which is recorded a program code (computer program) of software that realizes the functions of the above-mentioned embodiments is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. Then, a computer (or a CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the technology disclosed in this specification.
[0216] The technology disclosed in this specification can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments and modifications to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.
[0217] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA), and the processor or circuitry may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).
Claims
1. an estimation means for estimating a risk of a subject developing a disease by using a trained model that has learned a relationship between a feature amount acquired from a fundus image and a risk of developing a disease evaluated from the feature amount; a correction means for correcting the estimated risk of developing a disease by using a weighting coefficient determined for each biological information of the subject; An information processing device comprising:
2. an estimation means for estimating a risk of a subject developing a disease by using a trained model that has learned a relationship between a feature amount acquired from a fundus image and a risk of developing a disease evaluated from the feature amount; and a correction means for correcting the estimated risk of developing a disease based on biological information of the subject, An information processing device in which the instructions from the examiner regarding the estimation of the risk of developing the disease are information obtained using at least one of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition.
3. The information processing device described in claim 1 or 2, wherein the estimation means estimates the risk of the subject developing the first disease and the second disease using a trained model that has learned the relationship between features obtained from the fundus image and the risk of developing a first disease evaluated from the features, and the relationship between features obtained from the fundus image and the risk of developing a second disease evaluated from the features.
4. The information processing device described in claim 1 or 2, wherein the estimation means estimates the risk of the subject developing the first disease and the second disease using a first trained model that has learned the relationship between features obtained from the fundus image and the risk of developing a first disease evaluated from the features, and a second trained model that has learned the relationship between features obtained from the fundus image and the risk of developing a second disease evaluated from the features.
5. The apparatus further includes an acquisition means for acquiring a fundus image of the subject, The information processing device according to any one of claims 1 to 3, wherein the estimation means estimates a probability as a risk that the subject will develop a disease by inputting features obtained from a fundus image of the subject into the trained model.
6. The trained model learns a relationship between a vascular morphology obtained from the fundus image and a risk of developing cardiovascular disease evaluated from the vascular morphology, 6. The information processing device according to claim 1, wherein the vascular morphology includes at least one of features indicating an artery diameter, a vein diameter, a ratio of an artery diameter to a vein diameter, a branching angle of the blood vessel, asymmetry of the branching, arterial-venous stenosis, or blood vessel tortuosity.
7. an estimation means for estimating a risk of a subject developing a disease using a trained model that has learned a relationship between a vascular morphology obtained from a fundus image and a risk of developing a cardiovascular disease evaluated from the vascular morphology; and a correction means for correcting the estimated risk of developing a disease based on biological information of the subject, The vascular morphology includes at least one of features indicating artery diameter, vein diameter, ratio of artery diameter to vein diameter, branching angle of blood vessels, asymmetry of the branching, arterial-venous stenosis, or blood vessel tortuosity.
8. An acquisition means for acquiring a fundus image obtained by capturing an image of a fundus of a subject; an estimation means for estimating a risk of the subject developing a disease by inputting the acquired fundus image into a trained model that has undergone deep learning of a relationship between a fundus image and a risk of developing a disease evaluated from the fundus image; a correction means for correcting the estimated risk of developing a disease by using a weighting coefficient determined for each biological information of the subject; An information processing device comprising:
9. An acquisition means for acquiring a fundus image obtained by capturing an image of a fundus of a subject; an estimation means for estimating a risk of the subject developing a disease by inputting the acquired fundus image into a trained model that has undergone deep learning of a relationship between a fundus image and a risk of developing a disease evaluated from the fundus image; and a correction means for correcting the estimated risk of developing a disease based on biological information of the subject, An information processing device in which the instructions from the examiner regarding the estimation of the risk of developing the disease are information obtained using at least one of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition.
10. An information processing device described in any one of claims 1 to 9, wherein the correction means performs correction using at least one of the subject's biometric information including blood pressure, BMI index, age, gender, medical history, or presence or absence of smoking habit.
11. The information processing device according to claim 1 , further comprising a display control unit that causes a display unit to display the corrected risk of developing a disease.
12. The information processing device according to claim 11 , wherein the display control means causes the display unit to display the corrected risk of developing a disease in a state where the risk is classified into a plurality of classes or in a state where the risk is displayed in parallel with a fundus image of the subject.
13. The information processing device according to claim 11 , wherein the display control means causes the display unit to display a graph based on the probability corresponding to the corrected risk of developing a disease.
14. The information processing apparatus according to claim 11 , wherein the display control means causes the display unit to display the fundus image of the subject in a state in which a site highly correlated with the disease is emphasized.
15. The information processing device according to claim 11 , wherein the display control means causes the display unit to display medical institutions recommended according to the corrected disease development risk and the type of the disease.
16. The information processing device of claim 15, configured to, when a reservation at the recommended medical institution is completed, transmit the fundus image used to estimate the risk of developing the disease, the biometric information used to correct the estimated risk of developing the disease, and the estimated risk of developing the disease to the recommended medical institution via the system of the recommended medical institution.
17. 17. The information processing device according to claim 15 or 16, configured to enable a user to receive a diagnosis or consultation from a doctor at the recommended medical institution via a system capable of video communication, depending on the corrected risk of developing a disease and the type of the disease.
18. an estimation means for estimating a risk of a subject developing a disease using a trained model that has learned a relationship between a feature amount acquired from a fundus image, biological information acquired by an examination device, and the risk of developing a disease; and a display control means for causing a display unit to display the estimated risk of developing a disease, An information processing device in which the instructions from the examiner regarding the estimation of the risk of developing the disease are information obtained using at least one of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition.
19. The information processing device according to any one of claims 11 to 18, wherein the display control means causes the display unit to display at least one of: an analysis result generated using a trained model for generating analysis results obtained by learning a fundus image, the analysis result relating to the fundus image used to estimate the risk of developing the disease; a diagnosis result generated using a trained model for generating diagnosis results obtained by learning a fundus image, the diagnosis result relating to the fundus image used to estimate the risk of developing the disease; a similar case image searched using a trained model for searching for similar case images obtained by learning a fundus image, the similar case image relating to the fundus image used to estimate the risk of developing the disease; and an object detection result or segmentation result generated using a trained model for object recognition or a trained model for segmentation obtained by learning a fundus image, the object detection result or segmentation result relating to the fundus image used to estimate the risk of developing the disease.
20. An information processing device as claimed in any one of claims 11 to 19, wherein the display control means causes the display unit to display information regarding the difference between an image generated using a generative adversarial network or autoencoder to which a fundus image is input and the fundus image input to the generative adversarial network or autoencoder as information regarding an abnormal area.
21. an ophthalmologic apparatus for photographing an image of a fundus of a subject; A testing device for testing the subject and acquiring biological information; An information processing device according to any one of claims 1 to 20; An information processing system comprising:
22. an estimation step of estimating a risk of a subject developing a disease using a trained model that has learned a relationship between a feature amount acquired from a fundus image and a risk of developing a disease evaluated from the feature amount; a correction step of correcting the estimated risk of developing a disease by using a weighting coefficient determined for each biological information of the subject; An information processing method comprising:
23. an estimation step of estimating a risk of a subject developing a disease using a trained model that has learned a relationship between a vascular morphology obtained from a fundus image and a risk of developing a cardiovascular disease evaluated from the vascular morphology; and a correction step of correcting the estimated risk of developing a disease based on biological information of the subject, An information processing method in which the vascular morphology includes at least one of features indicating artery diameter, vein diameter, ratio of artery diameter to vein diameter, branching angle of blood vessels, asymmetry of the branching, arterial-venous stenosis, or blood vessel tortuosity.
24. An acquisition step of acquiring a fundus image of a fundus of a subject; an estimation step of estimating a risk of the subject developing a disease by inputting the acquired fundus image into a trained model that has learned a relationship between a fundus image and a risk of developing a disease evaluated from the fundus image; a correction step of correcting the estimated risk of developing a disease by using a weighting coefficient determined for each biological information of the subject; An information processing method comprising:
25. an estimation step of estimating a risk of a subject developing a disease using a trained model that has learned the relationship between the feature amount acquired from the fundus image, the biological information acquired by the inspection device, and the risk of developing the disease; A display control step of displaying the estimated disease development risk on a display unit, An information processing method in which the instructions from the examiner regarding the estimation of the risk of developing the disease are information obtained using at least one of a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition.
26. A program causing a computer to execute the information processing method according to any one of claims 22 to 25.
Citation Information
Patent Citations
Disease symptoms prediction server, disease symptoms predicting system, disease symptoms prediction method, and program
JP2005063456A
Ophthalmologic examination apparatus and ophthalmologic examination system
JP2017000386A
System and method for predicting the vulnerability of coronary artery plaques from image data of a patient's inherent anatomical structure
JP2017503561A
Image retrieval device, method and program
JP2019082881A
Imaging system with machine learning guide
JP2019118814A
Cited By
Automated disease detection using retinal images
US20250166829A1
