Diagnosis assistance device, learning device, diagnosis assistance method, learning method, and program

Through the learning relationship between the learning model based on fundus images and fluorescent fundus contrast images, the risk and geographical limitations of fluorescent fundus contrast examination are solved, early circulating abnormality detection of diseases such as diabetic retinopathy is realized, and diagnostic efficiency and safety are improved .

CN118975776APending Publication Date: 2024-11-19NIKON CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411101768.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-01-19
Filing Date
2019-01-18
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Due to its risk, invasiveness and geographical limitations, existing fluorescent fundogram examinations have delayed the early diagnosis and treatment of ophthalmic ischemic diseases such as diabetic retinopathy, and existing non-invasive detection equipment cannot effectively grasp the dynamics of retinal circulation.

Method used

Using a learning model based on fundus images, the retinal circulation abnormality area is determined through the relationship learning with fluorescent fundus contrast images, and early cyclic abnormality detection without fluorescent fundus contrast examination is achieved.

Benefits of technology

The circulatory abnormalities can be easily predicted without fluorescent fundogram, which improves the early detection efficiency of diseases such as diabetic retinopathy and reduces the burden on patients and medical practitioners.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975776A_ABST
    Figure CN118975776A_ABST
Patent Text Reader

Abstract

Provided are a diagnosis assistance device, a learning device, a diagnosis assistance method, a learning method, and a program, the diagnosis assistance device comprising: a specifying unit that specifies a blood circulation abnormality region in an eye fundus image, which is an image of the eye fundus, using a learning model, a learning model obtained by learning the relationship between the fundus image and a blood circulation abnormality region in the fundus image on the basis of the fundus image and the blood circulation abnormality region determined from a fluorescent fundus contrast image of the fundus; and an output unit that outputs information indicating a blood circulation abnormality region in the eye fundus image of the patient identified by the identification unit using the eye fundus image of the patient and the learning model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This invention application is a divisional application of the invention application with an international application date of January 18, 2019, an international application number of PCT / JP2019 / 001470, a national application number of 2019800183677 entering the Chinese national phase, and an invention name of “Diagnostic Assistance Device, Learning Device, Diagnostic Assistance Method, Learning Method and Program”. Technical Field

[0002] The present invention relates to a diagnosis assisting device, a learning device, a diagnosis assisting method, a learning method and a program.

[0003] This application claims priority based on Japanese Patent Application No. 2018-7585 filed in Japan on January 19, 2018, and the contents are incorporated herein by reference. Background Art

[0004] Diabetic retinopathy is an important disease that ranks second in the cause of blindness in Japan. The subjective symptoms appear after the disease has progressed for a considerable period of time, so early detection and early treatment in physical examinations are important. To address this issue, a fundus image analysis system that emphasizes capillary hemangiomas, which are the initial changes of diabetic retinopathy, and an image analysis system that screens for diabetic retinopathy from fundus images have been proposed (see Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2010-178802

[0008] Patent document 2: US2014 / 0314288A1 Summary of the invention

[0009] The technology proposed in patent document 1 and patent document 2 is aimed at the discovery and screening of diseases as described above. On the other hand, in the case of further diagnosis and treatment of patients with diabetic retinopathy from the disease screening of diabetic retinopathy, it is known that diabetic retinopathy produces vascular disorders accompanied by high blood sugar and falls into an ischemic state, and the disease stage continues to progress, so it is most important to grasp the dynamics of retinal circulation. In order to grasp the dynamics of retinal circulation, a retinal angiography examination called fluorescein fundus angiography must be performed. However, retinal angiography is a heavy burden for both patients and medical personnel due to its danger, invasiveness, geographical constraints limited to large hospitals, etc., and sometimes the shooting and repeated shooting of patients with good vision and low physical function will produce hesitation. Therefore, there are many cases of delayed onset and delayed treatment. The same situation applies to all ocular ischemic diseases represented by diabetic retinopathy. In addition, OCT angiography is a device that can detect retinal circulation abnormalities non-invasively. However, unlike angiography, it cannot even grasp the dynamics of blood flow, and its viewing angle is narrow. Therefore, although it is effective for local diseases such as macular diseases, it cannot grasp the overall image of the retina. Therefore, it is not suitable for grasping the dynamics of retinal circulation in ocular ischemic diseases such as diabetic retinopathy.

[0010] The present invention has been made in view of such circumstances, and an object of the present invention is to identify circulatory abnormalities early and easily from fundus images without performing fluorescein fundus angiography.

[0011] (1) One aspect of the present invention is a diagnosis support device having:

[0012] a determination unit that determines an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model obtained by learning a relationship between the fundus image and the abnormal blood circulation region in the fundus image based on the fundus image and the abnormal blood circulation region determined based on a fluorescein fundus angiography image of the fundus; and

[0013] An output unit outputs information indicating a blood circulation abnormality region in the fundus image of the patient determined by the determination unit using the fundus image of the patient and the learning model.

[0014] (2) One aspect of the present invention is a diagnosis support device according to (1) above, wherein:

[0015] The abnormal blood circulation area is generated based on the fluorescein fundus angiography image and an ophthalmologist's diagnosis annotation related to one or both of the retinal non-perfusion area and the neovascularization, which is added to the fluorescein fundus angiography image.

[0016] (3) One aspect of the present invention is a diagnosis support device according to (1) or (2), wherein:

[0017] The specifying unit specifies one or both of a retinal non-perfusion area and an area corresponding to a neovascularization in the fundus image.

[0018] (4) One aspect of the present invention is a diagnosis support device according to any one of (1) to (3), wherein:

[0019] The output unit outputs an image in which the abnormal blood circulation region specified by the specifying unit is superimposed on the fundus image.

[0020] (5) One form of the present invention is a learning device comprising a learning unit that generates, through learning, a learning model representing the relationship between a fundus image and an abnormal blood circulation area in the fundus image based on a fundus image that is an image of the fundus and an abnormal blood circulation area determined based on a fluorescent fundus angiography image of the fundus.

[0021] (6) One aspect of the present invention is a learning device according to (5) above, wherein:

[0022] The abnormal blood circulation area is generated based on the fluorescein fundus angiography image and an ophthalmologist's diagnosis annotation related to one or both of the retinal non-perfusion area and the neovascularization, which is added to the fluorescein fundus angiography image.

[0023] (7) One aspect of the present invention is a diagnosis supporting method, in which:

[0024] determining an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model, the learning model being obtained by learning a relationship between the fundus image and the abnormal blood circulation region determined based on the fundus image and the fundus fluorescence angiography image of the fundus;

[0025] Information indicating an abnormal blood circulation region in the fundus image of the patient determined using the fundus image of the patient and the learning model is output.

[0026] (8) One aspect of the present invention is a learning method, in which:

[0027] acquiring information indicating a fundus image as an image of the fundus and information indicating a blood circulation abnormality region determined based on the fundus fluorescence angiography image of the fundus;

[0028] Based on the acquired fundus image and the abnormal blood circulation region specified based on the fluorescent fundus angiography image of the fundus, a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image is generated through learning.

[0029] (9) One aspect of the present invention is a program, wherein:

[0030] The program causes a computer of a diagnosis assisting device to determine an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model obtained by learning a relationship between the fundus image and the abnormal blood circulation region determined based on a fundus fluorescence angiography image of the fundus; and

[0031] The program causes a computer of a diagnosis supporting apparatus to output information indicating a blood circulation abnormality region in the fundus image of the patient, which is identified using the fundus image of the patient and the learning model.

[0032] (10) One aspect of the present invention is a program, wherein:

[0033] The program causes a computer of a learning device to acquire information representing a fundus image as an image of the fundus and information representing a blood circulation abnormality region determined based on the fundus fluorescein angiography image of the fundus; and

[0034] The program causes the computer of the learning device to generate a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image through learning based on the acquired fundus image and the abnormal blood circulation area determined based on the fluorescent fundus angiography image of the fundus.

[0035] It should be noted that, in the present invention, "fundus image" refers to an image obtained by photographing the fundus of a patient using a fundus camera in order to diagnose ophthalmic diseases, and "fluorescent fundus angiography image" refers to an image obtained by continuously photographing the fundus and capturing an angiography of the circulatory state of the fundus while injecting fluorescent dye from a vein in the patient's arm using excitation light and filters corresponding to the fluorescent dye.

[0036] Effects of the Invention

[0037] According to the present invention, the presence of circulatory abnormalities can be easily predicted from fundus images without performing fluorescein fundus angiography. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a configuration diagram of an information processing system including a server as one embodiment of the information processing device of the present invention.

[0039] Figure 2 Yes means Figure 1 A block diagram of the hardware configuration of a server in an information processing system.

[0040] Figure 3 It is used to implement Figure 2 A functional block diagram showing an example of the functional configuration of various processes including NPA·NV existence probability map generation processing, incidental discovery existence probability map generation processing, estimated NPA·NV determination processing, and estimated NPA·NV display processing in the functional configuration of a server.

[0041] Figure 4 Yes Description Figure 3 A flowchart of the flow of various processes, including NPA·NV existence probability map generation processing, incidental discovery existence probability map generation processing, estimated NPA·NV determination processing, and estimated NPA·NV display processing, executed by the server.

[0042] Figure 5 It means in Figure 3 A diagram of the flow of various information used in the processing performed by a server.

[0043] Figure 6 It means in Figure 3 FIG. 1 is a diagram showing an example of fundus image information acquired during processing performed by a server.

[0044] Figure 7 It means in Figure 3 A diagram showing an example of fluorescein fundus angiography image information acquired during processing performed by a server.

[0045] Figure 8 It means in Figure 3 A diagram showing an example of an NPA·NV existence probability map output in the NPA·NV existence probability map generation process performed by the server.

[0046] Fig. 9 It means in Figure 3 A diagram showing an example of an estimated NPA and an estimated NV determined in the estimated NPA·NV determination process performed by the server.

[0047] Fig.10 It is a diagram showing the configuration of an information processing system according to a modified example of one embodiment of the present invention.

[0048] Fig.11 This is a block diagram showing an example of a learning device according to a modified example of one embodiment of the present invention.

[0049] Fig.12 is a diagram showing an example of a fundus image.

[0050] Fig.13 This is a diagram showing an example of a fluorescent fundus angiography image.

[0051] Fig.14 This is a diagram showing an example of an abnormal blood circulation area.

[0052] Fig.15 This is a diagram showing an example of the structure of a neural network.

[0053] Fig.16 This is a block diagram showing an example of a diagnosis support device according to a modified example of one embodiment of the present invention.

[0054] Fig.17 This is a flowchart showing an example of the operation of the learning device according to the modification example of one embodiment of the present invention.

[0055] Fig.18 This is a flowchart showing an example of the operation of the diagnosis support device according to the modified example of the embodiment of the present invention. DETAILED DESCRIPTION

[0056] Hereinafter, embodiments of the present invention will be described using the drawings.

[0057] Figure 1 It is a block diagram of an information processing system including a server 1 which is one embodiment of the information processing device of the present invention.

[0058] Figure 1 The information processing system shown is configured to include a server 1 , an ophthalmologist terminal 2 , and an examination device 3 .

[0059] The server 1 , the ophthalmologist terminal 2 , and the testing device 3 are connected to each other via a network N such as the Internet.

[0060] Server 1 is the management Figure 1 The server of the information processing system shown in FIG. 1 performs various processes such as NPA·NV existence probability map generation processing, incidental discovery existence probability map generation processing, and estimated NPA·NV determination processing. In addition, for the specific content of the processing performed by the server 1, refer to Figure 3 And then discuss.

[0061] The “NPA·NV existence probability map generation process” refers to a series of processes from generation of NPA·NV training information to generation of an NPA·NV existence probability map, among the processes executed by the server 1 .

[0062] "NPA·NV training information" refers to training information for calculating the probability of existence of a non-perfusion area (NPA / non-perfusion area) (hereinafter referred to as "NPA") and the probability of existence of a neovascularization (NV / neovascularization) (hereinafter referred to as "NV existence probability") in fundus image information of a patient. Specifically, "NPA·NV training information" is generated based on the fundus fluorescence angiography image information and the NPA·NV annotation information attached to the information.

[0063] "Area of ​​retinal nonperfusion" refers to an area of ​​poor circulation in the retina resulting from retinal vascular occlusion in ischemic eye disease.

[0064] The “NPA·NV existence probability map” is image information that displays the NPA existence probability and the NV existence probability in the fundus image information.

[0065] “Fundus image information” refers to image information based on a fundus image.

[0066] “Fluorescent fundus angiography image information” refers to information based on a fluorescent fundus angiography image.

[0067] The “NPA·NV annotation information” refers to the diagnostic annotation of the ophthalmologist D related to at least one of the retinal non-perfused area (NPA) and the neovascularization (NV) added to the fluorescein fundus angiography image information.

[0068] Neovascularization is an area of ​​poor retinal circulation and ischemia in the retinal nonperfusion area. If bleeding occurs from neovascularization and retinal detachment is caused by proliferative membrane formation, it will eventually lead to blindness. Therefore, it is very important to identify the retinal nonperfusion area, neovascularization and other areas of poor retinal circulation in diagnosis and treatment.

[0069] For the specific processing flow of the NPA·NV existence probability map generation process, refer to Figure 4 The flowchart is then discussed.

[0070] The “incidental discovery existence probability map generation process” refers to a series of processes from generation of incidental discovery training information to generation of an incidental discovery existence probability map, among the processes executed by the server 1 .

[0071] "Incidental finding training information" refers to training information when calculating the probability of the presence of incidental findings in the fundus image information of the patient. Specifically, "incidental finding training information" refers to training information generated based on the fluorescein fundus angiography image information and the fundus image information, and the incidental finding annotation information attached to these image information.

[0072] "Incidental finding annotation information" refers to diagnostic annotations in which ophthalmologist D makes an incidental judgment of "abnormal" on the fluorescein fundus angiography image information and the fundus image information, excluding diagnostic annotations related to retinal non-perfused areas (NPA) or neovascularization (NV). For example, information such as capillary hemangioma, fundus hemorrhage, hard leukoplakia, soft leukoplakia, venous abnormality, intraretinal fine blood vessel abnormality, vitreous hemorrhage, proliferative membrane, and retinal detachment are all examples of "incidental finding annotation information".

[0073] The “incidental finding existence probability map” is image information (not shown) that identifies and displays the existence probability of incidental findings in the fundus image information.

[0074] For the specific processing flow of incidental discovery existence probability map generation processing, refer to Figure 4 The flowchart is then discussed.

[0075] "Estimated NPA·NV determination processing" refers to a series of processing performed by server 1 until an area in the fundus image information estimated to be equivalent to a retinal non-perfused area (NPA) is determined as an estimated NPA based on the probability of NPA existence, and an area in the fundus image information estimated to be equivalent to a neovascularization (NV) is determined as an estimated NV based on the probability of NV existence.

[0076] For details on the specific process of determining the presumed NPA and NV, refer to Figure 4 The flowchart is then discussed.

[0077] The ophthalmologist terminal 2 is an information processing device operated by the ophthalmologist D, and is composed of, for example, a personal computer. The ophthalmologist terminal 2 transmits NPA·NV annotation information and incidental finding annotation information to the server 1, acquires information related to the estimated NPA·NV determined by the server 1, and the like. The various information acquired by the ophthalmologist terminal 2 is output from the ophthalmologist terminal 2 and used for the diagnosis by the ophthalmologist D.

[0078] The inspection device 3 is composed of various devices used in the eye inspection of the patient. The inspection device 3 transmits fundus image information obtained by photographing in fundus inspection and fluorescein fundus angiography image information obtained by photographing in fluorescein fundus angiography to the server 1 .

[0079] Figure 2 Yes means Figure 1 A block diagram of the hardware configuration of server 1 in the information processing system.

[0080] The server 1 includes a CPU (Central Processing Unit) 11 , a ROM (Read Only Memory) 12 , a RAM (Random Access Memory) 13 , a bus 14 , an input / output interface 15 , an output unit 16 , an input unit 17 , a storage unit 18 , a communication unit 19 , and a drive 20 .

[0081] The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 18 to the RAM 13 .

[0082] The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes as appropriate.

[0083] The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

[0084] The output unit 16 is composed of various liquid crystal displays and the like, and outputs various information.

[0085] The input unit 17 is composed of various hardware and the like, and inputs various information.

[0086] The storage unit 18 is composed of a DRAM (Dynamic Random Access Memory) or the like, and stores various data.

[0087] The communication unit 19 controls communication with other devices via a network N including the Internet.

[0088] The drive 20 is provided as needed. A removable medium 30 including a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately installed in the drive 20. A program read from the removable medium 30 by the drive 20 is installed in the storage unit 18 as needed. In addition, the removable medium 30 can also store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0089] Next, refer to Figure 3 The functional configuration of the server 1 having such a hardware configuration will be described.

[0090] Figure 3 Yes means Figure 1 information processing system Figure 2 A functional block diagram of an example of the functional structure of the server 1, which is used to implement NPA·NV existence probability map generation processing, incidental discovery existence probability map generation processing, estimated NPA·NV determination processing and estimated NPA·NV display processing.

[0091] like Figure 3 As shown, in the CPU 11 ( Figure 2 ), when the NPA·NV existence probability map generation process is executed, the image acquisition unit 101, the annotation acquisition unit 102, the training information generation unit 103 and the calculation unit 104 function.

[0092] When the incidental discovery existence probability map generation process is executed, the image acquisition unit 101 , the annotation acquisition unit 102 , the training information generation unit 103 , and the calculation unit 104 function.

[0093] When the estimated NPA·NV specifying process is executed, the estimated NPA·NV specifying unit 106 functions.

[0094] When the estimated NPA·NV display process is executed, the estimated NPA·NV display control unit 107 functions.

[0095] In the storage unit 18 ( Figure 2 ) is provided in one area of ​​the image DB 401, the annotation DB 402, and the training DB 403. In addition, the storage unit 18 may be arranged in the ophthalmologist terminal 2 instead of the server 1.

[0096] The image acquisition unit 101 acquires the patient's fluorescein fundus angiography image information and the patient's fundus image information. The fluorescein fundus angiography image information and the fundus image information acquired by the image acquisition unit 101 are respectively stored in the image DB 401 and managed. Specifically, in the fluorescein fundus angiography examination of the patient, if the patient's fluorescein fundus angiography image is photographed by the examination device 3, the fluorescein fundus angiography image information based on the fluorescein fundus angiography image is sent to the server 1. In addition, in the fundus examination of the patient, if the patient's fundus image is photographed by the examination device 3, the fundus image information based on the fundus image is sent to the server 1. The image acquisition unit 101 of the server 1 acquires the fluorescein fundus angiography image information and the fundus image information sent from the examination device 3 to the server 1, and stores these image information in the image DB 401.

[0097] Thereby, the server 1 can manage the patient's fluorescein fundus angiography image information and fundus image information accurately without omission.

[0098] The annotation acquisition unit 102 acquires the diagnostic annotation of the ophthalmologist D related to at least one of the retinal non-perfused area (NPA) and the neovascularization (NV) attached to the patient's fluorescein fundus angiography image information as NPA·NV annotation information. Specifically, in the fluorescein fundus angiography examination, if the diagnostic annotation of the ophthalmologist D related to the retinal non-perfused area (NPA) and the neovascularization (NV) is attached to the fluorescein fundus angiography image information, the ophthalmologist terminal 2 sends the diagnostic annotation as NPA·NV annotation information to the server 1 based on the operation of the ophthalmologist D. The annotation acquisition unit 102 of the server 1 acquires the NPA·NV annotation information sent from the ophthalmologist terminal 2 and stores the information in the annotation DB 402. In addition, the fluorescein fundus angiography image information and the NPA·NV annotation information attached to the image information are managed in association with each other.

[0099] Thus, the server 1 can manage the diagnosis annotation of the ophthalmologist D regarding at least one of the retinal non-perfused area (NPA) and the neovascularization (NV) added to the fluorescein fundus angiography image information without omission as NPA·NV annotation information.

[0100] In addition, the annotation acquisition unit 102 acquires the diagnostic annotation of the ophthalmologist D related to the incidental findings attached to the fluorescent fundus angiography image and the fundus image as incidental findings annotation information. Specifically, when the diagnostic annotation of the ophthalmologist D related to the incidental findings is attached to the fluorescent fundus angiography image information and the fundus image information, the ophthalmologist terminal 2 sends the diagnostic annotation as incidental findings annotation information to the server 1 based on the operation of the ophthalmologist D. The annotation acquisition unit 102 of the server 1 acquires the incidental findings annotation information sent from the ophthalmologist terminal 2 and stores the information in the annotation DB 402. In addition, the fluorescent fundus angiography image information and the fundus image information are managed in association with the incidental findings annotation information attached to these image information.

[0101] Thereby, the server 1 can manage the diagnosis annotations of the ophthalmologist D related to incidental findings added to the fluorescein fundus angiography image information and the fundus image information without omission as incidental finding annotation information.

[0102] The training information generating unit 103 generates NPA·NV training information for calculating the NPA existence probability and the NV existence probability based on the fluorescein fundus angiography image information and the NPA·NV annotation information corresponding to the information.

[0103] That is, the image DB 401 stores fluorescein fundus angiography image information obtained from a plurality of patients, and the annotation DB 402 stores NPA·NV annotation information. The training information generating unit 103 generates NPA·NV training information based on the information stored in the databases, and the NPA·NV training information becomes training information when calculating the existence probability of NPA·NV in the fundus image information.

[0104] Thereby, the server 1 can generate and store training information for calculating the NPA existence probability and the NV existence probability in the fundus image information of the patient.

[0105] Furthermore, the training information generating unit 103 generates incidental finding training information based on the fluorescein fundus angiography image information and the fundus image information, and the incidental finding annotation information corresponding to these image information.

[0106] That is, fundus image information and fluorescein fundus angiography image information obtained from a plurality of patients are stored in the image DB 401, and incidental finding annotation information is stored in the annotation DB 402. The training information generating unit 103 generates incidental finding training information based on the information stored in the database, and the incidental finding training information is training information for calculating the existence probability of incidental findings in the fundus image information.

[0107] Thereby, the server 1 can accumulate training information for calculating the existence probability of incidental findings in the fundus image information of the patient.

[0108] The operation unit 104 calculates the probability of existence of NPA and the probability of existence of NV based on at least NPA·NV training information. In addition, when the operation unit 104 generates the incidental finding probability map discussed later, it calculates the probability of existence of NPA and the probability of existence of NV based on the incidental finding probability map and the NPA·NV training information. In addition, the specific method of calculating the probability of existence of NPA and the probability of existence of NV is not particularly limited. For example, it can be that a feature common to fundus images with NPA·NV is extracted from the NPA·NV training information, and the degree of consistency with the feature is normalized to determine whether the fundus image information of the patient has the feature, thereby calculating the probability of existence of NPA and the probability of existence of NV. The probability of existence of NPA and the probability of existence of NV can also be calculated using deep learning (deep learning) technology.

[0109] This makes it possible to set a criterion for determining an area estimated to correspond to a retinal non-perfused area (NPA) and an area estimated to correspond to a neovascularization (NV) in the fundus image information.

[0110] In addition, the operation unit 104 calculates the probability of incidental findings in the fundus image information based on the incidental findings training information. The specific method for calculating the probability of incidental findings is not particularly limited. For example, it can be that a feature common to fundus images with incidental findings is extracted from the incidental findings training information, and the probability of incidental findings is calculated by normalizing the degree of consistency with respect to the feature in response to whether the fundus image information of the patient has the feature. The probability of incidental findings can also be calculated using deep learning (deep learning) technology.

[0111] This makes it possible to set a criterion for determining an area estimated to correspond to a retinal non-perfused area (NPA) and an area estimated to correspond to a neovascularization (NV) in the fundus image information.

[0112] The map generation unit 105 generates an NPA·NV existence probability map as image information that allows the NPA existence probability and the NV existence probability to be identified and displayed in the fundus image information. Specifically, Figure 8 The illustrated NPA·NV exists in image information such as the probability map E.

[0113] This makes it possible to generate information that serves as a basis for estimating the presence of a retinal non-perfused area (NPA) and a neovascularization (NV) in fundus image information.

[0114] The method of displaying the NPA and NV probabilities in the fundus image information is not particularly limited. For example, the NPA and NV probabilities may be displayed by using the difference in color or by using the shade of color.

[0115] Furthermore, the map generating unit 105 generates an incidental finding existence probability map (not shown) which is image information for identifying and displaying the incidental finding existence probability in the fundus image information.

[0116] This makes it possible to generate information that serves as a basis for estimating the presence of a retinal non-perfused area (NPA) and a neovascularization (NV) in fundus image information.

[0117] The method of displaying the incidental finding existence probability in the fundus image information is not particularly limited. For example, the incidental finding existence probability may be displayed by using a color difference or by using a color depth.

[0118] The estimated NPA·NV determination unit 106 determines the area estimated to be equivalent to the retinal non-perfusion area (NPA) in the fundus image information as the estimated NPA, and determines the area estimated to be equivalent to the neovascularization (NV) as the estimated NV based on the NPA existence probability and the NV existence probability. Specifically, the area in the fundus image information of the patient where the NPA existence probability and the NV existence probability exceed the prescribed threshold value is determined as the area estimated to be equivalent to the retinal non-perfusion area (NPA) (estimated NPA) or the area estimated to be equivalent to the neovascularization (NV) (estimated NV). In addition, the threshold value can be arbitrarily changed according to the judgment of the ophthalmologist D.

[0119] This makes it possible to easily and early identify a region where a retinal non-perfusion area (NPA) is estimated to exist (estimated NPA) and a region where a neovascularization (NV) is estimated to exist (estimated NV) in image information based on a fundus image.

[0120] The estimated NPA and NV display control unit 107 performs control to display the estimated NPA region and the estimated NV region on the fundus image information.

[0121] This allows the region where retinal non-perfusion is estimated to exist (estimated NPA) and the region where neovascularization (NV) is estimated to exist (estimated NV) to be displayed in an overlapping manner on image information based on the fundus image.

[0122] Next, refer to Figure 4 For Figure 3 The flow of a series of processes executed by the server 1 having the functional structure is described.

[0123] Figure 4 Yes Description Figure 3 A flowchart of a series of processing steps executed by server 1.

[0124] like Figure 4 As shown, in the server 1, the following series of processes are executed.

[0125] In step S1 , the image acquisition unit 101 determines whether or not fluorescein fundus angiography image information has been transmitted from the inspection device 3 .

[0126] If the fluorescent fundus angiography image information is transmitted, it is determined as yes in step S1, and the process proceeds to step S2. On the other hand, if the fluorescent fundus angiography image information is not transmitted, it is determined as no in step S1, and the process returns to step S1. That is, the determination process of step S1 is repeated until the fluorescent fundus angiography image information is transmitted. Thereafter, if the fluorescent fundus angiography image information is transmitted, it is determined as yes in step S1, and the process proceeds to step S2.

[0127] In step S2 , the image acquisition unit 101 acquires the transmitted fluorescein fundus angiography image information.

[0128] In step S3 , the annotation acquisition unit 102 determines whether NPA·NV annotation information has been transmitted from the ophthalmologist terminal 2 .

[0129] If the NPA·NV annotation information is sent, it is determined as yes in step S3, and the process proceeds to step S4. On the other hand, if the NPA·NV annotation information is not sent, it is determined as no in step S3, and the process returns to step S3. That is, the determination process of step S3 is repeated until the NPA·NV annotation information is sent. Thereafter, if the NPA·NV annotation information is sent, it is determined as yes in step S3, and the process proceeds to step S4.

[0130] In step S4 , the annotation acquisition unit 102 acquires the transmitted NPA·NV annotation information.

[0131] In step S5 , the training information generating unit 103 generates NPA·NV training information based on the fluorescein fundus angiography image information and the NPA·NV annotation information corresponding to the fluorescein fundus angiography image information.

[0132] In step S6 , the image acquisition section 101 determines whether fundus image information has been transmitted from the inspection device 3 .

[0133] If the fundus image information is sent, it is determined as yes in step S6, and the process proceeds to step S7. On the other hand, if the fundus image information is not sent, it is determined as no in step S6, and the process returns to step S6. That is, the determination process of step S6 is repeated until the fundus image information is sent. Thereafter, if the fundus image information is sent, it is determined as yes in step S6, and the process proceeds to step S7.

[0134] In step S7 , the image acquisition unit 101 acquires the transmitted fundus image information.

[0135] In step S8 , the annotation acquisition unit 102 determines whether or not the incidental finding annotation information has been transmitted from the ophthalmologist terminal 2 .

[0136] If incidental discovery annotation information is transmitted, it is determined as YES in step S8 and the process proceeds to step S9. On the other hand, if incidental discovery annotation information is not transmitted, it is determined as NO in step S8 and the process proceeds to step S12 by skipping steps S9 to S11.

[0137] In step S9 , the annotation acquisition section 102 acquires incidental discovery annotation information.

[0138] In step S10 , the training information generating unit 103 generates incidental finding training information based on the fluorescein fundus angiography image information and the NPA·NV annotation information corresponding to the fluorescein fundus angiography image information.

[0139] In step S11 , the calculation unit 104 calculates the existence probability of incidental findings in the fundus image information based on the incidental finding training information.

[0140] In step S12 , the map generating unit 105 generates an incidental finding existence probability map as image information for identifying and displaying the incidental finding existence probability in the fundus image information.

[0141] In step S13, the calculation unit 104 calculates the NPA existence probability and the NV existence probability based on at least the NPA·NV training information. In addition, when the incidental discovery existence probability map is generated, the calculation unit 104 calculates the NPA existence probability and the NV existence probability based on the NPA·NV training information and the incidental discovery existence probability map.

[0142] In step S14 , the map generating unit 105 generates an NPA·NV existence probability map as image information in which the NPA existence probability and the NV existence probability are distinguishably displayed in the fundus image information.

[0143] In step S15, the estimated NPA / NV determination unit 106 determines the area estimated to correspond to the retinal non-perfusion area (NPA) as the estimated NPA and the area estimated to correspond to the neovascularization (NV) as the estimated NV in the fundus image information based on the NPA existence probability and the NV existence probability.

[0144] In step S16 , the estimated NPA and NV display control unit 107 performs control to display the estimated NPA and NV regions on the fundus image information.

[0145] In step S17 , the server 1 determines whether there is an instruction to end the process.

[0146] If there is no instruction to end the process, it is determined as NO in step S17 and the process returns to step S1. On the other hand, if there is an instruction to end the process, it is determined as YES in step S17 and the process ends.

[0147] By executing the above-described series of processes by the server 1 , the estimated NPA region and the estimated NV region are displayed on the fundus image information.

[0148] Next, refer to Figure 5 The flow of various information used in various processes executed by the server 1 will be described.

[0149] Figure 5This is a diagram showing the flow of various information in the process executed by the server 1.

[0150] like Figure 5 As shown, in the fluorescent fundus angiography examination, if the fluorescent fundus angiography image of the patient is taken, the server 1 obtains the fluorescent fundus angiography image information based on the fluorescent fundus angiography image. The diagnostic annotation of the ophthalmologist D related to at least one of the retinal non-perfused area (NPA) and the neovascularization (NV) is attached to the fluorescent fundus angiography image information. The diagnostic annotation is the NPA·NV annotation information, and together with the fluorescent fundus angiography image information, it constitutes the NPA·NV training information.

[0151] During fundus examination, if a fundus image of a patient is taken, fundus image information based on the fundus image is acquired by server 1. The fundus image information and NPA·NV training information are used for calculation processing performed by calculation unit 104 based on the NPA·NV annotation program. As a result of the calculation processing based on the NPA·NV annotation program, an NPA·NV existence probability map is generated. Based on the NPA·NV existence probability map, the estimated NPA and estimated NV in the fundus image information are determined.

[0152] Ophthalmologist D's diagnostic annotations on incidental findings may be added to the fluorescein fundus angiography image information and the fundus image information. The diagnostic annotations in this case constitute incidental findings training information together with the fluorescein fundus angiography image information and the fundus image information as incidental findings annotation information.

[0153] The fundus image information and the NPA·NV training information are used for the calculation processing performed by the calculation unit 104 based on the incidental finding determination program. As a result of the calculation processing based on the incidental finding determination program, an incidental finding existence probability map is generated. When the incidental finding existence probability map is generated in this way, the estimated NPA and estimated NV in the fundus image information are determined based on the incidental finding existence probability map and the NPA·NV training information.

[0154] Figure 6 2 is a diagram showing an example of fundus image information acquired in the process executed by the server 1 .

[0155] Figure 7 1 is a diagram showing an example of fluorescein fundus angiography image information acquired in the process executed by the server 1 .

[0156] During the fundus examination of the patient, the examination device 3 takes a picture of the fundus of the patient to obtain Figure 6 Ophthalmologist D refers to the fundus image information shown in FIG. Figure 6 Diagnosis is performed while using fundus image information as shown. Figure 6As shown in the figure, the fundus image information only shows findings caused by perfusion abnormalities such as hemorrhage and white spots, and it is difficult to read the circulation dynamics and the abnormal circulation sites. Figure 7 Such fundus fluorescein angiography image information is shown. Figure 7 As shown in the figure, the image information of the fluorescein fundus angiography can clearly identify the dynamics of the retinal circulation. However, due to the risk and invasiveness of the examination and the geographical restrictions of being limited to large hospitals, the fluorescein fundus angiography examination is a heavy examination for both patients and medical personnel. Patients with good vision and low physical function are hesitant to take pictures and take pictures repeatedly. Therefore, there are many problems such as delayed understanding of the onset of the disease and the condition, and delayed treatment.

[0157] Therefore, the server 1 uses NPA·NV training information generated based on the accumulated fluorescein fundus angiography image information obtained from multiple patients and the NPA·NV annotation information respectively attached to the information to perform NPA·NV existence probability map generation processing. As a result, circulatory abnormalities can be easily determined from fundus images that can be easily taken in nearby clinics, physical examinations, etc. without performing a heavy fluorescein fundus angiography examination.

[0158] Figure 8 This is a diagram showing an example of an NPA·NV existence probability map output in the NPA·NV existence probability map generation process executed by the server 1 .

[0159] Figure 8 The NPA·NV existence probability map E shown is image information that enables fundus image information to be identified and displayed with the NPA existence probability and the NV existence probability. The NPA·NV existence probability map E can identify and display the NPA existence probability and the NV existence probability by using different colors and color shades. For example, it is possible to identify and display that the NPA existence probability and the NV existence probability are higher in areas represented by warm colors, and that the NPA existence probability and the NV existence probability are lower in areas represented by cool colors. In addition, even in areas represented by cool colors where the NPA existence probability and the NV existence probability are lower, it is possible to identify and display that the NPA existence probability and the NV existence probability are lower in areas with darker colors than in areas with lighter colors.

[0160] This makes it possible to generate information that serves as a basis for estimating the presence of a retinal non-perfused area (NPA) and a neovascularization (NV) in fundus image information.

[0161] Fig. 9 The figure shows an example of the estimated NPA and the estimated NV determined in the estimated NPA·NV determination process executed by the server 1 .

[0162] Based on the contents of the NPA·NV existence probability map, the server 1 determines the area in the fundus image information that is estimated to be equivalent to the retinal non-perfusion area (NPA) as the estimated NPA, and determines the area that is estimated to be equivalent to the neovascularization (NV) as the estimated NV. Fig. 9 As shown in FIG. 1 , the region A indicated by the dotted line can be set as the estimated NPA, and the region B can be set as the estimated NV. The estimated NPA and the estimated NV can be displayed in a superimposed manner on the fundus image information.

[0163] Thus, without performing a fluorescein fundus angiography examination requiring a special fundus camera and diagnostic apparatus, it is possible to easily and early identify an area where retinal non-perfusion is estimated to exist and an area where neovascularization is estimated to exist from a fundus image obtained by photographing the fundus of a patient.

[0164] As mentioned above, although one embodiment of the present invention has been described, the present invention is not limited to the above-mentioned embodiment, and modifications, improvements, etc. within the scope that can achieve the object of the present invention are included in the present invention.

[0165] For example, in the above-mentioned embodiment, if various image information is sent from the inspection device 3, the image acquisition unit 101 acquires the information, but it may be configured so that if the inspection device 3 is used to take pictures, the image acquisition unit 101 spontaneously acquires various image information. Similarly, in the above-mentioned embodiment, if various annotation information is sent from the ophthalmologist terminal 2, the annotation acquisition unit 102 acquires the information, but it may be configured so that if various annotation information is input to the ophthalmologist terminal 2, the annotation acquisition unit 102 spontaneously acquires various annotation information.

[0166] in addition, Figure 2 The hardware configurations shown are merely examples for achieving the purpose of the present invention and are not particularly limited.

[0167] in addition, Figure 3 The functional block diagram shown is only an example and is not particularly limited. That is, as long as the information processing system has the function of being able to perform the above series of processing as a whole, it is sufficient, and what kind of functional blocks are used to achieve the function is not particularly limited. Figure 3 Example.

[0168] In addition, the location of the function block is not limited to Figure 3 The location shown may be any location. For example, at least a part of the functional blocks of the server 1 may be installed in the ophthalmologist terminal 2 or the testing device 3.

[0169] Furthermore, one functional block may be constituted by hardware alone or by a combination of hardware alone and software alone.

[0170] When the processing of each functional block is executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium.

[0171] The computer may be a computer equipped with dedicated hardware. In addition, the computer may be a computer capable of executing various functions by installing various programs, such as a general-purpose smart phone or a personal computer in addition to a server.

[0172] The recording medium including such a program may be constituted not only by a removable medium distributed separately from the apparatus body in order to provide the program to each user, but also by a recording medium provided to each user in a state of being pre-installed in the apparatus body.

[0173] In the present specification, the steps describing the program recorded in the recording medium naturally include processing performed in time series according to the order, and also include processing that is not necessarily processed in time series but is executed in parallel or individually.

[0174] For example, in Figure 4 In step S6, the image acquisition unit 101 determines whether fundus image information has been sent from the inspection device, and acquires the fundus image information in step S7. However, in the case of performing the generation process of the probability map of incidental findings, it is sufficient as long as the fundus image information is properly managed at the time point of generating the incidental finding training information. In addition, in the case of not performing the generation process of the probability map of incidental findings, it is sufficient as long as the fundus image information is properly managed at the time point of calculating the NPA existence probability and the NV existence probability. Therefore, in the case of generating the incidental finding training information, the fundus image information can be stored in the image DB 401 and managed from any time point before the incidental finding training information is generated in step S10. In addition, in the case of not generating the incidental finding training information, it can be stored in the image DB 401 and managed from any time point before the NPA existence probability and the NV existence probability are calculated in step S13.

[0175] To summarize the above, it is sufficient that the program to which the present invention is applied has the following configuration, and various embodiments can be adopted.

[0176] That is, the program to which the present invention is applied causes a computer that controls an information processing device to execute the following steps, which steps include:

[0177] Fluorescent fundus angiography image acquisition steps (eg Figure 4 Step S2), obtaining fluorescent fundus angiography image information (eg Figure 7 Fluorescent fundus angiography image information C);

[0178] NPA·NV annotation acquisition steps (e.g. Figure 4 Step S4), obtaining the ophthalmologist's diagnostic annotation related to at least one of the retinal non-perfused area (NPA) and the neovascularization (NV) attached to the above-mentioned fundus fluorescein angiography image information as NPA·NV annotation information;

[0179] NPA·NV training information generation step (e.g. Figure 4 Step S5), based on the above-mentioned fluorescent fundus angiography image information and the above-mentioned NPA·NV annotation information corresponding to the fluorescent fundus angiography image information, generate NPA·NV training information, and the NPA·NV training information becomes training information for calculating the existence probability of the above-mentioned retinal non-perfused area (NPA) and the existence probability of the new blood vessels (NV);

[0180] Fundus image acquisition steps (eg Figure 4 Step S7), obtaining fundus image information (eg Figure 6 Fundus image information F);

[0181] NPA·NV has probability operation steps (e.g. Figure 4 Step S13), calculating the existence probability of the retinal non-perfusion area (NPA) and the existence probability of the neovascularization (NV) in the fundus image information based on the NPA·NV training information; and

[0182] Presumed NPA·NV determination steps (e.g. Figure 4 In step S15), based on the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of the above-mentioned neovascularization, the area in the above-mentioned fundus image information estimated to be equivalent to the above-mentioned retinal non-perfusion area (NPA) is determined as the estimated NPA, and the area estimated to be equivalent to the above-mentioned neovascularization (NV) is determined as the estimated NV.

[0183] Thus, without performing a fluorescein fundus angiography examination that requires a special fundus camera and diagnostic apparatus, it is possible to easily and early identify an area where retinal non-perfusion is estimated to exist from a fundus image obtained by photographing the fundus of a patient.

[0184] In addition, it is possible to further include a step of generating an NPA·NV existence probability map (for example Figure 4 In the NPA·NV existence probability map generation step, an NPA·NV existence probability map (e.g., Figure 8 NPA·NV existence probability diagram E).

[0185] This makes it possible to generate information that serves as a basis for estimating the presence of a retinal non-perfused area (NPA) and a neovascularization (NV) in fundus image information.

[0186] In addition, the method may further include the following steps:

[0187] With annotations to get the steps (e.g. Figure 4 Step S9), acquiring the ophthalmologist's diagnostic annotations related to incidental findings attached to the fluorescent fundus angiography image information and the fundus image information as incidental findings annotation information;

[0188] Incidentally, it is found that the training information generation step (e.g. Figure 4 Step S10), based on the fluorescent fundus angiography image information and the fundus image information, and the incidental finding annotation information corresponding to the fluorescent fundus angiography image information and the fundus image information, generating incidental finding training information, the incidental finding training information being training information for calculating the existence probability of the incidental finding in the fundus image information; and

[0189] Incidentally, it is found that there are probability calculation steps (such as Figure 4 Step S11), calculating the existence probability of the incidental findings in the fundus image information based on the incidental findings training information,

[0190] In the above-mentioned NPA·NV existence probability calculation step, the following control processing is also performed: based on the existence probability of the above-mentioned incidental discovery and the above-mentioned NPA·NV training information, the existence probability of the above-mentioned retinal non-perfusion area (NPA) and the existence probability of new blood vessels (NV) in the above-mentioned fundus image information are calculated.

[0191] In addition, it is possible to further include a step of generating a probability map of existence (eg Figure 4 In step S12), in the incidental finding existence probability map generating step, an incidental finding existence probability map is generated so that the fundus image information can be identified and displayed with the existence probability of the incidental finding.

[0192] Hereinafter, a modified example of one embodiment of the present invention will be described with reference to the drawings.

[0193] (Variation Example)

[0194] Fig.10 1 is a block diagram of an information processing system according to a modification of an embodiment of the present invention. The information processing system according to a modification of an embodiment of the present invention includes an ophthalmologist terminal 2, a testing device 3, a learning device 200, and a diagnosis support device 300. These devices are connected to each other via a network N.

[0195] The learning device 200 generates a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image by learning, based on the fundus image as an image of the fundus and the abnormal blood circulation area determined from the fundus fluorescence angiography image of the fundus. Here, the abnormal blood circulation area is an area where the blood circulation is abnormal due to blood obstruction on the retina caused by an ocular ischemic disease such as diabetic retinopathy.

[0196] The learning device 200 acquires information representing a fundus image of a patient and information representing a fluorescent fundus angiography image of the patient, and stores the acquired information representing the fundus image and the acquired information representing the fluorescent fundus angiography image in association with each other. Specifically, during fundus examination of a patient, the inspection device 3 captures a fundus image of the patient, generates fundus image notification information including a patient ID and information representing the captured fundus image, and having the learning device 200 as a destination, and sends the generated fundus image notification information to the learning device 200. In addition, during the fluorescent fundus angiography examination of a patient, the inspection device 3 captures a fluorescent fundus angiography image of the patient, generates fluorescent fundus angiography image notification information including a patient ID and information representing the captured fluorescent fundus angiography image, and having the learning device 200 as a destination, and sends the generated fluorescent fundus angiography image notification information to the learning device 200.

[0197] The learning device 200 obtains the patient ID and information representing the fundus image contained in the fundus image notification information sent from the inspection device 3 to the learning device 200, as well as the patient ID and information representing the fluorescent fundus angiography image contained in the fluorescent fundus angiography image notification information, and stores the acquired patient ID, information representing the fundus image, and information representing the fluorescent fundus angiography image in an associated manner.

[0198] The learning device 200 acquires the diagnostic annotation of the ophthalmologist D related to one or both of the non-perfused area of ​​the retina (NPA) and the neovascularization (NV) attached to the fluorescent fundus angiography image of the patient as NPA·NV annotation information. Specifically, in the fluorescent fundus angiography examination, if the diagnostic annotation of the ophthalmologist D related to the non-perfused area of ​​the retina (NPA) and the neovascularization (NV) is attached to the fluorescent fundus angiography image, the ophthalmologist terminal 2 generates NPA·NV annotation notification information including the patient ID and the diagnostic annotation and with the learning device 200 as the sending destination based on the operation of the ophthalmologist D, and sends the generated NPA·NV annotation notification information to the learning device 200.

[0199] The learning device 200 receives the NPA·NV annotation notification information sent by the ophthalmologist terminal 2 and stores the NPA·NV annotation information included in the received NPA·NV annotation notification information. In addition, information representing the fluorescein fundus angiography image and the NPA·NV annotation information added to the fluorescein fundus angiography image are stored in association with each other.

[0200] The learning device 200 acquires the diagnostic annotation of the ophthalmologist D related to the incidental findings attached to the fundus image and the fluorescent fundus angiography image as incidental findings annotation information. Specifically, for the information representing the fluorescent fundus angiography image and the information representing the fundus image, in the case where the diagnostic annotation of the ophthalmologist D related to the incidental findings is attached, the ophthalmologist terminal 2 generates incidental findings annotation notification information including the patient ID and the diagnostic annotation and with the learning device 200 as the sending destination based on the operation of the ophthalmologist D, and sends the generated incidental findings annotation notification information to the learning device 200. The learning device 200 receives the incidental findings annotation notification information sent by the ophthalmologist terminal 2, acquires the patient ID and the incidental findings annotation information contained in the received incidental findings annotation notification information, and stores the acquired patient ID and incidental findings annotation information. In addition, the information representing the fundus image and the information representing the fluorescent fundus angiography image are stored in association with the incidental findings annotation information.

[0201] The learning device 200 acquires information representing a fundus image, information representing a fluorescent fundus angiography image associated with the information representing the fundus image, NPA·NV annotation information, and incidental finding annotation information, and determines an abnormal blood circulation region based on the acquired information representing the fluorescent fundus angiography image, NPA·NV annotation information, and incidental finding annotation information. The learning device 200 generates NPA·NV learning information obtained by associating the information representing the fundus image with the abnormal blood circulation region determined based on the fluorescent fundus angiography image corresponding to the fundus image, and stores the generated NPA·NV learning information.

[0202] The learning device 200 uses the information representing the fundus image contained in the NPA·NV learning information as input information, and uses the abnormal blood circulation area determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and generates a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image through learning. In addition, the specific method of generating the learning model is not particularly limited. For example, it can be to extract the common features of the fundus image with the abnormal blood circulation area from the NPA·NV learning information, and derive the relationship between the extracted common features and the abnormal blood circulation area in the fundus image. It is also possible to use neural network, deep learning (deep learning) technology to derive the relationship between the fundus image and the abnormal blood circulation area in the fundus image. The learning device 200 stores the generated learning model, and generates learning model notification information including the generated learning model and with the diagnosis support device 300 as the sending destination, and sends the generated learning model notification information to the diagnosis support device 300.

[0203] The diagnosis support device 300 receives the learning model transmitted by the learning device 200 and stores the received learning model.

[0204] The ophthalmologist terminal 2 generates patient information including a patient ID and information indicating a fundus image of the patient and to which the diagnosis supporting apparatus 300 is sent, and transmits the generated patient information to the diagnosis supporting apparatus 300 .

[0205] The diagnosis support device 300 receives the patient information sent by the ophthalmologist terminal 2, and acquires the patient ID and the information indicating the fundus image of the patient contained in the received patient information. The diagnosis support device 300 uses the stored learning model to determine the abnormal blood circulation area in the fundus image based on the acquired information indicating the fundus image. The diagnosis support device 300 generates a diagnosis result and sends the generated diagnosis result to the ophthalmologist terminal 2, wherein the diagnosis result includes the information indicating the abnormal blood circulation area in the fundus image determined using the fundus image of the patient and the learning model, and the patient ID, and the ophthalmologist terminal 2 is used as the sending destination.

[0206] Hereinafter, the learning device 200 and the diagnosis support device 300 included in the information processing system will be described.

[0207] (Learning device 200)

[0208] Fig.11 This is a block diagram showing an example of a learning device according to a modification of the embodiment of the present invention.

[0209] The learning device 200 includes a communication unit 205, a storage unit 210, an operation unit 220, an information processing unit 230, a display unit 240, and a Fig.11 A bus 250 such as an address bus and a data bus are electrically connected as shown.

[0210] The communication unit 205 is implemented by a communication module. The communication unit 205 communicates with external communication devices such as the ophthalmologist terminal 2, the inspection device 3, and the diagnosis support device 300 via the network N. Specifically, the communication unit 205 receives the fundus image notification information sent by the inspection device 3, and outputs the received fundus image notification information to the information processing unit 230. The communication unit 205 receives the fluorescein fundus angiography image notification information sent by the inspection device 3, and outputs the received fluorescein fundus angiography image notification information to the information processing unit 230. The communication unit 205 receives the NPA·NV annotation notification information sent by the ophthalmologist terminal 2, and outputs the received NPA·NV annotation notification information to the information processing unit 230. The communication unit 205 receives the incidental finding annotation notification information sent by the ophthalmologist terminal 2, and outputs the received incidental finding annotation notification information to the information processing unit 230. The communication unit 205 acquires the learning model notification information output by the information processing unit 230 , and transmits the acquired learning model notification information to the diagnosis support device 300 .

[0211] The storage unit 210 is implemented by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or a hybrid storage device combining multiple of them. The storage unit 210 stores a program 211 executed by the information processing unit 230, an APP (Application) 212, an image DB 213, an annotation DB 214, learning information 215, and a learning model 216.

[0212] The program 211 is, for example, an operating system, which is located between users, application programs, and hardware, and provides a standard interface to users and application programs while efficiently managing various resources such as hardware.

[0213] APP212 enables the learning device 200 to receive the fundus image notification information sent by the inspection device 3, and enables the learning device 200 to store the patient ID and the information representing the fundus image contained in the received fundus image notification information in association with each other. APP212 enables the learning device 200 to receive the fluorescein fundus angiography image notification information sent by the inspection device 3, and enables the learning device 200 to store the patient ID and the information representing the fluorescein fundus angiography image contained in the received fluorescein fundus angiography image notification information in association with each other. APP212 enables the learning device 200 to receive the NPA·NV annotation notification information sent by the ophthalmologist terminal 2, and enables the learning device 200 to store the patient ID and the NPA·NV annotation information contained in the received NPA·NV annotation notification information in association with each other.

[0214] APP212 enables the learning device 200 to receive the incidental finding annotation notification information sent by the ophthalmologist terminal 2, and enables the learning device 200 to store the patient ID and the incidental finding annotation information contained in the received incidental finding annotation notification information in association with each other. APP212 enables the learning device 200 to acquire the information representing the fundus image, the information representing the fluorescein fundus angiography image, the NPA·NV annotation information, and the incidental finding annotation information associated with the patient ID. APP212 enables the learning device 200 to determine the abnormal blood circulation area based on the acquired information representing the fluorescein fundus angiography image, the NPA·NV annotation information, and the incidental finding annotation information.

[0215] APP212 causes the learning device 200 to generate NPA·NV learning information (the NPA·NV learning information is information obtained by associating information representing a fundus image with an abnormal blood circulation region determined based on a fluorescent fundus angiography image corresponding to the fundus image), and causes the learning device 200 to store the generated NPA·NV learning information. APP212 causes the learning device 200 to use the information representing the fundus image contained in the NPA·NV learning information as input information, and the abnormal blood circulation region determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and to generate a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image through learning. APP212 causes the learning device 200 to store the generated learning model, and causes the learning device 200 to send the learning model to the diagnosis support device 300.

[0216] The image DB 213 stores the patient ID, information indicating the fundus image, and information indicating the fluorescein fundus angiography image in association with each other.

[0217] The annotation DB 214 stores the patient ID, NPA·NV annotation information, and incidental finding annotation information in association with each other.

[0218] The learning information 215 stores NPA·NV learning information obtained by associating information indicating a fundus image with a blood circulation abnormality region specified based on a fluorescein fundus angiography image corresponding to the fundus image.

[0219] The learning model 216 uses the information representing the fundus image contained in the NPA·NV learning information as input information, and uses the abnormal blood circulation area determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and stores a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image.

[0220] The operation unit 220 is constituted by, for example, a touch panel, detects a touch operation performed on a screen displayed on the display unit 240 , and outputs the detection result of the touch operation to the information processing unit 230 .

[0221] The display unit 240 is composed of, for example, a touch panel, and displays a screen for accepting information representing the fundus image and information representing the fluorescein fundus angiography image received by the learning device 200. In addition, the display unit 240 displays a screen for accepting an operation of processing the NPA·NV annotation information and incidental discovery annotation information received by the learning device 200.

[0222] All or part of the information processing unit 230 is, for example, a functional unit (hereinafter referred to as a software functional unit) implemented by executing the program 211 and APP 212 stored in the storage unit 210 by a processor such as a CPU (Central Processing Unit). In addition, all or part of the information processing unit 230 may also be implemented by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), or may be implemented by a combination of software functional units and hardware. The information processing unit 230 includes, for example, an image acquisition unit 231, an annotation acquisition unit 232, a learning information generation unit 233, and a learning unit 234.

[0223] The image acquisition unit 231 acquires the fundus image notification information output by the communication unit 205 , acquires the patient ID and the information indicating the fundus image included in the acquired fundus image notification information, and stores the acquired patient ID and the information indicating the fundus image in the image DB 213 in association with each other.

[0224] The image acquisition unit 231 acquires the fluorescein fundus angiography image notification information output by the communication unit 205, and acquires the patient ID and the information indicating the fluorescein fundus angiography image included in the acquired fluorescein fundus angiography image notification information. The image acquisition unit 231 associates the acquired patient ID and the information indicating the fluorescein fundus angiography image and stores them in the image DB 213.

[0225] The annotation acquisition unit 232 acquires the NPA·NV annotation notification information output by the communication unit 205 , and acquires the patient ID and NPA·NV annotation information included in the acquired NPA·NV annotation notification information. The annotation acquisition unit 232 associates the acquired patient ID and NPA·NV annotation information and stores them in the annotation DB 214 .

[0226] The annotation acquisition unit 232 acquires the incidental finding annotation notification information output by the communication unit 205 , and acquires the patient ID and incidental finding annotation information included in the acquired incidental finding annotation notification information. The annotation acquisition unit 232 associates the acquired patient ID and incidental finding annotation information and stores them in the annotation DB 214 .

[0227] The learning information generating unit 233 acquires the patient ID, the information indicating the fundus image associated with the patient ID, and the information indicating the fluorescein fundus angiography image stored in the image DB 213 of the storage unit 210. The learning information generating unit 233 acquires the patient ID, the NPA·NV annotation information associated with the patient ID, and the incidental finding annotation information stored in the annotation DB 214 of the storage unit 210. The learning information generating unit 233 specifies the abnormal blood circulation region based on the acquired information indicating the fluorescein fundus angiography image, the NPA·NV annotation information, and the incidental finding annotation information.

[0228] Hereinafter, each configuration of the learning device 200 will be described in detail.

[0229] Fig.12 is a diagram showing an example of a fundus image. Fig.13 This is a diagram showing an example of a fluorescent fundus angiography image.

[0230] The learning information generation unit 233 extracts the green component of the fundus image and removes the noise component from the fundus image obtained by extracting the green component, that is, the green component extracted fundus image. The learning information generation unit 233 divides the green component extracted fundus image from which the noise component has been removed into rectangles. The learning information generation unit 233 adjusts the green component extracted fundus image from which the noise component has been removed to a predetermined size based on the image divided into rectangles. When adjusting to the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.

[0231] The learning information generating unit 233 extracts the green component of the fluorescent fundus angiography image, and removes the noise component from the fluorescent fundus angiography image obtained by extracting the green component, that is, the green component extracted fluorescent fundus angiography image. The learning information generating unit 233 divides the green component extracted fluorescent fundus angiography image from which the noise component has been removed into rectangles. The learning information generating unit 233 adjusts the green component extracted fluorescent fundus angiography image from which the noise component has been removed to a predetermined size based on the image divided into rectangles. When adjusting to the predetermined size, interpolation is performed by an interpolation method such as bicubic interpolation.

[0232] The learning information generating unit 233 corrects the rotation component of the green component extracted fundus image from which the noise component is removed and the green component extracted fluorescent fundus angiography image from which the noise component is removed so as to make the position of the eyeball consistent, and adjusts the image to a predetermined size. The learning information generating unit 233 determines the abnormal blood circulation area based on the green component extracted fundus image from which the noise component is removed and the green component extracted fluorescent fundus angiography image from which the noise component is removed, which have the rotation component corrected so as to make the position of the eyeball consistent and have been adjusted to a predetermined size.

[0233] Fig.14 This is a diagram showing an example of an abnormal blood circulation area.

[0234] The learning information generating unit 233 generates NPA·NV learning information by associating information representing the fundus image with the blood circulation abnormality region determined based on the fundus fluorescence angiography image corresponding to the fundus image, and stores the generated NPA·NV learning information in the learning information 215 of the storage unit 210. Return Fig.11 Continue with the explanation.

[0235] The learning unit 234 acquires the NPA·NV learning information stored in the learning information 215 of the storage unit 210. The learning unit 234 acquires the information representing the fundus image and the information representing the abnormal blood circulation region contained in the acquired NPA·NV learning information. The learning unit 234 uses the acquired information representing the fundus image as input information and the abnormal blood circulation region determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and generates a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image through learning.

[0236] Specifically, in a variation of the present embodiment, the following case is described: it is assumed that it is difficult to learn the entire image at once and make predictions due to the memory constraints of the GPU (graphics processing unit), so a method of extracting patches from the image and making the neural network learn the extracted patches is used. Here, as an example, the patch size is set to 64px×64px, and the stride (the interval at which the frame of the extracted patch is moved) is set to 2px.

[0237] The generated fragments are divided into two groups, fragments including positive regions and fragments including nothing. The fragments are selected in such a way that the proportions of the two groups used for learning are equal.

[0238] The phenomenon that a neural network only shows good performance for the learning data itself or images that are very similar to the learning data, and shows significantly lower performance for unknown images is called over-fitting. In order to solve this phenomenon, it is improved by collecting more samples or applying geometric operations such as rotation to the learning data.

[0239] In a variation of the present embodiment, after the fragments are generated, the rotation angle is determined based on a normal distribution of σ=3 degrees, and horizontal flipping is performed with a probability of 50%, and vertical flipping is performed with a probability of 20%.

[0240] Fig.15 This is a diagram showing an example of the structure of a neural network.

[0241] An example of a neural network construction is based on U-Net. Fig.15 In the figure, b represents convolution (kernel size = (3, 3)), g represents maximum pooling (pooling size = (2, 2)), and o represents upsampling (size = (2, 2)). Batch normalization is performed using the activation function ReLU after each convolution layer. However, in the last convolution layer, sigmoid is used as the activation function and batch normalization is not performed.

[0242] In addition, the arrows indicated by a1, a2, a3, and a4 represent skip connections based on concatenation, which are considered to contribute to the position information of the restored image.

[0243] The learning unit 234 stores the generated learning model in the learning model 216 of the storage unit 210 .

[0244] The learning unit 234 generates learning model notification information including the generated learning model and addressed to the diagnosis support apparatus 300 , and outputs the generated learning model notification information to the communication unit 205 .

[0245] (Diagnosis Support Device 300)

[0246] Fig.16 This is a block diagram showing an example of a diagnosis support device according to a modification of the embodiment of the present invention.

[0247] The diagnosis support device 300 includes a communication unit 305, a storage unit 310, an operation unit 320, an information processing unit 330, a display unit 340, and a display unit 341 for displaying each component. Fig.16 A bus 350 such as an address bus and a data bus are electrically connected as shown.

[0248] The communication unit 305 is implemented by a communication module. The communication unit 305 communicates with external communication devices such as the ophthalmologist terminal 2 and the learning device 200 via the network N. Specifically, the communication unit 305 receives the learning model notification information sent by the learning device 200, and outputs the received learning model notification information to the information processing unit 330. The communication unit 305 receives the patient information sent by the ophthalmologist terminal 2, and outputs the received patient information to the information processing unit 330. The communication unit 305 obtains the diagnosis information output by the information processing unit 230, and sends the obtained diagnosis information to the ophthalmologist terminal 2.

[0249] The storage unit 310 is implemented by, for example, RAM, ROM, HDD, flash memory, or a hybrid storage device combining multiple of them, etc. The storage unit 310 stores a program 311 executed by the information processing unit 330 , an APP 312 , and a learning model 216 .

[0250] The program 311 is, for example, an operating system, which is located between users, applications, and hardware. While providing a standard interface to users and applications, it also efficiently manages various resources such as hardware.

[0251] APP312 enables the diagnosis assisting device 300 to receive the learning model notification information sent by the learning device 200, and enables the diagnosis assisting device 300 to store the learning model included in the received learning model notification information. APP312 enables the diagnosis assisting device 300 to receive the patient information sent by the ophthalmologist terminal 2, and enables the diagnosis assisting device 300 to obtain the patient ID and fundus image included in the received patient information. APP312 enables the diagnosis assisting device 300 to determine the blood circulation abnormality area in the acquired fundus image using the stored learning model.

[0252] APP312 causes the diagnosis assisting device 300 to generate diagnosis information (the diagnosis information includes information indicating an abnormal blood circulation area in the fundus image of the patient determined using the fundus image of the patient and a learning model, and the patient ID, and the ophthalmologist terminal 2 is the destination for sending), and causes the diagnosis assisting device 300 to send the generated diagnosis information to the ophthalmologist terminal 2.

[0253] The operation unit 320 is constituted by, for example, a touch panel, detects a touch operation performed on a screen displayed on the display unit 340 , and outputs the detection result of the touch operation to the information processing unit 330 .

[0254] The display unit 340 is constituted by, for example, a touch panel, and displays a screen for receiving information representing a fundus image included in the patient information received by the diagnosis supporting apparatus 300. In addition, the display unit 240 displays the result of diagnosis by the diagnosis supporting apparatus 300.

[0255] All or part of the information processing unit 330 is a software functional unit implemented by, for example, a processor such as a CPU executing the program 311 and the APP 312 stored in the storage unit 310. In addition, all or part of the information processing unit 330 may also be implemented by hardware such as an LSI, an ASIC, or an FPGA, or by a combination of a software functional unit and hardware. The information processing unit 330 includes, for example, a receiving unit 331, a determining unit 332, and a generating unit 333.

[0256] The receiving unit 331 receives the learning model notification information output by the communication unit 305 , and receives the learning model included in the received learning model notification information. The receiving unit 331 receives the received learning model, and stores the received learning model in the learning model 216 of the storage unit 310 .

[0257] The receiving unit 331 receives the patient information output by the communication unit 305 and receives the patient ID and information indicating the fundus image contained in the received patient information. The receiving unit 331 receives the received patient ID and information indicating the fundus image and outputs the received patient ID and information indicating the fundus image to the specifying unit 332.

[0258] The determination unit 332 acquires the patient ID and the information indicating the fundus image output by the receiving unit 331. The determination unit 332 acquires the learning model stored in the learning model 216 of the storage unit 310, and determines the blood circulation abnormality region in the acquired fundus image using the acquired learning model. The determination unit 332 outputs the information indicating the blood circulation abnormality region in the determined fundus image and the patient ID to the generation unit 333.

[0259] Specifically, in a modified example of the present embodiment, the following case is described: similarly to the learning device 200, fragments are extracted from the image, and the blood circulation abnormality region in the fundus image is determined using the extracted fragments and the learning model. Here, as an example, the fragment size is set to 64px×64px, and the stride (the interval at which the fragment extraction frame moves) is set to 2px. All the generated fragments are selected.

[0260] The determination unit 332 acquires a 64×64×1 image based on the learning model. The determination unit 332 votes for the acquired pixel values ​​at the corresponding pixels in the original image and averages them. Here, the determination unit 332 may also convert the acquired image to color display.

[0261] The generating unit 333 acquires the patient ID and the information indicating the abnormal blood circulation area in the fundus image output by the determining unit 332. The generating unit 333 generates diagnostic information including the acquired patient ID and the information indicating the abnormal blood circulation area in the fundus image, and sends the diagnostic information to the ophthalmologist terminal 2. The generating unit 333 outputs the generated diagnostic information to the communication unit 305.

[0262] (Operation of the information processing system)

[0263] Reference Fig.17 and Fig.18 An example of the operation of the information processing system according to the modification of the present embodiment will be described.

[0264] Fig.17 This is a flowchart showing an example of the operation of the learning device included in the information processing system according to the modification example of the present embodiment. Fig.17 This shows the operation after the ophthalmologist terminal 2 sends NPA·NV annotation notification information and incidental finding annotation notification information to the learning device 200 , and the testing device 3 sends fundus image notification information and fluorescein fundus angiography image notification information to the learning device 200 .

[0265] (Step S201)

[0266] The communication unit 205 of the learning device 200 receives the fundus image notification information transmitted by the inspection device 3, and outputs the received fundus image notification information to the information processing unit 230. The image acquisition unit 231 of the information processing unit 230 acquires the fundus image notification information output by the communication unit 205, and associates the patient ID and the information indicating the fundus image included in the acquired fundus image notification information with each other and stores them in the image DB 213 of the storage unit 210.

[0267] (Step S202)

[0268] The communication unit 205 of the learning device 200 receives the fluorescein fundus angiography image notification information transmitted by the inspection device 3, and outputs the received fluorescein fundus angiography image notification information to the information processing unit 230. The image acquisition unit 231 of the information processing unit 230 acquires the fluorescein fundus angiography image notification information output by the communication unit 205, and associates the patient ID and the information indicating the fluorescein fundus angiography image included in the acquired fluorescein fundus angiography image notification information with each other and stores them in the image DB 213 of the storage unit 210.

[0269] (Step S203)

[0270] The communication unit 205 of the learning device 200 receives the NPA·NV annotation notification information sent by the ophthalmologist terminal 2, and outputs the received NPA·NV annotation notification information to the information processing unit 230. The annotation acquisition unit 232 of the information processing unit 230 acquires the NPA·NV annotation notification information output by the communication unit 205, and associates the patient ID and NPA·NV annotation information included in the acquired NPA·NV annotation notification information with each other and stores them in the annotation DB 214 of the storage unit 210.

[0271] The communication unit 205 of the learning device 200 receives the incidental finding annotation notification information sent by the ophthalmologist terminal 2, and outputs the received incidental finding annotation notification information to the information processing unit 230. The annotation acquisition unit 232 of the information processing unit 230 acquires the incidental finding annotation notification information output by the communication unit 205, and associates the patient ID and the incidental finding annotation information included in the acquired incidental finding annotation notification information with each other and stores them in the annotation DB 214 of the storage unit 210.

[0272] (Step S204)

[0273] The learning information generating unit 233 of the learning device 200 acquires the patient ID, the information representing the fundus image associated with the patient ID, and the information representing the fluorescein fundus angiography image stored in the image DB 213 of the storage unit 210. The learning information generating unit 233 acquires the patient ID, the NPA·NV annotation information associated with the patient ID, and the incidental finding annotation information stored in the annotation DB 214 of the storage unit 210. The learning information generating unit 233 determines the abnormal blood circulation region based on the acquired information representing the fluorescein fundus angiography image, the NPA·NV annotation information, and the incidental finding annotation information. The learning information generating unit 233 generates NPA·NV learning information obtained by associating the information representing the fundus image with the abnormal blood circulation region determined based on the fluorescein fundus angiography image corresponding to the fundus image, and stores the generated NPA·NV learning information in the learning information 215 of the storage unit 210.

[0274] (Step S205)

[0275] The learning unit 234 of the learning device 200 acquires the NPA·NV learning information stored in the learning information 215 of the storage unit 210. The learning unit 234 acquires the information representing the fundus image and the information representing the abnormal blood circulation region contained in the acquired NPA·NV learning information. The learning unit 234 uses the acquired information representing the fundus image as input information and the abnormal blood circulation region determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and generates a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image through learning. The learning unit 234 stores the generated learning model in the learning model 216 of the storage unit 210.

[0276] exist Fig.17 In the flowchart shown, the order of steps S201, S202, and S203 may also be changed.

[0277] according to Fig.17 According to the flowchart shown, the learning device 200 can determine the abnormal blood circulation area based on the information representing the fluorescent fundus angiography image, the NPA·NV annotation information, and the incidental finding annotation information. The learning device 200 can use the information representing the fundus image as input information, and the abnormal blood circulation area determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information, and generate a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image through learning.

[0278] Fig.18 This is a flowchart showing an example of the operation of the diagnosis support device included in the information processing system according to the modification example of the present embodiment. Fig.18This diagram shows the operation after the learning device 200 transmits the learning model notification information to the diagnosis supporting device 300 and the ophthalmologist terminal 2 transmits the patient information to the diagnosis supporting device 300 .

[0279] (Step S301)

[0280] The communication unit 305 of the diagnosis support device 300 receives the learning model notification information sent by the learning device 200, and outputs the received learning model notification information to the information processing unit 330. The receiving unit 331 obtains the learning model notification information output by the communication unit 305, and obtains the learning model included in the obtained learning model notification information. The receiving unit 331 accepts the obtained learning model and stores the accepted learning model in the learning model 216 of the storage unit 310.

[0281] (Step S302)

[0282] The communication unit 305 receives the patient information sent by the ophthalmologist terminal 2, and outputs the received patient information to the information processing unit 330. The receiving unit 331 acquires the patient information output by the communication unit 305, and acquires the patient ID and the information indicating the fundus image contained in the acquired patient information. The receiving unit 331 accepts the acquired patient ID and the information indicating the fundus image, and outputs the accepted patient ID and the information indicating the fundus image to the determining unit 332.

[0283] (Step S303)

[0284] The specifying unit 332 acquires the patient ID and the information indicating the fundus image output by the accepting unit 331. The specifying unit 332 acquires the learning model stored in the learning model 216 of the storage unit 310.

[0285] (Step S304)

[0286] The determination unit 332 acquires the learning model stored in the learning model 216 of the storage unit 310, and uses the acquired learning model to determine the abnormal blood circulation region in the fundus image as the determination target. The determination unit 332 outputs information indicating the abnormal blood circulation region in the determined fundus image and the patient ID to the generation unit 333.

[0287] (Step S305)

[0288] The generating unit 333 acquires the patient ID and the information indicating the abnormal blood circulation region in the fundus image output by the determining unit 332. The generating unit 333 generates diagnostic information including the patient ID and the acquired information indicating the abnormal blood circulation region in the fundus image, and sends the diagnostic information to the ophthalmologist terminal 2. The generating unit 333 outputs the generated diagnostic information to the communication unit 305.

[0289] The communication unit 305 acquires the diagnostic information output by the generation unit 333 , and transmits the acquired diagnostic information to the ophthalmologist terminal 2 .

[0290] according to Fig.18 As shown in the flowchart, the diagnosis assisting device 300 is able to determine the abnormal blood circulation area in the fundus image as the determination object using a learning model representing the relationship between the fundus image and the abnormal blood circulation area in the fundus image, wherein the learning model is generated using information representing the fundus image as input information and the abnormal blood circulation area determined based on the fluorescent fundus angiography image corresponding to the fundus image as training information.

[0291] In the above-mentioned modified example, the following case is described: the diagnosis supporting device 300 receives the learning model notification information sent by the learning device 200, and stores the learning model included in the received learning model notification information in the storage unit 310; the diagnosis supporting device 300 uses the fundus image included in the patient information sent by the ophthalmologist terminal 2 and the stored learning model to determine the abnormal blood circulation area in the fundus image, and sends the diagnosis result including the information indicating the determined abnormal blood circulation area to the ophthalmologist terminal 2, but the present invention is not limited to this example. For example, the diagnosis supporting device 300 may also send the patient information to the learning device 200. The learning device 200 receives the patient information sent by the diagnosis supporting device 300, and uses the fundus image included in the received patient information and the learning model to determine the abnormal blood circulation area in the fundus image. The learning device 200 generates a diagnosis result including the information indicating the determined abnormal blood circulation area, and sends the generated diagnosis result to the diagnosis supporting device 300. The diagnosis support device 300 receives the diagnosis result transmitted by the learning device 200 , and transmits the received diagnosis result to the ophthalmologist terminal 2 .

[0292] According to a variation of the present embodiment, the diagnosis assisting device 300 includes: a determination unit that determines a blood circulation abnormality region in a fundus image as an image of the fundus using a learning model obtained by learning the relationship between the fundus image and the blood circulation abnormality region in the fundus image based on the fundus image and the blood circulation abnormality region determined from a fundus fluorescence angiography image of the fundus; and an output unit that outputs information indicating the blood circulation abnormality region in the fundus image of the patient determined by the determination unit using the fundus image of the patient and the learning model. By configuring in this way, the diagnosis assisting device 300 can estimate the circulation abnormality region from a normal fundus image with high accuracy using the learning model obtained by machine learning using the circulation abnormality region information determined by a doctor from a fundus fluorescence angiography image and the fundus image corresponding to the circulation abnormality region information as training information.

[0293] The abnormal blood circulation region is generated based on the fluorescent fundus angiography image and the ophthalmologist's diagnostic annotation related to one or both of the retinal non-perfusion region and the neovascularization image attached to the fluorescent fundus angiography image. With such a configuration, the abnormal blood circulation region can be generated as training information based on the fluorescent fundus angiography image and the ophthalmologist's diagnostic annotation related to one or both of the retinal non-perfusion region and the neovascularization image attached to the fluorescent fundus angiography image.

[0294] The identification unit identifies one or both of the retinal non-perfusion area and the area corresponding to the neovascularization in the fundus image. With this configuration, it is possible to identify one or both of the retinal non-perfusion area and the area corresponding to the neovascularization with high accuracy from a normal fundus image.

[0295] The output unit outputs an image in which the abnormal blood circulation region specified by the specifying unit is superimposed on the fundus image. With this configuration, an image in which the abnormal blood circulation region is superimposed on the fundus image can be acquired from a normal fundus image.

[0296] According to a variation of the present embodiment, the learning device 200 includes a learning unit that generates a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image through learning based on the fundus image as an image of the fundus and the abnormal blood circulation region determined based on the fundus fluorescence angiography image of the fundus. By configuring in this way, the learning device 200 can generate a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image through machine learning.

[0297] The abnormal blood circulation region is generated based on the fluorescent fundus angiography image and the ophthalmologist's diagnostic annotation related to one or both of the retinal non-perfusion region and the neovascularization image attached to the fluorescent fundus angiography image. With such a configuration, the abnormal blood circulation region can be generated as training information based on the fluorescent fundus angiography image and the ophthalmologist's diagnostic annotation related to one or both of the retinal non-perfusion region and the neovascularization image attached to the fluorescent fundus angiography image.

[0298] The embodiments and modifications of the present invention are described above, but these embodiments and modifications are provided as examples and are not intended to limit the scope of the invention. These embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, changes, and combinations can be performed without departing from the scope of the invention. These embodiments and modifications are included in the scope and purpose of the invention, and are included in the invention described in the claims and their equivalents.

[0299] In addition, the aforementioned server 1, ophthalmologist terminal 2, inspection equipment 3, learning device 200, and diagnosis support device 300 have computers inside. Furthermore, the processes of each of the aforementioned devices are stored in a computer-readable recording medium in the form of a program, and the above-mentioned processes are performed by reading and executing the program by a computer. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the computer program may be distributed to a computer using a communication line, and the computer receiving the distribution may execute the program.

[0300] In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned functions.

[0301] Furthermore, it may be a so-called differential file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0302] In addition, the following supplementary notes are further disclosed with respect to the above description.

[0303] (Supplementary Note 1) A program that causes a computer that controls an information processing device to execute a control process including the following steps:

[0304] A step of acquiring a fluorescent fundus angiography image, acquiring fluorescent fundus angiography image information;

[0305] an NPA·NV annotation acquisition step of acquiring an ophthalmologist's diagnostic annotation related to at least one of a retinal non-perfused area and a neovascularization, which is attached to the above-mentioned fundus fluorescein angiography image information, as NPA·NV annotation information;

[0306] An NPA·NV training information generating step, based on the above-mentioned fluorescent fundus angiography image information and the above-mentioned NPA·NV annotation information corresponding to the fluorescent fundus angiography image information, generates NPA·NV training information, and the NPA·NV training information becomes training information for calculating the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of neovascularization;

[0307] A fundus image acquisition step, acquiring fundus image information;

[0308] NPA·NV existence probability calculation step, calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information based on the NPA·NV training information; and

[0309] The estimated NPA·NV determination step determines, based on the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of the above-mentioned neovascularization, the area in the above-mentioned fundus image information estimated to be equivalent to the above-mentioned retinal non-perfusion area as the estimated NPA, and the area estimated to be equivalent to the above-mentioned neovascularization is determined as the estimated NV.

[0310] (Supplementary Note 2) A program according to Supplementary Note 1, further comprising an NPA·NV existence probability map generating step, in which an NPA·NV existence probability map is generated so that the fundus image information can identify and display the NPA existence probability and NV existence probability.

[0311] (Supplement 3) The procedure according to Supplement 1 or 2, further comprising the following steps:

[0312] an incidental annotation acquisition step of acquiring, as incidental finding annotation information, the ophthalmologist's diagnostic annotation related to the incidental finding, which is added to the fluorescein fundus angiography image information and the fundus image information;

[0313] a step of generating incidental finding training information, based on the fluorescent fundus angiography image information and the fundus image information, and the incidental finding annotation information corresponding to the fluorescent fundus angiography image information and the fundus image information, generating incidental finding training information, the incidental finding training information being training information for calculating the existence probability of the incidental finding in the fundus image information; and

[0314] an incidental finding existence probability calculation step of calculating the existence probability of the incidental finding in the fundus image information based on the incidental finding training information,

[0315] In the NPA·NV existence probability calculation step, the following control processing is further performed: based on the existence probability of the incidental finding and the NPA training information, the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information are calculated.

[0316] (Supplementary Note 4) A program according to Supplementary Note 3, further comprising a step of generating a probability map of incidental findings, in which a probability map of incidental findings is generated so that the fundus image information can be identified to display the probability of the incidental findings.

[0317] (Supplementary Note 5) An information processing method is an information processing method executed by an information processing device, wherein:

[0318] The information processing method comprises the following steps:

[0319] A step of acquiring a fluorescent fundus angiography image, acquiring fluorescent fundus angiography image information;

[0320] an NPA·NV annotation acquisition step of acquiring an ophthalmologist's diagnostic annotation related to at least one of a retinal non-perfused area and a neovascularization, which is attached to the above-mentioned fundus fluorescein angiography image information, as NPA·NV annotation information;

[0321] An NPA·NV training information generating step, based on the above-mentioned fluorescent fundus angiography image information and the above-mentioned NPA·NV annotation information corresponding to the fluorescent fundus angiography image information, generates NPA·NV training information, and the NPA·NV training information becomes training information for calculating the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of neovascularization;

[0322] A fundus image acquisition step, acquiring fundus image information;

[0323] NPA·NV existence probability calculation step, calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information based on the NPA·NV training information; and

[0324] The estimated NPA·NV determination step determines, based on the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of the above-mentioned neovascularization, the area in the above-mentioned fundus image information estimated to be equivalent to the above-mentioned retinal non-perfusion area as the estimated NPA, and the area estimated to be equivalent to the above-mentioned neovascularization is determined as the estimated NV.

[0325] (Supplementary Note 6) An information processing device comprising:

[0326] A fluorescent fundus angiography image acquisition unit, which acquires fluorescent fundus angiography image information;

[0327] an NPA·NV annotation acquisition unit that acquires an ophthalmologist's diagnostic annotation related to at least one of a retinal non-perfused area and a neovascularization, which is added to the above-mentioned fluorescein fundus angiography image information, as NPA·NV annotation information;

[0328] An NPA·NV training information generating unit, which generates NPA·NV training information based on the fluorescent fundus angiography image information and the NPA·NV annotation information corresponding to the fluorescent fundus angiography image information, the NPA·NV training information being training information for calculating the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization;

[0329] A fundus image acquisition unit, which acquires fundus image information;

[0330] An NPA·NV existence probability calculation unit calculates the existence probability of the retinal non-perfusion area and the existence probability of the neovascularization in the fundus image information based on the NPA·NV training information; and

[0331] An estimated NPA·NV determination unit determines, based on the existence probability of the above-mentioned retinal non-perfusion area and the existence probability of the above-mentioned neovascularization, an area in the above-mentioned fundus image information estimated to be equivalent to the above-mentioned retinal non-perfusion area as an estimated NPA, and determines an area estimated to be equivalent to the above-mentioned neovascularization as an estimated NV.

[0332] Description of Reference Numerals

[0333] 1: Server,

[0334] 2: Ophthalmologist terminal,

[0335] 3: Check the equipment,

[0336] 11: CPU,

[0337] 12: ROM,

[0338] 13: RAM,

[0339] 14: bus,

[0340] 15: Input and output interface,

[0341] 16: Display unit,

[0342] 17: Input section,

[0343] 18: Storage Department,

[0344] 19: Ministry of Communications,

[0345] 20: Driver,

[0346] 30: Removable media,

[0347] 101: Image acquisition unit,

[0348] 102: Annotation acquisition unit,

[0349] 103: training information generation unit,

[0350] 104: Operation unit,

[0351] 105: Graph Generation Department,

[0352] 106: Presumed NPA and NV determination department,

[0353] 107: estimated NPA·NV display control unit,

[0354] 200: learning device,

[0355] 205: Ministry of Communications,

[0356] 210: Storage Department,

[0357] 211: Procedure,

[0358] 212: APP,

[0359] 213: Image DB,

[0360] 214: Annotation DB,

[0361] 215: Learning information,

[0362] 216: Learning Model,

[0363] 220: Operation Department,

[0364] 230: Information Processing Department,

[0365] 231: Image acquisition unit,

[0366] 232: Annotation acquisition department,

[0367] 233: Learning Information Generation Department,

[0368] 234: Learning Department,

[0369] 240: display unit,

[0370] 250: bus,

[0371] 300: Diagnostic aids,

[0372] 305: Ministry of Communications,

[0373] 310: storage unit,

[0374] 311: Procedure,

[0375] 312: APP,

[0376] 320: Operation Department,

[0377] 330: Information Processing Department,

[0378] 331: Receiving Department,

[0379] 332: Determination Department,

[0380] 333: Generation Department,

[0381] 240: display unit,

[0382] 350: bus,

[0383] 401: Image DB,

[0384] 402: Annotation DB,

[0385] 403: Training DB,

[0386] A: Presumed NPA,

[0387] B: Presumed NV,

[0388] C: Fluorescent fundus angiography image information,

[0389] D: Ophthalmologist,

[0390] E: NPA·NV existence probability diagram,

[0391] F: Fundus image information.

Claims

1. A diagnostic aid device comprising: a determination unit that determines an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model obtained by learning a relationship between the fundus image and the abnormal blood circulation region in the fundus image based on the fundus image and the abnormal blood circulation region determined based on a fluorescein fundus angiography image of the fundus; and An output unit outputs information indicating a blood circulation abnormality region in the fundus image of the patient determined by the determination unit using the fundus image of the patient and the learning model.

2. The diagnostic aid according to claim 1, wherein: The abnormal blood circulation area is generated based on the fluorescein fundus angiography image and an ophthalmologist's diagnosis annotation related to one or both of a retinal non-perfusion area and a neovascularization, which is attached to the fluorescein fundus angiography image.

3. The diagnostic aid according to claim 1 or 2, wherein: The specifying unit specifies one or both of a retinal non-perfusion area and an area corresponding to a neovascularization in the fundus image.

4. The diagnostic support device according to any one of claims 1 to 3, wherein: The output unit outputs an image in which the abnormal blood circulation region specified by the specifying unit is superimposed on the fundus image.

5. A learning device comprising a learning unit that generates, by learning, a learning model representing the relationship between a fundus image and the abnormal blood circulation area in the fundus image, based on a fundus image as an image of the fundus and an abnormal blood circulation area determined based on a fluorescent fundus angiography image of the fundus.

6. The learning device according to claim 5, wherein: The abnormal blood circulation area is generated based on the fluorescein fundus angiography image and an ophthalmologist's diagnosis annotation related to one or both of a retinal non-perfusion area and a neovascularization, which is attached to the fluorescein fundus angiography image.

7. A diagnosis assistance method, which is a diagnosis assistance method performed by a diagnosis assistance device, wherein: In this diagnostic aid method, determining an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model, the learning model being obtained by learning a relationship between the fundus image and the abnormal blood circulation region determined based on the fundus image and the fluorescent fundus angiography image of the fundus; Information indicating an abnormal blood circulation region in the fundus image of the patient determined using the fundus image of the patient and the learning model is output.

8. A learning method, which is a learning method performed by a learning device, wherein: In this learning method, acquiring information representing a fundus image as an image of the fundus and information representing a blood circulation abnormality region determined based on the fundus fluorescence angiography image of the fundus; Based on the acquired fundus image and the abnormal blood circulation region determined based on the fluorescent fundus angiography image of the fundus, a learning model representing the relationship between the fundus image and the abnormal blood circulation region in the fundus image is generated through learning.

9. A program, wherein: The program causes a computer of a diagnosis assisting device to determine an abnormal blood circulation region in a fundus image as an image of the fundus using a learning model obtained by learning a relationship between the fundus image and the abnormal blood circulation region determined based on a fundus fluorescence angiography image of the fundus; and The program causes a computer of a diagnosis supporting apparatus to output information indicating a blood circulation abnormality region in a fundus image of a patient determined using the fundus image of the patient and the learning model.

10. A program, wherein: The program causes a computer of a learning device to acquire information representing a fundus image as an image of the fundus and information representing an abnormal blood circulation region determined based on a fluorescein fundus angiography image of the fundus; and The program causes a computer of a learning device to generate a learning model representing a relationship between a fundus image and an abnormal blood circulation area in the fundus image through learning based on the acquired fundus image and the abnormal blood circulation area determined based on the fluorescent fundus angiography image of the fundus.