Ophthalmic image processing device and ophthalmic image processing program

Abstract

To provide an ophthalmologic image processing device and an ophthalmologic image processing program capable of appropriately acquiring an image useful for diagnosis, etc. by a doctor.SOLUTION: A control unit of an ophthalmologic image processing device executes an image acquisition step and a predicted disease image acquisition step. In the image acquisition step, the control unit acquires a plurality of kinds of ophthalmologic images 30A and 30B taken of the same tissue of an eye to be examined. In the predicted disease image acquisition step, the control unit acquires a predicted disease image 60 by inputting the plurality of kinds of ophthalmologic images 30A and 30B in a mathematical model trained by a machine-learning algorithm. The predicted disease image 60 is an image in which a predicted future lesioned part is modified for one of the plurality of kinds of ophthalmologic images 30A and 30B input in the mathematical model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an ophthalmic image processing device that processes an ophthalmic image, which is an image of tissue of a subject's eye, and an ophthalmic image processing program executed by the ophthalmic image processing device. [Background technology]

[0002] In recent years, various techniques have been proposed for processing ophthalmologic images to obtain useful information. For example, an image processing device described in Patent Document 1 obtains a diagnosis result of a subject's eye and processes the image of the subject's eye using a processing method according to the obtained diagnosis result in order to obtain an image analysis result suitable for a disease. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-121885 A Summary of the Invention [Problem to be solved by the invention]

[0004] It would be useful for both doctors and patients if doctors could diagnose or explain to patients the state of tissue when a disease occurs, when a disease progresses, or when a disease is cured, using predicted images based on images of the patient's own tissues. However, conventionally, it was only possible to use images of other patients who were in a similar state to the predicted state of the patient.

[0005] A typical object of the present disclosure is to provide an ophthalmic image processing device and an ophthalmic image processing program capable of appropriately acquiring images that are useful for diagnosis by a doctor, etc. [Means for solving the problem]

[0006] An ophthalmic image processing device provided by a typical embodiment of the present disclosure is an ophthalmic image processing device that processes ophthalmic images, which are images of tissues of a subject's eye, and a control unit of the ophthalmic image processing device executes an image acquisition step of acquiring multiple types of ophthalmic images captured of the same tissue of the subject's eye, and a predicted disease image acquisition step of acquiring a predicted disease image, which is an image in which a predicted future lesion is modified for one of the input multiple types of ophthalmic images by inputting the multiple types of ophthalmic images into a mathematical model trained by a machine learning algorithm, and the multiple types of ophthalmic images input to the mathematical model in the predicted disease image acquisition step include a base image, which is a two-dimensional front image based on which the predicted disease image is modified, and a reference image, which is a two-dimensional tomographic image whose photographed subject is common to the photographed subject of the base image, and the predicted disease image acquisition step includes a step of acquiring information on the position and range of the lesion when a lesion is present in the base image, The mathematical model includes a prediction model that outputs a predicted lesion status image indicating the predicted future position and range of the lesion by inputting information on the position and range of the lesion in a base image, a reference image, and at least one of time information indicating the time from the capture of the base image and the reference image to the time the disease is predicted, and degree information indicating the degree of the disease at the time of the prediction, and a predictive disease image generation model that outputs the predicted disease image in which the lesion has been modified with respect to the base image by inputting the predicted lesion status image output by the prediction model, wherein the prediction model is trained using, as input training data, information on the position and range of the lesion, a two-dimensional tomographic image of the same type as the reference image, and at least one of time information and degree information, and using, as output training data, information on the position and range of the lesion in a two-dimensional frontal image captured of the same subject after the two-dimensional tomographic image used as the input training data, and the predictive disease image generation model ,two A three-dimensional front image is used as training data for output, and trainingThe mathematical model is trained using information on the location and extent of the lesion in the data as input training data, and when the base image, the reference image, and at least one of time information and extent information are input, the mathematical model outputs the predicted disease image in which the lesion has been modified from the input base image.

[0007] An ophthalmic image processing program provided by an exemplary embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images that are images of tissues of a subject's eye, and the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device to cause the ophthalmic image processing device to execute an image acquisition step of acquiring multiple types of ophthalmic images captured of the same tissue of the subject's eye, and a predicted disease image acquisition step of inputting the multiple types of ophthalmic images into a mathematical model trained by a machine learning algorithm to acquire a predicted disease image that is an image in which a predicted future lesion is modified for one of the input multiple types of ophthalmic images, and the multiple types of ophthalmic images input to the mathematical model in the predicted disease image acquisition step include a base image that is a two-dimensional front image that is a basis for modification to the predicted disease image, and a reference image that is a two-dimensional tomographic image whose photographed subject is common to the photographed subject of the base image, and in the predicted disease image acquisition step, a lesion area when a lesion area is present in the base image is acquired. the mathematical model includes a prediction model which outputs a predicted lesion condition image indicating a predicted future position and range of a lesion by inputting information on the position and range of the lesion in an original image, a reference image, and at least one of time information indicating the time from the capture of the original image and the reference image to the time of disease prediction, and degree information indicating the degree of the disease at the time of the prediction, and a predictive disease image generation model which outputs the predicted disease image in which the lesion is modified with respect to the original image by inputting the predicted lesion condition image output by the prediction model, the prediction model uses as input training data the information on the position and range of the lesion, a two-dimensional tomographic image of the same type as the reference image, and at least one of time information and degree information, and has been trained using as output training data information on the position and range of the lesion in a two-dimensional frontal image captured of the same subject after the two-dimensional tomographic image used as the input training data, and the predictive disease image generation model ,two A three-dimensional front image is used as training data for output, and trainingThe mathematical model is trained using information on the location and extent of the lesion in the data as input training data, and when the base image, the reference image, and at least one of time information and extent information are input, the mathematical model outputs the predicted disease image in which the lesion has been modified from the input base image.

[0008] According to the ophthalmic image processing device and the ophthalmic image processing program according to the present disclosure, images that are useful for diagnosis by a doctor or the like are appropriately obtained. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmologic image processing device 21, and photographing devices 11A and 11B. [Diagram 2] 1 is an explanatory diagram for explaining an example of a model structure of this embodiment for acquiring a predicted disease image 60. FIG. [Diagram 3] 4 is a flowchart showing an example of ophthalmologic image processing executed by the ophthalmologic image processing device 21. [Figure 4] FIG. 13 is an explanatory diagram for explaining an example of a model structure according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Summary> The control unit of the ophthalmic image processing device exemplified in the present disclosure executes an image acquisition step and a predicted disease image acquisition step. In the image acquisition step, the control unit acquires multiple types of ophthalmic images captured of the tissue of the same subject eye. In the predicted disease image acquisition step, the control unit acquires a predicted disease image by inputting the multiple types of ophthalmic images into a mathematical model trained by a machine learning algorithm. The predicted disease image is an image in which a predicted future lesion is modified from one of the multiple types of ophthalmic images input into the mathematical model.

[0011] According to the ophthalmic image processing device exemplified in the present disclosure, a predicted disease image of a subject is acquired based on the subject's own ophthalmic image. In other words, a predicted disease image in which an ophthalmic image of the subject who is to be explained by a doctor is appropriately modified is presented to the subject, not an image of a subject other than the subject to whom the doctor is to explain. Furthermore, in the ophthalmic image processing device exemplified in the present disclosure, in addition to the ophthalmic image (hereinafter referred to as the "base image") that is the basis for modifying the predicted disease image, an ophthalmic image of a type different from the base image (hereinafter referred to as the "reference image") is also input to a mathematical model to acquire a predicted disease image. The degree to which the disease state is expressed may differ depending on the type of ophthalmic image. Therefore, a predicted disease image in which the disease state is more appropriately predicted is acquired compared to when a predicted disease image is acquired based only on the base image.

[0012] For example, even if the state of a disease is not shown in the base image, the state of the disease may be shown in the reference image. Even in such a case, the ophthalmologic image processing device exemplified in the present disclosure can appropriately predict the state of the disease, including at least one of the position and the area of ​​the lesion caused by the disease, based on multiple types of ophthalmologic images.

[0013] In the predicted disease image acquisition step, the control unit acquires a predicted disease image based on an ophthalmic image of the subject that has been actually captured. That is, the control unit acquires a predicted disease image based on at least one of pixel values, tissue structure, brightness, contrast, etc. in a plurality of types of ophthalmic images that have been actually captured. Therefore, it is easier for the subject to feel the predicted disease state compared to when a predicted disease image is acquired without being based on the subject's own ophthalmic image (for example, when a predicted disease image is generated by ordinary image depiction software, etc.).

[0014] Devices that execute various steps including the image acquisition step and the predicted disease image acquisition step can be appropriately selected. For example, a control unit of a personal computer (hereinafter referred to as "PC") may execute all steps. That is, the control unit of the PC may acquire an ophthalmic image from an imaging device, and acquire a predicted disease image based on the acquired ophthalmic image. Also, control units of multiple devices (for example, at least one of an imaging device, a PC, a mobile terminal, and a server) may cooperate to execute each step.

[0015] The mathematical model may be trained by a plurality of training data sets. The training data set may include input training data and output training data (e.g., ground truth data). The input training data may include a plurality of types of ophthalmic images taken of the same tissue of the subject's eye. The output training data may include ophthalmic images of the same subject as the input training data that are actually taken after the input training data are taken. In this case, the mathematical model for outputting the predicted disease image is appropriately trained based on the actually taken ophthalmic images.

[0016] The control unit may further execute a prediction time information acquisition step. In the prediction time information acquisition step, the control unit acquires at least one of time information indicating the time from the capture time of the multiple types of ophthalmic images acquired in the image acquisition step to the prediction time at which the disease state of the test eye is predicted, and degree information indicating the degree of the disease of the test eye at the time of prediction. In the predicted image acquisition step, the control unit may acquire a predicted disease image by inputting at least one of the time information and degree information acquired in the prediction time information acquisition step (hereinafter, sometimes simply referred to as "time and degree information") into a mathematical model together with the multiple types of ophthalmic images.

[0017] In this case, a predicted disease image after the time indicated by the time information has elapsed, or a predicted disease image in which the disease state has changed to the extent indicated by the degree information is obtained based on a plurality of types of ophthalmologic images actually taken, thereby obtaining a predicted disease image in which the disease state is more appropriately predicted.

[0018] The method by which the control unit acquires the time / level information can be appropriately selected. For example, the control unit may receive an input of an instruction from a user to specify the time / level information. The control unit may acquire the time / level information designated by the user and input it to the mathematical model. In addition, one or more pieces of time / level information may be preset. The control unit may acquire the preset time / level information and input it to the mathematical model.

[0019] When using the time and degree information, the input training data for training the mathematical model may include the time and degree information in addition to the multiple types of ophthalmic images. In this case, the time information may be automatically obtained by calculating the difference between the shooting times of the multiple types of ophthalmic images included in the input training data and the shooting times of the ophthalmic images included in the output training data. In addition, the degree information may be input by a user who has confirmed the degree of the disease in the output training data.

[0020] The control unit may acquire a plurality of pieces of time information or a plurality of pieces of degree information different from each other in the prediction time information acquisition step. The control unit may acquire a plurality of predicted disease images different in time or degree by inputting each of the plurality of pieces of time information or the plurality of pieces of degree information together with the plurality of types of ophthalmic images into a mathematical model in the predicted disease image acquisition step. The control unit may display the plurality of predicted disease images on the display device in an order according to time or degree or by switching between them.

[0021] In this case, the user can more easily check a plurality of predicted disease images having different times from the time of shooting to the time of prediction or different degrees of disease at the time of prediction, so that the user can more appropriately diagnose the subject's eye.

[0022] The control unit may further acquire a plurality of ophthalmic images of the same type taken at different times in the image acquisition step. The control unit may acquire a predicted disease image by inputting a plurality of ophthalmic images taken at different times into a mathematical model in the predicted disease image acquisition step. A progression of the disease state is reflected in a plurality of ophthalmic images of the same type taken at different times. Therefore, a predicted disease image that takes into account the progression of the disease state is appropriately acquired by using a plurality of ophthalmic images taken at different times.

[0023] The plurality of ophthalmological images captured at different times may be ophthalmological images of the same type as the base image or may be ophthalmological images of a different type from the base image. Furthermore, for each of the plurality of types of ophthalmological images, a plurality of ophthalmological images captured at different times may be obtained.

[0024] When using multiple ophthalmic images captured at different times, the input training data for training the mathematical model may include multiple ophthalmic images of the same type captured at different times. In this case, the mathematical model can output a predicted disease image in which the progression of the disease state is appropriately taken into account by inputting multiple ophthalmic images captured at different times.

[0025] The control unit may further execute a condition information acquisition step. In the condition information acquisition step, the control unit acquires condition information indicating whether the condition of the disease at the time of future prediction is to be worsened or cured for the condition of the disease in the multiple types of ophthalmic images acquired in the image acquisition step. In the predicted image acquisition step, the control unit may acquire a predicted disease image by inputting the condition information together with the multiple types of ophthalmic images into a mathematical model.

[0026] In this case, at least one of the predicted disease images when the disease state worsens and the predicted disease images when the disease state is cured is obtained based on a plurality of types of ophthalmologic images actually taken, so that the user can more appropriately diagnose the subject's eye.

[0027] The method of the control unit acquiring the condition information can be appropriately selected. For example, the control unit may accept an input of an instruction from a user to specify the condition information (worsening or cured). The control unit may acquire the condition information designated by the user and input it to the mathematical model. In addition, when a plurality of ophthalmic images captured at different times are input to the mathematical model, the control unit may determine whether the progression of the disease state is worsening or cured from the plurality of ophthalmic images captured at different times, and acquire the determination result as the condition information. In this case, the control unit may determine the progression of the disease state by performing known image processing on the plurality of ophthalmic images. In addition, the control unit may acquire the condition information from the plurality of ophthalmic images captured at different times using a machine learning algorithm.

[0028] When obtaining a predicted disease image using status information, the input training data for training the mathematical model may include status information indicating whether the disease status of the output training data relative to the disease status of the input training data is aggravated or cured. The status information included in the input training data may be input by a user or may be automatically obtained by image processing or the like.

[0029] In the ophthalmic image acquiring step, the control unit may acquire a plurality of types of ophthalmic images captured by a plurality of different types of imaging devices (i.e., different modalities). When the same tissue is imaged by a plurality of types of imaging devices, the state of a disease may be strongly reflected in the ophthalmic images captured by some of the imaging devices. Therefore, by inputting a plurality of types of ophthalmic images captured by a plurality of types of imaging devices into a mathematical model, a predicted disease image in which the state of a disease is more accurately predicted is acquired.

[0030] In the image acquisition step, the control unit may acquire a plurality of types of ophthalmologic images including a front image of the tissue of the subject's eye captured in a direction along the imaging optical axis and a tomographic image of the tissue. A disease state may be strongly reflected in only one of the front image and the tomographic image. Therefore, by inputting both the front image and the tomographic image of the same tissue into the mathematical model, a predicted disease image in which the disease state is more accurately predicted is acquired.

[0031] The multiple types of ophthalmic images acquired in the image acquiring step are not limited to two types, a front image and a tomographic image. For example, the multiple types of ophthalmic images may include two types of front images captured by different types of imaging devices (e.g., a fundus camera and a scanning laser ophthalmoscope (SLO), etc.). The multiple types of ophthalmic images may include multiple tomographic images of different types. The multiple types of ophthalmic images may include a map image (front map image) that shows the distribution of the fundus structure (e.g., the distribution of the thickness of a specific layer, etc.) in two dimensions. The map image shows changes in the structure caused by a disease. Therefore, by including the map image in the multiple types of ophthalmic images, a more appropriate predicted disease image is generated. The types of ophthalmic images acquired in the image acquiring step are not limited to two types, and may be three or more types.

[0032] In addition, the imaging devices for capturing each of the multiple types of ophthalmologic images may be different types of imaging devices (i.e., different modalities) as described above, or may be the same modality. For example, when a front image and a tomographic image are captured in the image capturing step, the front image and the tomographic image captured by the same type of imaging device (e.g., OCT device) may be captured. In addition, a front image captured by a front image capturing device (e.g., fundus camera or SLO, etc.) and a tomographic image captured by a tomographic image capturing device (e.g., OCT device) may be captured in the image capturing step.

[0033] The ophthalmic image processing device may execute only a part of the techniques exemplified in the present disclosure. For example, the control unit of the ophthalmic image processing device may execute a process of acquiring a predicted disease image based on time information, degree information and an ophthalmic image, without executing a process of acquiring a predicted disease image based on a plurality of types of ophthalmic images. Even in this case, a predicted disease image after the time indicated by the time information has elapsed, or a predicted disease image in which the state of the disease has changed to the degree indicated by the degree information, is acquired based on an actually captured ophthalmic image.

[0034] In this case, the ophthalmic image processing device can also be expressed as follows: An ophthalmic image processing device processes an ophthalmic image, which is an image of the tissue of a test eye, and a control unit of the ophthalmic image processing device executes an image acquisition step of acquiring an ophthalmic image of the tissue of the test eye captured by an imaging device, a prediction time information acquisition step of acquiring at least one of time information indicating the time from the capture of the ophthalmic image acquired in the image acquisition step to a future prediction time for predicting the state of a disease of the test eye and degree information indicating the degree of the disease of the test eye at the prediction time, and a predicted disease image acquisition step of acquiring a predicted disease image, which is an image in which a predicted future lesion is modified from the input ophthalmic image, by inputting at least one of the time information and the degree information and the ophthalmic image into a mathematical model trained by a machine learning algorithm.

[0035] The control unit of the ophthalmic image processing device may execute a process of acquiring a predicted disease image based on a plurality of ophthalmic images taken at different times, instead of executing a process of acquiring a predicted disease image based on a plurality of types of ophthalmic images. Even in this case, a predicted disease image that takes into account the progression of the disease state is appropriately acquired.

[0036] In this case, the ophthalmic image processing device can also be expressed as follows: An ophthalmic image processing device processes ophthalmic images, which are images of tissues of a subject's eye, and a control unit of the ophthalmic image processing device executes an image acquisition step of acquiring multiple ophthalmic images of the same type captured at different times for the tissues of the same subject's eye, and a predicted disease image acquisition step of inputting the multiple ophthalmic images into a mathematical model trained by a machine learning algorithm, thereby acquiring a predicted disease image, which is an image in which a predicted future lesion has been modified from the input ophthalmic image (for example, an ophthalmic image captured most recently).

[0037] <Embodiment> (Device configuration) A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and photographing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The constructed mathematical model outputs information according to the training content based on the input information (details of the mathematical model will be described later with reference to FIG. 2). The ophthalmic image processing device 21 executes various processes for acquiring a predicted disease image 60 (see FIG. 2) using the mathematical model. The predicted disease image 60 is an image showing the state of tissue when a disease is predicted to occur, the state of tissue when a disease is predicted to worsen (progress), or the state of tissue when a disease is predicted to be cured. The predicted disease image 60 is acquired based on an image of the tissue of a subject (test eye) that is actually photographed. The photographing devices 11A and 11B photograph an ophthalmic image 30 (see FIG. 2), which is an image of the tissue of the test eye.

[0038] In the present embodiment, an example is shown in which images of the fundus tissue of the subject's eye are used as the ophthalmic image 30 and the predicted disease image 60. However, even if the ophthalmic image 30 and the predicted disease image 60 are images of tissue of the subject's eye other than the fundus, at least a part of the technology exemplified in this disclosure can be applied. In addition, medical images of biological tissue other than the subject's eye (e.g., images of internal organs, etc.) may be used.

[0039] As an example, a personal computer (hereinafter, referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using ophthalmologic images acquired from the photographing device 11A. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the photographing device 11A may function as the mathematical model construction device 1. Furthermore, control units of multiple devices (for example, the CPU of the PC and the CPU 13A of the photographing device 11A) may cooperate to construct a mathematical model.

[0040] Further, a PC is used as the ophthalmic image processing device 21 of this embodiment. However, a device capable of functioning as the ophthalmic image processing device 21 is not limited to a PC. For example, the photographing device 11B or a server may function as the ophthalmic image processing device 21. Furthermore, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmic image processing device 21. Control units of multiple devices (for example, the CPU of the PC and the CPU 13B of the photographing device 11B) may cooperate to perform various processes.

[0041] In addition, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than a CPU may be used for at least a part of the various devices. For example, a GPU may be used as a controller to speed up the processing.

[0042] The mathematical model construction device 1 will be described. The mathematical model construction device 1 is disposed, for example, at a manufacturer that provides an ophthalmic image processing device 21 or an ophthalmic image processing program to a user. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller that manages control, and a storage device 4 that can store programs, data, and the like. A mathematical model construction program for constructing a mathematical model is stored in the storage device 4. In addition, the communication I / F 5 connects the mathematical model construction device 1 to other devices (for example, an imaging device 11A and an ophthalmic image processing device 21, etc.).

[0043] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the mathematical model construction device 1. For example, at least one of a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 7. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 7. The display device 8 displays various images. For the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that, in this disclosure, "image" includes both still images and moving images.

[0044] The mathematical model construction device 1 can acquire information on the ophthalmic image 30 (hereinafter, may be simply referred to as "ophthalmic image") from the photographing device 11A. The mathematical model construction device 1 may acquire the information on the ophthalmic image 30 from the photographing device 11A by at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), and the like.

[0045] The ophthalmological image processing device 21 will be described. The ophthalmological image processing device 21 is disposed, for example, in a facility (for example, a hospital or a medical examination facility) where a diagnosis or examination is performed on a subject. The ophthalmological image processing device 21 includes a control unit 22 that performs various control processes, and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller that manages control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmological image processing program for executing ophthalmological image processing (see FIG. 3) described later. The ophthalmological image processing program includes a program for realizing a mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmological image processing device 21 to other devices (for example, the photographing device 11B and the mathematical model construction device 1, etc.).

[0046] The ophthalmologic image processing device 21 is connected to an operation unit 27 and a display device 28. As with the operation unit 7 and the display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0047] The ophthalmological image processing device 21 can acquire the ophthalmological image 30 from the photographing device 11B. The ophthalmological image processing device 21 may acquire the ophthalmological image 30 from the photographing device 11B by at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), etc. The ophthalmological image processing device 21 may also acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

[0048] The medical imaging devices 11A and 11B will be described. As an example, in this embodiment, a case will be described in which the medical imaging device 11A that provides the mathematical model construction device 1 with a medical image 30 and the medical imaging device 11B that provides the medical image 30 to the medical image processing device 21 are used. However, the number of medical imaging devices used is not limited to two. For example, the mathematical model construction device 1 and the medical image processing device 21 may acquire the medical image 30 from a plurality of medical imaging devices. Also, the mathematical model construction device 1 and the medical image processing device 21 may acquire the medical image 30 from one common medical imaging device. The two medical imaging devices 11A and 11B exemplified in this embodiment have the same configuration. Therefore, the two medical imaging devices 11A and 11B will be described together below.

[0049] The photographing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmologic image photographing section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that performs control, and a storage device 14 (14A, 14B) that can store programs, data, and the like.

[0050] The photographing device 11 of this embodiment can photograph a plurality of types of ophthalmologic images 30 using a plurality of photographing methods. That is, the photographing device 11 can photograph a plurality of types of ophthalmologic images 30 that differ from each other in at least one of the photographing direction (front image or tomographic image), resolution, brightness, contrast, etc., by photographing the same tissue of the subject eye (fundus tissue in this embodiment) using different photographing methods. In other words, the photographing device 11 of this embodiment has a plurality of modalities in one housing. As an example, the medical image photographing unit 16 of this embodiment has both a fundus camera configuration (illumination light source, illumination optical system, light receiving optical system, photographing element, etc.) that photographs a front image of the fundus of the subject eye, and an OCT device configuration (OCT light source, scanning unit that scans OCT light, irradiation optical system, light receiving optical system, light receiving element, etc.) that photographs a tomographic image of the fundus of the subject eye.

[0051] In this embodiment, one photographing device 11 photographs multiple types of ophthalmic images 30. However, the ophthalmic images 30 may be photographed by multiple types of photographing devices that photograph images using different methods. In other words, multiple modalities may be used to photograph the multiple types of ophthalmic images 30.

[0052] In addition, in this embodiment, a two-dimensional front image of the fundus captured by a fundus camera and a two-dimensional tomographic image of the fundus captured by an OCT device are used as the multiple types of ophthalmic images 30. However, it is also possible to change the type of ophthalmic images 30. For example, a two-dimensional front image of the fundus captured by a scanning laser ophthalmoscope (SLO) may be included in the multiple types of ophthalmic images 30. Also, a three-dimensional tomographic image of tissue (for example, a three-dimensional tomographic image of the fundus captured by an OCT device) may be included in the multiple types of ophthalmic images 30. A corneal endothelial cell imaging device (CEM), a computed tomography (CT) device, or the like may be used as the imaging device 11. Also, multiple types of two-dimensional front images or multiple types of tomographic images may be captured.

[0053] (Model structure of mathematical model) 2, a model structure of the mathematical model of this embodiment for the ophthalmologic image processing device 21 to obtain a predicted disease image 60 will be described. In this embodiment, a lesion removal model 71, a lesion identification model 72, a prediction model 73, and a predicted disease image generation model 74 are used as the mathematical models for obtaining the predicted disease image 60.

[0054] In addition, in this embodiment, a plurality of types of ophthalmic images 30 are input to the mathematical model to obtain a predicted disease image 60. As an example, in this embodiment, a base image 30A and a reference image 30B are input to the mathematical model as a plurality of types of ophthalmic images 30. The base image 30A is an image that is a base image to be modified into the predicted disease image 60. That is, the mathematical model outputs the predicted disease image 60 by modifying the base image 30A, which is one of the plurality of types of ophthalmic images 30, with a predicted future lesion. Therefore, the base image 30A and the predicted disease image 60 are the same type of image (in this embodiment, a two-dimensional front image of the fundus). The reference image 30B is an image whose photographed subject is the same as that of the base image 30A and is a different type of image from the base image 30A. The base image 30B is referred to in order to generate the predicted disease image 60.

[0055] As an example, in this embodiment, a two-dimensional front image of the fundus (a two-dimensional front image captured by a fundus camera) is used as the base image 30A, and a two-dimensional tomographic image of the fundus (a two-dimensional tomographic image captured by an OCT device) is used as the reference image 30B. However, as described above, it is also possible to change the types of the base image 30A and the reference image 30B. For example, a two-dimensional tomographic image may be used as the base image 30A, and a two-dimensional front image may be used as the reference image 30B. Furthermore, two or more types of reference images may be used.

[0056] Lesion-removed model 71 receives base image 30A and outputs lesion-removed image 40 in which information about lesion 31 has been removed from base image 30A. Various images that do not include information about lesion 31 and include information about tissue structure can be used as lesion-removed image 40. In this embodiment, an image of blood vessels (fundus blood vessel image), which is one of the tissue structures, is used as lesion-removed image 40.

[0057] Lesion identification model 72 receives base image 30A and outputs a detection result (lesion information) of lesion 31 when lesion 31 is present in base image 30A. In this embodiment, the lesion information output by lesion identification model 72 includes information on the position of lesion 31, information on the area of ​​lesion 31, and information on the type of lesion 31 (type of disease).

[0058] As an example, lesion identification model 72 of the present embodiment outputs attention map 80 when obtaining a detection result of the type of lesion 31 included in base image 30A. Information on the position and range of lesion 31 is acquired from attention map 80. Attention map 80 indicates the distribution within the image region of the influence (attention) of each position influenced when lesion identification model 72 obtains a detection result of the type of lesion 31. Regions with high influence have a stronger influence on the detection result of the type of lesion 31 than regions with low influence. An example of attention map 80 is described in, for example, the following paper. "Ramprasaath R. Selvaraju, et al. "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" Proceedings of the IEEE International Conference on Computer Vision, 2017-Oct, pp. 618-626"

[0059] However, it is also possible to change the method of acquiring information on the position and range of the lesion 31. For example, known image processing may be performed on the base image 30A to segment the area of ​​the lesion 31, thereby acquiring information on the position and range of the lesion 31.

[0060] The prediction model 73 predicts the future state of the lesion (in this embodiment, the position and range of the lesion). As an example, the prediction model 73 of this embodiment inputs information on the position and range of the lesion 31 output by the lesion identification model 72 (in this embodiment, the attention map 80) and the reference image 33 of the tissue of the subject's eye, and outputs a predicted lesion state image 81 that two-dimensionally indicates the predicted future position and range of the lesion. Depending on the type of disease, the degree of progression of the disease, the state of the subject's eye, etc., the state of the lesion 31 may not be accurately shown in the base image 30, and the state of the lesion 31 may be easily shown in the reference image 33. For example, when the degree of progression of the disease is shallow and the lesion 31 is formed only at a position deeper than the surface of the fundus, in this embodiment, the state of the lesion 31 is not easily shown in the base image 30, which is a two-dimensional front image, while the state of the lesion is easily shown in the reference image 33, which is a two-dimensional tomographic image. Therefore, by using a plurality of ophthalmologic images 30 (in this embodiment, the base image 30A and the reference image 30B), the future state of the affected area (disease state) is predicted more accurately.

[0061] Moreover, the prediction model 73 of this embodiment outputs a predicted lesion condition image 81 by inputting at least one of time information and degree information (hereinafter simply referred to as "time / degree information"). The time information is information indicating the time from the capture of the multiple types of ophthalmic images 30 to the prediction time when the disease state of the subject's eye is predicted. The degree information is information indicating the degree of disease of the subject's eye at the time of prediction. In this embodiment, the predicted lesion condition image 81 indicating the state of the lesion after the time indicated by the time information has elapsed, or the predicted lesion condition image 81 in which the state of the lesion (disease) has changed to the degree indicated by the degree information is acquired based on the multiple types of ophthalmic images 30 actually captured. Thus, a predicted disease image 60 in which the disease state is more appropriately predicted is acquired.

[0062] Furthermore, it is also possible to input a plurality of different time / degree information to the prediction model 73 of this embodiment. In this case, the prediction model 73 can output a plurality of predicted lesion state images 81 in which the time from the capture of the ophthalmologic image 30 to the time of prediction or the degree of disease at the time of prediction is different from each other. As a result, a plurality of predicted disease images 60 in which the time or degree is different is finally output.

[0063] In addition, the prediction model 73 of this embodiment can also input a plurality of ophthalmic images 30 (a plurality of reference images 30B in this embodiment) of the same type taken at different times and information indicating the time of each ophthalmic image 30. The transition of the disease state is shown in a plurality of ophthalmic images 30 of the same type taken at different times and of the same photographed subject. Therefore, by using a plurality of ophthalmic images 30 taken at different times and information indicating the respective photographing times, a predicted lesion state image 81 in which the transition of the disease state is taken into consideration is output by the prediction model 73. As a result, a predicted disease image 60 in which the transition of the disease state is taken into consideration is finally output.

[0064] In this embodiment, a plurality of reference images 30B captured at different times are input to the prediction model 73. However, a plurality of base images 30A captured at different times may be input to the prediction model 73. In addition, a plurality of base images 30A and a plurality of reference images 30B may be input to the prediction model 73 together.

[0065] Moreover, it is also possible to input state information to the prediction model 73 of this embodiment. The state information is information indicating whether the state of the disease (lesion) in the multiple types of ophthalmologic images 30 will be worsened or cured at the time of future prediction. By inputting state information to the prediction model 73, at least one of a predicted lesion state image 81 (predicted disease image 60) when the disease state is worsening and a predicted lesion state image 81 (predicted disease image 60) when the disease state is cured is output based on the multiple types of ophthalmologic images 30 that have actually been captured.

[0066] The predicted disease image generation model 74 outputs a predicted disease image 60 by inputting the lesion removal image 40, the type of disease (type of lesion 31), and information indicating the predicted state of the lesion (in this embodiment, a predicted lesion state image 81).

[0067] In this embodiment, the predicted disease image 60 is acquired based on at least one of the pixel values, tissue structure, brightness, contrast, etc. (in this embodiment, the tissue structure shown in the lesion-removed image 40 acquired from the base image 30A) in the actually captured ophthalmologic image 30 (specifically, the base image 30A). Therefore, it is easier for the subject to feel the predicted disease state compared to when the predicted disease image is acquired without being based on the subject's own ophthalmologic image 30.

[0068] An example of a mathematical model construction process executed by the mathematical model construction device 1 will be described. The CPU 3 of the mathematical model construction device 1 constructs a mathematical model by executing training of the mathematical model using a training data set by a machine learning algorithm. A program for realizing the constructed mathematical model is stored in the storage device 24 of the ophthalmologic image processing device 21. Commonly known examples of machine learning algorithms include neural networks, random forests, boosting, and support vector machines (SVMs).

[0069] Neural networks are a method to mimic the behavior of biological neural networks. Examples of neural networks include feedforward neural networks, radial basis function (RBF) networks, spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0070] Random forest is a method to generate a large number of decision trees by learning based on randomly sampled training data. When using random forest, the branches of multiple decision trees that have been trained in advance as classifiers are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

[0071] Boosting is a method to generate a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple weak classifiers.

[0072] SVM is a method for constructing a two-class pattern classifier using linear input elements. For example, SVM learns the parameters of the linear input elements based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

[0073] In this embodiment, each mathematical model is constructed using generative adversarial networks (GAN), which utilizes two competing neural networks, a convolutional neural network (CNN), which is a type of multi-layer neural network, or the like.

[0074] A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by training using multiple training data sets. A training data set is a set of input training data and output training data. A mathematical model is trained so that when certain input training data is input, corresponding output training data is output. For example, correlation data (e.g., weights) between each input and output is updated by training.

[0075] An example of a method for constructing the lesion removal model 71 will be described. In this embodiment, when constructing the lesion removal model 71, the CPU 3 of the mathematical model construction device 1 trains the mathematical model using an ophthalmologic image 30 (specifically, a two-dimensional front image of the same type as the base image 30) of the subject captured by the imaging device 11A as input training data and a lesion removal image (in this embodiment, a fundus blood vessel image) 40 of the same subject as output training data. The lesion removal image 40 may be captured by the imaging device 11A using a method different from the method of capturing the two-dimensional front image used as the input training data, or may be captured by a device different from the imaging device 11A. In addition, the lesion removal image 40 may be extracted from the two-dimensional front image used as the input training data by known image processing. In addition, the lesion removal image 40 may be extracted from the two-dimensional front image according to an instruction input by a user.

[0076] An example of a method for constructing the lesion identification model 72 will be described. In this embodiment, when constructing the lesion identification model 72, the CPU 3 trains a mathematical model using an ophthalmic image 30 of the subject captured by the imaging device 11A (specifically, a two-dimensional front image of the same type as the base image 30) as input training data, and information on the lesion 31 in the input training data (ophthalmic image 30) (in this embodiment, information on the position and range of the lesion 31 and information on the type of the lesion 31) as output training data. The information on the lesion 31 may be generated, for example, in response to an instruction input by a user who has confirmed the input training data (ophthalmic image 30). Also, at least a part of the information on the lesion 31 (for example, information on the position and range) may be generated by performing known image processing on the input training data.

[0077] An example of a method for constructing the prediction model 73 will be described. In this embodiment, when the CPU 3 constructs the prediction model 73, the input training data includes the above-mentioned information on the lesion 31 (in this embodiment, information on the position and range of the lesion 31) and the ophthalmic image 30 of the subject captured by the imaging device 11A (specifically, the same type of two-dimensional tomographic image as the reference image 30B). In addition, the output training data when constructing the prediction model 73 includes information on the state (position and range) of the lesion in the ophthalmic image 30 (two-dimensional front image) captured by the imaging device 11A of the same imaging subject after the ophthalmic image 30 used as the input training data when training the lesion removal model 71 and the lesion identification model 72. The information on the position and range of the lesion may be generated according to an instruction input by the user or may be generated by known image processing. In this embodiment, a two-dimensional image showing the position and range of the lesion is used as the information on the state of the lesion.

[0078] Furthermore, time and degree information is included in the input training data when constructing the prediction model 73. The time information may be automatically acquired by calculating the difference between the capture time of the input training data of the lesion removal model 71 and the lesion identification model 72 and the capture time of the ophthalmologic image 30 that is the source of the output training data of the prediction model 73. Furthermore, the time and degree information may be input by the user.

[0079] Furthermore, the input training data for constructing the prediction model 73 includes a plurality of ophthalmic images 30 taken at different times by the photographing device 11A and information indicating the time of photographing each of the ophthalmic images. Therefore, the constructed prediction model 73 can output a predicted lesion state image 81 in which the progression of the disease is appropriately taken into account, based on a plurality of ophthalmic images 80.

[0080] Furthermore, the input training data when constructing the prediction model 73 includes condition information indicating whether the condition of the lesion (disease) in the output training data is worsened or cured, relative to the condition of the lesion (disease) in the input training data. The condition information may be input by a user, or may be automatically obtained by image processing or the like.

[0081] An example of a method for constructing the predicted disease image generation model 74 will be described. In this embodiment, when constructing the predicted disease image generation model 74, the CPU 3 sets, as the output training data, an ophthalmic image (two-dimensional front image) 30 of the same subject photographed by the photographing device 11A after the ophthalmic image 30 used as the input training data when training the lesion removal model 71 and the lesion identification model 72. The CPU 3 also sets, as the input training data, a lesion removal image 40 in which information on the lesion has been removed from the output training data (ophthalmic image 30) and information on the lesion 31 related to the output training data (in this embodiment, information on the position and range of the lesion 31, and information on the type of the lesion 31). The method by which the CPU 3 acquires the lesion removal image 40 and the information on the lesion can be the same as the above-mentioned method.

[0082] As described above, the entire mathematical model of this embodiment including multiple mathematical models (lesion removal model 71, lesion identification model 72, prediction model 73, and predicted disease image generation model 74) is trained using at least multiple types of ophthalmic images 30 (two-dimensional front images and two-dimensional tomographic images) captured on the tissue of the same subject eye as input training data, and using ophthalmic images 30 (two-dimensional front images) captured after the input training data as output training data. Therefore, the mathematical model of this embodiment can appropriately output the predicted disease image 60 based on the ophthalmic images 30 actually captured. In addition, the input training data when training the entire mathematical model of this embodiment includes time and degree information, multiple ophthalmic images 30 (two-dimensional tomographic images) captured at different times, and condition information.

[0083] (Ophthalmology Image Processing) The ophthalmic image processing executed by the ophthalmic image processing device 21 of this embodiment will be described with reference to Fig. 3. The ophthalmic image processing is executed by the CPU 23 in accordance with an ophthalmic image processing program stored in the storage device 24. When an instruction to start generating and displaying a predicted disease image 60 is input by a user, the CPU 23 starts the ophthalmic image processing exemplified in Fig. 3.

[0084] As shown in FIG. 3, when the CPU 23 starts ophthalmic image processing, it acquires a plurality of types of ophthalmic images 30 (in this embodiment, a base image 30A and a reference image 30B) captured on the tissue of the same subject eye (S1).

[0085] In S1, the CPU 23 can also acquire multiple ophthalmic images 30 (multiple reference images 30B in this embodiment) that are of the same type but captured at different times. In this case, the CPU 23 also acquires information indicating the time when each of the multiple ophthalmic images 30 was captured.

[0086] Next, the CPU 23 acquires time and extent information (S2). As described above, the time information is information indicating the time from the time of capturing the ophthalmic image 30 acquired in S1 (when multiple ophthalmic images 30 captured at different times are acquired, the time of capturing the latest ophthalmic image 30) to the prediction time when the state of the disease of the subject's eye is predicted. The extent information is information indicating the extent of the disease of the subject's eye at the time of prediction. The CPU 23 may acquire the time and extent information by accepting an input of an instruction from a user to specify the time and extent information. The CPU 23 may also acquire predetermined time and extent information in S2. It is also possible to omit the process of S2.

[0087] In addition, in S2, the CPU 23 can also obtain a plurality of pieces of time information or a plurality of pieces of degree information that are different from each other. In this case, in S4 described later, a plurality of predicted disease images 60 with different prediction times or degrees are obtained.

[0088] Next, the CPU 23 acquires status information (S3). As described above, the status information is information indicating whether the status of the disease (lesion) in the multiple types of ophthalmic images 30 acquired in S1 will be aggravated or cured at the time of future prediction. The CPU 23 may acquire the status information by receiving an input of an instruction from a user to specify the status information. In addition, when multiple ophthalmic images 30 captured at different times are acquired in S1, the CPU 23 may determine whether the progression of the disease status is aggravated or cured from the multiple ophthalmic images 30 captured at different times, and acquire the determination result as the status information. In this case, the CPU 23 may determine the progression of the disease status by performing known image processing on the multiple ophthalmic images 30. In addition, the CPU 23 may acquire the status information by utilizing a mathematical model that inputs multiple ophthalmic images 30 and outputs the progression of the disease status. It is also possible to omit the processing of S3.

[0089] Next, the CPU 23 inputs at least the multiple types of ophthalmic images 30 (base image 30A and reference image 30B) acquired in S1 into a mathematical model trained by a machine learning algorithm, thereby acquiring a predicted disease image 60 output by the mathematical model (S4). As explained by way of example with reference to Fig. 2, in this embodiment, of the entire mathematical model, the base image 30A is input into the lesion removal model 71 and the lesion identification model 72, and the reference image 30B and time / degree information are input into the prediction model 73, and the predicted disease image 60 is output via the predicted disease image generation model 74.

[0090] Furthermore, when a plurality of ophthalmological images 30 (in this embodiment, a plurality of reference images 30B) taken at different times are acquired in S1, the CPU 23 inputs the plurality of ophthalmological images 30 to the prediction model 73. In this case, a predicted disease image 60 in which the progression of disease in the plurality of ophthalmological images 30 is taken into consideration is finally output by the predicted disease image generation model 74.

[0091] Furthermore, if multiple pieces of time / extent information have been acquired in S2, CPU 23 inputs the multiple pieces of time / extent information to prediction model 73. In this case, prediction model 73 outputs multiple predicted lesion condition images 81 (see FIG. 2) corresponding to each of the multiple pieces of time / extent information input. As a result, multiple predicted disease images 60 corresponding to each predicted lesion condition image 81 are output by predicted disease image generation model 74.

[0092] Moreover, when condition information is input in S3, the CPU 23 inputs the condition information to the prediction model 73. In this case, at least one of the predicted disease image 60 when the disease state worsens and the predicted disease image 60 when the disease state is cured is acquired based on a plurality of types of ophthalmologic images 30 that are actually captured.

[0093] Next, the CPU 23 displays the predicted disease image 60 acquired in S4 on the display device 28. Therefore, the user can appropriately perform diagnosis of the subject's eye by checking the predicted disease image 60 in which the disease state is more appropriately predicted.

[0094] Furthermore, when a plurality of predicted disease images 60 corresponding to each of a plurality of pieces of time / degree information are acquired in S4, the CPU 23 arranges or switches the plurality of predicted disease images 60 in order of time or degree and displays them on the display device 28. As a result, the user can more easily check the plurality of predicted disease images 60 with different prediction times or disease degrees.

[0095] The technology disclosed in the above embodiment is merely an example. Therefore, it is possible to change the technology exemplified in the above embodiment. For example, in the mathematical model of the above embodiment, as shown in FIG. 2, the lesion-removed image 40 output by the lesion identification model 71 is input to the predicted disease image generation model 74, and a predicted disease image 60 is output. However, instead of the lesion-removed image 40, the predicted disease image 60 may be generated based on the captured base image 30A itself. For example, the base image 30A may be input to the predicted disease image generation model 74, and the predicted disease image 60 may be output. In this case, the lesion-removed model 71 may be omitted.

[0096] In the above embodiment, among the entire mathematical models, the base image 30A is input to the lesion removal model 71 and the lesion identification model 72, and the reference image 30B is input to the prediction model 73, so that the predicted disease image 60 is output based on the base image 30A and the reference image 30B. However, it is also possible to change the configuration of the mathematical model to which the base image 30A and the reference image 30B are input. For example, both the base image 30A and the reference image 30B may be input to the lesion identification model 72. In this case, the lesion identification model 72 may output information indicating the position and range of the lesion (e.g., attention map 80, etc.) based on the input base image 30A and reference image 30B. Even in this case, the predicted disease image 60 in which the state of the disease is appropriately predicted is output based on multiple types of ophthalmological images 30 (base image 30A and reference image 30B).

[0097] In the above embodiment, a plurality of reference images 30B taken at different times are input to the mathematical model, and a predicted disease image 60 that takes into account the progression of the disease is output. However, even when a plurality of base images 30A taken at different times are input to the mathematical model, the progression of the disease is appropriately reflected in the predicted disease image 60.

[0098] In the above embodiment, a two-dimensional front image of the fundus is used as the base image 30A, and a two-dimensional tomographic image of the same fundus is used as the reference image 30B. However, as described above, it is also possible to change the base image and the reference image. For example, a two-dimensional tomographic image may be used as the base image, and a two-dimensional front image may be used as the reference image. In this case, the predicted disease image is a two-dimensional tomographic image.

[0099] It is also possible to adopt only a part of the techniques exemplified in the above embodiment. For example, the ophthalmological image processing device 21 can acquire the predicted disease image 60 based on the time / degree information and one type of ophthalmological image 30, without executing the process of acquiring the predicted disease image 60 based on multiple types of ophthalmological images 30. In addition, the ophthalmological image processing device 21 can acquire the predicted disease image 60 based on multiple ophthalmological images 30 of the same type taken at different times, without executing the process of acquiring the predicted disease image 60 based on multiple types of ophthalmological images 30.

[0100] In the above embodiment, the predicted disease image 60 is obtained based on multiple types of ophthalmological images 30 obtained by photographing tissues of the same subject (i.e., tissues of the same subject photographed at similar times). However, the predicted disease image 60 may be obtained based on multiple ophthalmological images 30 obtained by photographing each of multiple tissues with different disease states (e.g., multiple ophthalmological images 30 obtained by photographing tissues of the same subject at different times, or multiple ophthalmological images 30 obtained by photographing each of tissues of multiple subjects with different degrees of disease, etc.).

[0101] For example, in the example shown in FIG. 4, a first ophthalmological image 30X (in the example shown in FIG. 4, an image of a normal eye or an eye with a low degree of disease) of the subject to be diagnosed is input to the lesion removal model 71, and a lesion removal image 40X is output. In addition, a second ophthalmological image 30Y of another subject with a worsening degree of disease is input to the lesion identification model 72, and an attention map 80Y indicating the position and range of the lesion and information on the type of disease are output. The lesion removal image 40X, the type of disease, and information indicating the predicted state of the lesion (attention map 80Y in FIG. 4) are input to the predicted disease image generation model 74, and a predicted disease image 60Z is output. As a result, the predicted disease image 60Z is an image in which the lesion in the first ophthalmological image 30X is modified to the same degree as the lesion in the second ophthalmological image 30Y. The first ophthalmological image 30X may be directly input to the predicted disease image generation model 74 instead of the lesion removal image 40X.

[0102] The ophthalmic image processing device exemplified in Fig. 4 can also be expressed as follows: An ophthalmic image processing device processes ophthalmic images, which are images of tissues of a subject's eye, and a control unit of the ophthalmic image processing device executes an image acquisition step of acquiring a first ophthalmic image and a second ophthalmic image having different degrees of disease, and a predicted disease image acquisition step of acquiring a predicted disease image in which a diseased area in the first ophthalmic image is modified to the same degree as that in the second ophthalmic image by inputting the first ophthalmic image and the second ophthalmic image into a mathematical model trained by a machine learning algorithm.

[0103] The process of acquiring multiple types of ophthalmologic images 30 in S1 of FIG. 3 is an example of an "image acquiring step." The process of acquiring a predicted disease image 60 in S4 of FIG. 3 is an example of a "predicted disease image acquiring step." The process of acquiring time / degree information in S2 of FIG. 3 is an example of a "prediction time information acquiring step." The process of displaying the predicted disease image 60 on the display device 28 in S5 of FIG. 3 is an example of a "display step." The process of acquiring status information in S3 of FIG. 3 is an example of a "status information acquiring step." [Explanation of symbols]

[0104] 1. Mathematical model construction device 11 Imaging Device 21 Ophthalmological image processing device 23 CPU 24 Storage device 28 Display device 30 Ophthalmology Images 60 Predictive Disease Imaging 71 Lesion removal model 72 Lesion Identification Model 73 Predictive Models 74 Predictive disease image generation model

Claims

1. An ophthalmic image processing device that processes an ophthalmic image which is an image of tissue of a subject's eye, The control unit of the ophthalmologic image processing device An image acquisition step of acquiring a plurality of types of ophthalmologic images captured of tissue of the same subject's eye; a predicted disease image acquisition step of inputting the plurality of types of ophthalmic images into a mathematical model trained by a machine learning algorithm to acquire a predicted disease image, which is an image in which a predicted future lesion is modified for one of the plurality of types of ophthalmic images input; Run The multiple types of ophthalmologic images input to the mathematical model in the predicted disease image acquisition step include a base image, which is a two-dimensional front image that is a basis for converting into the predicted disease image, and a reference image, which is a two-dimensional tomographic image of a subject that is common to the subject of the base image; The predicted disease image acquisition step includes a step of acquiring information on the position and range of a lesion when a lesion is present in the base image, The mathematical model includes: a prediction model that outputs a predicted lesion state image showing the predicted future position and range of the lesion by inputting at least one of information on the position and range of the lesion in the base image, a reference image, time information showing the time from the capture of the base image and the reference image to the prediction of the disease, and extent information showing the extent of the disease at the time of the prediction; a predictive disease image generation model that receives the predicted lesion state image output by the predictive model and outputs the predicted disease image in which the lesion state image is modified from the base image; Contains The prediction model is trained using, as input training data, information on the position and range of a lesion, a two-dimensional tomographic image of the same type as a reference image, and at least one of time information and degree information, and using, as output training data, information on the position and range of a lesion in a two-dimensional front image captured of the same subject after the two-dimensional tomographic image used as the input training data; The disease image generation model is trained using a two-dimensional front image as output training data and information on the position and range of a lesion related to the output training data as input training data; The mathematical model is an ophthalmologic image processing device characterized in that it outputs the predicted disease image in which the diseased area has been modified from the input base image by inputting the base image, the reference image, and at least one of time information and severity information.

2. 2. The ophthalmologic image processing device according to claim 1, The control unit is Acquire a plurality of pieces of time information or a plurality of pieces of degree information that are different from each other, In the predicted disease image acquisition step, each of the plurality of pieces of time information or the plurality of pieces of degree information is input to the mathematical model together with the plurality of types of ophthalmic images to acquire a plurality of predicted disease images having different times or degrees; An ophthalmologic image processing device characterized by further executing a display step of displaying on a display device the multiple predicted disease images acquired in the predicted disease image acquisition step in an orderly or switched manner along the time or severity.

3. 3. The ophthalmologic image processing device according to claim 1, The control unit is In the image acquisition step, a plurality of ophthalmologic images of the same type taken at different times are further acquired; An ophthalmologic image processing apparatus, characterized in that, in the predicted disease image acquisition step, the predicted disease image is acquired by inputting the plurality of ophthalmologic images captured at different times into the mathematical model.

4. 4. An ophthalmologic image processing apparatus according to claim 1, The control unit is further performing a state information acquiring step of acquiring state information indicating whether the state of the disease in the plurality of types of ophthalmologic images acquired in the image acquiring step is to be worsened or cured at the time of future prediction; An ophthalmologic image processing apparatus, characterized in that in the predicted disease image obtaining step, the predicted disease image is obtained by inputting the condition information together with the plurality of types of ophthalmologic images into the mathematical model.

5. 5. An ophthalmologic image processing apparatus according to claim 1, The ophthalmologic image processing apparatus according to claim 1, wherein the plurality of types of ophthalmologic images acquired in the image acquisition step include a plurality of types of ophthalmologic images captured by different types of imaging devices.

6. An ophthalmic image processing program executed by an ophthalmic image processing device for processing an ophthalmic image which is an image of tissue of a subject's eye, The ophthalmologic image processing program is executed by a control unit of the ophthalmologic image processing device, An image acquisition step of acquiring a plurality of types of ophthalmologic images captured of tissue of the same subject's eye; a predicted disease image acquisition step of inputting the plurality of types of ophthalmic images into a mathematical model trained by a machine learning algorithm to acquire a predicted disease image, which is an image in which a predicted future lesion is modified for one of the plurality of types of ophthalmic images input; and causing the ophthalmologic image processing device to execute the above steps; The multiple types of ophthalmologic images input to the mathematical model in the predicted disease image acquisition step include a base image which is a two-dimensional front image that is a basis for converting into the predicted disease image, and a reference image which is a two-dimensional tomographic image of a subject that is common to the subject of the base image, The predicted disease image acquisition step includes a step of acquiring information on the position and range of a lesion when a lesion is present in the base image, The mathematical model includes: a prediction model that outputs a predicted lesion state image showing the predicted future position and range of the lesion by inputting at least one of information on the position and range of the lesion in the base image, a reference image, time information showing the time from the capture of the base image and the reference image to the prediction of the disease, and extent information showing the extent of the disease at the time of the prediction; a predictive disease image generation model that receives the predicted lesion state image output by the predictive model and outputs the predicted disease image in which the lesion state image is modified from the base image; Contains The prediction model is trained using, as input training data, information on the position and range of a lesion, a two-dimensional tomographic image of the same type as a reference image, and at least one of time information and degree information, and using, as output training data, information on the position and range of a lesion in a two-dimensional front image captured of the same subject after the two-dimensional tomographic image used as the input training data; The disease image generation model is trained using a two-dimensional front image as output training data and information on the position and range of a lesion related to the output training data as input training data; The mathematical model is an ophthalmologic image processing program characterized in that it outputs the predicted disease image in which the diseased area has been modified from the input base image by inputting the base image, the reference image, and at least one of time information and severity information.

Images