Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program that generate a learned model for estimating a region of abnormal blood circulation in a medical image with less learning data and with high accuracy.SOLUTION: In an information processing system in which a learning device is connected to an inference device and a storage device via a communication network, the learning device 10 includes: a learning unit 124 for subjecting a machine learning model to learning by inputting, to the machine learning model, a medical image to which annotation information about a region of abnormal blood circulation has been appended, and a blood vessel image obtained by estimating a blood vessel region in the medical image from the medical image; and a model output unit 126 for outputting a learned model that has been learned by the learning unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In retinal diseases such as diabetic retinopathy, where vascular disorders lead to an ischemic state, it is essential to understand the dynamics of retinal circulation. The dynamics of retinal circulation can be understood by performing fluorescein fundus angiography. While fluorescein fundus angiography is an examination that can provide useful information, it also places a certain burden on patients and medical professionals, as there is a possibility of side effects from the contrast agent. Therefore, a method has been proposed to identify abnormalities in circulation from fundus images without performing fluorescein fundus angiography. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 142910 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent document 1 discloses a diagnostic support device that identifies areas of abnormal blood circulation in a fundus image using a trained model that has learned the relationship between the fundus image and areas of abnormal blood circulation in the fundus image, based on a fundus image, which is an image of the fundus, and areas of abnormal blood circulation identified based on a fluorescent fundus angiography image of the fundus.

[0005] In order to generate such a trained model, annotation is performed to add tags indicating regions of abnormal blood circulation, and a large amount of fundus images with this information (hereinafter referred to as annotation information) is required. A non-perfusion area (NPA) (hereinafter also referred to as the "NPA area"), which is an example of a region of abnormal blood circulation, is a region in the fundus where no or almost no blood flows due to blockage of the retinal capillary bed. A fundus image with annotation information of the NPA area is created as follows. First, a set of a fundus image and a fluorescein angiography image taken of the same patient in the same examination is prepared. Next, an ophthalmologist or the like annotates the NPA region on the fluorescein angiography image while checking both images. Finally, the annotation information is added to the fundus image. A similar process is also required to add annotation information to a fundus image for a region where neovascularization has occurred, which is another example of a region of abnormal blood circulation. It takes a lot of time and effort to collect such training data.

[0006] In addition, unlike conventional semantic segmentation, annotation of NPA regions requires identifying regions with unclear boundaries in fluorescein angiograms, which makes it even more difficult to collect training data.

[0007] Therefore, an object of a first aspect of the present invention is to provide an information processing device, an information processing method, and a program capable of generating a trained model for inferring an abnormal blood circulation region in a medical image acquired by photographing a patient's examination target part with less training data.An object of a second aspect of the present invention is to provide an information processing device, an information processing method, and a program capable of inferring an abnormal blood circulation region in a medical image with higher accuracy than the conventional technology. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present invention includes a learning unit that inputs a medical image to which annotation information of an abnormal blood circulation area has been added and a vascular image in which a vascular area in the medical image has been estimated from the medical image into a machine learning model to train the machine learning model, and a model output unit that outputs the trained model trained by the learning unit.

[0009] According to this aspect, by using vascular images derived from medical images to train the machine learning model, the machine learning model can be trained more efficiently than when training is performed using medical images alone, and as a result, learning can converge with less training data.

[0010] The information processing device may further include a learning data acquisition unit that acquires medical images to which annotation information has been added so that a predetermined ratio of medical images including regions of abnormal blood circulation and medical images not including regions of abnormal blood circulation is achieved. According to this aspect, it is possible to avoid a situation that is detrimental to learning, in which medical images not including regions of abnormal blood circulation are significantly more numerous in the learning data than medical images including regions of abnormal blood circulation.

[0011] In the information processing device, the learning unit may train the machine learning model using a loss function that dynamically attenuates the weight of the cross entropy loss of an area where inference is easy. According to this aspect, the learning of the machine learning model can be prevented from being dominated by the learning of an area where inference is easy and does not include an area where blood circulation is abnormal, and it is possible to effectively learn an area where inference is difficult and includes an area where blood circulation is abnormal.

[0012] In the information processing device, the abnormal blood circulation region may include a non-perfusion region and a region where neovascularization has occurred, and the learning unit may train the machine learning model based at least on an ordinal scale of the region without abnormality, the non-perfusion region, and the region where neovascularization has occurred. According to this aspect, the machine learning model can efficiently learn the ordinal relationship of the region without abnormality, the NPA region, and the region where neovascularization has occurred, and as a result, it is expected that the learning can be converged with less training data.

[0013] In the above information processing device, the learning unit may train the machine learning model using a loss function that takes into account classification errors between regions in the order of regions without abnormalities, non-perfusion regions, and regions with neovascularization.

[0014] In the information processing device, the learning unit may train the machine learning model using learning data to which a probability distribution that assigns probabilities to other areas related to the correct answer data is added as annotation information.

[0015] According to another aspect of the present invention, a method for generating a trained model includes acquiring a medical image to which annotation information of an abnormal blood circulation area is added, acquiring a vascular image in which a vascular area in the medical image is estimated from the medical image, inputting the medical image and the vascular image into a machine learning model to train the machine learning model, and outputting the trained model obtained as a result of the training.

[0016] An information processing device according to another aspect of the present invention includes a first acquisition unit that acquires a first image including a medical image, a second acquisition unit that acquires a second image from the first image, the second image indicating a vascular region in the first image, an inference unit that infers an abnormal blood circulation region in the first image based on the first image and the second image, and an output unit that outputs the result inferred by the inference unit.

[0017] According to this aspect, by using a second image derived from a first image including the medical image that is the subject of inference, which shows the vascular region in the first image, to infer the region of abnormal blood circulation, it is possible to perform inference with higher accuracy than when inference is performed using only the medical image.

[0018] In the information processing device, the second acquisition unit acquires the second image by inputting the first image to a first trained model, and the first trained model may be a trained model trained to estimate a blood vessel region from a medical image. According to this aspect, the second image having a desired accuracy can be easily acquired.

[0019] In the above information processing device, the inference unit infers an abnormal blood circulation region in the first image by inputting the first image and the second image to a second trained model, and the second trained model may be a trained model trained to estimate an abnormal blood circulation region in the medical image from a medical image and a blood vessel image showing a blood vessel region in the medical image, which is acquired by inputting the medical image to the first trained model. According to this aspect, an inference result with a desired accuracy can be obtained for the abnormal blood circulation region in the first image.

[0020] In the information processing device, the second trained model may be a neural network having a large stride convolutional layer. According to this aspect, the second trained model can estimate an abnormal blood circulation region in a medical image by using information on a blood vessel region as information on a global range.

[0021] The information processing device may further include a classification unit that classifies the first image into an image that may include an abnormal blood circulation region and an image other than the image, and the inference unit processes only the image that may include the abnormal blood circulation region. According to this aspect, it is possible to efficiently perform inference with high accuracy.

[0022] In the information processing device, the abnormal blood circulation region may include at least one of a non-perfusion region and a region where neovascularization has occurred. According to this aspect, it is possible to perform inference regarding one or both of a non-perfusion region and a region where neovascularization has occurred in a medical image.

[0023] A method according to another aspect of the present invention includes acquiring a first image including a medical image, acquiring from the first image a second image showing a vascular region in the first image, inferring an area of ​​abnormal blood circulation in the first image based on the first image and the second image, and outputting the inferred result.

[0024] A program according to another aspect of the present invention causes one or more computers to execute the following processes: acquiring a first image including a medical image; acquiring a second image from the first image, the second image indicating a vascular region in the first image; inferring an area of ​​abnormal blood circulation in the first image based on the first image and the second image; and outputting the inferred result. Effect of the Invention

[0025] According to a first aspect of the present invention, it is possible to provide an information processing device, an information processing method, and a program capable of generating a trained model for inferring regions of abnormal blood circulation in medical images with less training data. Also, according to a second aspect of the present invention, it is possible to provide an information processing device, an information processing method, and a program capable of inferring regions of abnormal blood circulation in medical images with higher accuracy than the conventional technology. [Brief description of the drawings]

[0026] [Figure 1] 1 is a diagram showing a network configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic diagram illustrating the processing of a learning device and the processing of an inference device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a block diagram of a learning device according to an embodiment of the present invention. [Figure 4] 1 is a block diagram of an inference device according to an embodiment of the present invention; [Diagram 5] 4 is a flowchart showing a learning process of the learning device according to one embodiment of the present invention. [Figure 6] 4 is a flowchart showing an inference process of the inference device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] With reference to the accompanying drawings, the embodiments of the present invention will be described. Note that the following embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. In addition, the present invention can be modified in various ways without departing from the gist of the invention. Furthermore, a person skilled in the art can adopt an embodiment in which each element described below is replaced with an equivalent, and such an embodiment is also included in the scope of the present invention.

[0028] (System Configuration) An overview of the present invention will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing a network configuration of an information processing system according to an embodiment of the present invention. Figure 2 is a schematic diagram explaining the processing of a learning device and the processing of an inference device according to an embodiment of the present invention.

[0029] The information processing system 1 includes a learning device 10, an inference device 20, and a storage device 30. The learning device 10 is connected to the inference device 20 and the storage device 30 via a communication network N. The communication network N may be either a wired communication network or a wireless communication network formed by wired or wireless lines, and may be the Internet or a Local Area Network (LAN).

[0030] The learning device 10 learns a machine learning model based on the learning data stored in the storage device 30, and stores the learned model in the storage device 30. The learning device 10 according to this embodiment includes a machine learning model, but the machine learning model may be included in a device separate from the learning device 10.

[0031] Here, the machine learning model is a model that has a predetermined model structure and processing parameters that vary through a learning process, and the processing parameters are optimized based on experience obtained from the learning data, thereby improving the discrimination accuracy. That is, the machine learning model is a model that learns optimal processing parameters through a learning process. The algorithm of the machine learning model may be, for example, a support vector machine, a logistic regression, a neural network, or the like, but the type is not particularly limited. The machine learning model that performs the learning may be one that has already undergone some kind of learning using the learning data, or one that has not yet been learned.

[0032] A trained model is a model that has been trained in advance using appropriate training data for a machine learning model based on an arbitrary machine learning algorithm. However, a trained model is not limited to a model that does not undergo further training, and can also undergo additional training.

[0033] The inference device 20 uses the trained model to output output data according to the characteristics of the input data. The inference device 20 according to this embodiment performs inference using the trained model acquired from the storage device 30. Here, acquiring the trained model means acquiring information necessary to reproduce the function of the trained model in the inference device 20. For example, when a neural network is used as the machine learning model, acquiring the trained model means acquiring information on at least the number of layers of the neural network, the number of nodes for each layer, weight parameters of links connecting the nodes, bias parameters for each node, and the functional form of the activation function for each node.

[0034] The storage device 30 stores learning data used for learning the machine learning model. The storage device 30 according to the present embodiment stores, as learning data, fundus images to which annotation information of the NPA region is added. The storage device 30 also stores a learned model output by the learning device 10. In FIG. 1, the storage device 30 is illustrated as a single storage device, but the storage device 30 may be configured with one or more file servers. In the present embodiment, the fundus image to which annotation information of the NPA region is added is used as learning data as an example of a blood circulation abnormality region. In another embodiment, however, a fundus image to which annotation information of other blood circulation abnormality regions, such as a region where neovascularization has occurred, is added can be used as learning data, or a fundus image to which annotation information of both the NPA region and the region where neovascularization has occurred can be used as learning data.

[0035] Here, as shown in FIG. 2, the learning device 10 according to this embodiment uses an auxiliary model (first trained model) when training the machine learning model. The auxiliary model of this embodiment is a trained model that receives a fundus image, which is training data for the machine learning model, as input data, and outputs a blood vessel image that estimates a blood vessel region in the fundus image. When training the machine learning model, the learning device 10 uses, as input data, a plurality of blood vessel images output from the auxiliary model in addition to a plurality of fundus images to which annotation information has been added. In this embodiment, an example using a fundus image, which is an example of a medical image, will be described, but in another embodiment, an image of the brain or myocardium acquired by photographing another part of the patient to be examined can be used.

[0036] By using blood vessel images derived from fundus images for training the machine learning model, the machine learning model can be trained more efficiently than when training is performed using only fundus images with annotation information, and as a result, training can be converged with less training data. The training data for the auxiliary model is an image with annotation information for blood vessel regions with clear boundaries. Therefore, the training data for the auxiliary model can be collected relatively easily.

[0037] Similarly, the inference device 20 according to the present embodiment uses an auxiliary model (first learned model) when performing inference using a trained model (second learned model). In addition to a fundus image (first image) to be inferred, the inference device 20 uses a blood vessel image (second image) output from the auxiliary model using the fundus image as input data as input data for a trained model that infers an NPA region. By using a blood vessel image derived from a fundus image to infer an NPA region, it is possible to perform inference with higher accuracy than when inference is performed using only a fundus image. In the present embodiment, an NPA region is inferred as an example of an abnormal blood circulation region, but in another embodiment, other abnormal blood circulation regions, such as a region where neovascularization has occurred, may be inferred, or both an NPA region and a region where neovascularization has occurred may be inferred.

[0038] However, it is difficult to directly infer the NPA region from a trained model that estimates the vascular region in a fundus image. This is because inference of the blood vessel position mainly uses local information, whereas inference of the NPA region requires information from a wider area.

[0039] (Functional configuration: learning device) Fig. 3 is a block diagram of a learning device according to an embodiment of the present invention. Note that Fig. 3 assumes a single learning device 10 and shows only the necessary functional configuration, but the learning device 10 can also be configured as a part of a multi-functional distributed system consisting of multiple computer systems.

[0040] The learning device 10 includes an input unit 110, a control unit 120, a storage unit 130, and a communication unit 140.

[0041] The input unit 110 is configured to accept operations from an administrator of the study device 10, and can be realized by a keyboard, a mouse, a touch panel, or the like.

[0042] The control unit 120 includes an arithmetic processing unit 121 such as a CPU or MPU equivalent to a processor, and a memory 122 such as a RAM. The arithmetic processing unit 121 (processor) loads a program recorded in the storage unit 130 into the memory 122 and executes it based on various inputs, thereby realizing the functions and processing described below in the arithmetic processing unit 121. This program may be stored in a computer-readable non-transitory recording medium such as a CD-ROM, or distributed via a network and installed in the computer. The memory 122 functions as a work memory required for the arithmetic processing unit 121 (processor) to execute the program.

[0043] The storage unit 130 is configured with a storage device such as a hard disk, and records various programs necessary for executing processes in the control unit 120, data necessary for executing the various programs, etc. In this embodiment, it is desirable for the storage unit 130 to have a learning data storage unit 131 and an auxiliary model 132.

[0044] The learning data storage unit 131 stores learning data used for learning the machine learning model 125 described later. In this embodiment, the learning data storage unit 131 stores a fundus image to which annotation information of the NPA region is added.

[0045] The auxiliary model 132 stores a trained model used to assist in learning of the machine learning model 125. In this embodiment, the auxiliary model 132 stores a trained model that receives a fundus image as input data and outputs a blood vessel image in which a blood vessel region in the fundus image is estimated. For example, in one embodiment, a trained model trained using a fundus image to which annotation information of a blood vessel region is added may be used as the auxiliary model 132.

[0046] The communication unit 140 is configured to connect the learning device 10 to a network. For example, the communication unit 140 can be realized by a LAN card, an analog modem, an ISDN modem, or the like, and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0047] Furthermore, as shown in FIG. 3, the arithmetic processing unit 121 includes a learning data acquisition unit 123, a learning unit 124, a machine learning model 125, and a model output unit 126 as functional units.

[0048] The learning data acquisition unit 123 acquires learning data used for learning the machine learning model 125 described later, and stores it in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires a fundus image to which annotation information of the NPA region is added from the storage device 30, and stores it in the learning data storage unit 131. In one embodiment, the learning data acquisition unit 123 acquires learning data including a fundus image including an NPA region and a fundus image not including an NPA region from the storage device 30. The learning data acquisition unit 123 may acquire learning data such that the fundus image including the NPA region and the fundus image not including the NPA region have a predetermined ratio, for example, 1:1. In this way, it is possible to avoid a situation that is disadvantageous to learning, in which the number of fundus images not including an NPA region in the learning data is significantly greater than the number of fundus images including an NPA region.

[0049] The learning data acquisition unit 123 also acquires a blood vessel image indicating a blood vessel region in the fundus image from the fundus image, and stores the blood vessel image in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 inputs the fundus image to the auxiliary model 132 to acquire the blood vessel image, and stores the blood vessel image in the learning data storage unit 131.

[0050] The learning unit 124 trains the machine learning model 125 using the learning data acquired by the learning data acquisition unit 123. In the present embodiment, the learning unit 124 inputs a fundus image and a blood vessel image derived from the fundus image to the machine learning model 125, and trains the machine learning model 125.

[0051] The machine learning model 125 receives, as input data, a fundus image to which annotation information of the NPA region is added and a blood vessel image derived from the fundus image, and outputs information indicating the NPA region in the fundus image. In this embodiment, an example using a neural network as an example of the machine learning model 125 will be described. However, the neural network is merely an example of the machine learning model 125, and the learning device 10 may use other configurations as the machine learning model 125.

[0052] In one embodiment, the machine learning model 125 has a convolution layer with a large stride compared to the initial setting of the machine learning model 125. In this way, the machine learning model 125 can learn information about the vascular region as information of a global range.

[0053] In one embodiment, the learning unit 124 may train the machine learning model 125 using a loss function that dynamically attenuates the weight of the cross entropy loss in an easy-to-infer region. In this way, the learning of the machine learning model 125 can be prevented from being dominated by the learning of an easy-to-infer region that does not include an NPA region, and the region that is difficult to infer and includes an NPA region can be effectively trained.

[0054] In addition, when a fundus image to which annotation information is added for both the NPA region and the neovascularized region is used as training data, the training unit 124 may train the machine learning model based at least on the order scale of the region without abnormality, the nonperfusion region, and the region with neovascularization. In one embodiment, the training unit 124 may train the machine learning model 125 using a loss function that takes into account the error in classification between regions having the order of the region without abnormality, the NPA region, and the region with neovascularization, utilizing the characteristic that the region with neovascularization is a region that has progressed from the NPA region. For example, the training unit 124 may train the machine learning model 125 using a loss function that reduces the error in inferring the NPA region from the fundus image to which annotation information for the region with neovascularization is added compared to inferring the region without abnormality. In this way, the machine learning model can efficiently learn the order relationship between the region without abnormality, the NPA region, and the region with neovascularization, and as a result, it is expected that the learning will converge with less training data.

[0055] In addition, when a fundus image to which annotation information is added for both the NPA region and the region where neovascularization has occurred is used as learning data, in one embodiment, the learning unit 124 may use the learning data to which annotation information is added that considers the relationship between regions having the order of the region without abnormality, the NPA region, and the region where neovascularization has occurred, taking advantage of the characteristic that the region where neovascularization has occurred is a region where the NPA region has progressed, to train the machine learning model 125. For example, instead of assigning a probability distribution in which the probability of each of the region without abnormality, the NPA region, and the region where neovascularization has occurred is 0, 0, and 1 with only the region where neovascularization has occurred as correct answer data, the learning unit 124 may use learning data to which a probability distribution in which the probability of each of the region without abnormality, the NPA region, and the region where neovascularization has occurred is 0, 0.1, and 0.9 is assigned to other regions related to the correct answer data as annotation information for the region where neovascularization has occurred. By doing this, the machine learning model can efficiently learn the sequential relationship between areas without abnormalities, NPA areas, and areas with neovascularization, which is expected to enable learning to converge with less training data.

[0056] When the learning of the machine learning model 125 is completed, the model output unit 126 outputs the learned model to the storage device 30. Note that the learning unit 124 may complete the learning after training the machine learning model 125 using a predetermined number of training data, for example, or may complete the learning when the accuracy of the machine learning model 125 satisfies a predetermined condition.

[0057] (Functional configuration: Inference device) Fig. 4 is a block diagram of an inference device according to an embodiment of the present invention. Note that Fig. 4 assumes a single inference device 20 and shows only the necessary functional configuration, but the inference device 20 can also be configured as a part of a multi-functional distributed system consisting of multiple computer systems.

[0058] The inference device 20 includes an input unit 210, a control unit 220, a memory unit 230, and a communication unit 240.

[0059] The input unit 210 is configured to accept operations from an administrator of the inference device 20, and can be realized by a keyboard, a mouse, a touch panel, or the like.

[0060] The control unit 220 includes an arithmetic processing unit 221 such as a CPU or MPU equivalent to a processor, and a memory 222 such as a RAM. The arithmetic processing unit 221 (processor) implements functions and processing described below in the arithmetic processing unit 221 by expanding a program recorded in the storage unit 230 into the memory 222 and executing it based on various inputs. This program may be stored in a computer-readable non-transitory recording medium such as a CD-ROM, or distributed via a network and installed in the computer. The memory 222 functions as a work memory required for the arithmetic processing unit 221 (processor) to execute the program.

[0061] The storage unit 230 is configured with a storage device such as a hard disk, and records various programs necessary for executing processes in the control unit 220, data necessary for executing the various programs, and the like. In this embodiment, it is desirable that the storage unit 230 has an image storage unit 231, an auxiliary model 232, and a trained model 233. In one embodiment, the storage unit 230 may further have a classification model 234.

[0062] An image to be inferred is stored in the image storage unit 231. In this embodiment, the image storage unit 231 stores a fundus image for inferring an NPA region.

[0063] A trained model used to assist inference is stored in the auxiliary model 232. In this embodiment, a trained model that receives a fundus image as input data and outputs a blood vessel image in which a blood vessel region in the fundus image is estimated is stored in the auxiliary model 232. For example, in one embodiment, a trained model trained using a fundus image to which annotation information of a blood vessel region is added may be used as the auxiliary model 232.

[0064] A trained model used for inference is stored in the trained model 233. In this embodiment, a trained model that receives a fundus image and a blood vessel image derived from the fundus image as input data and infers an NPA region in the fundus image is stored in the trained model 233. That is, the trained model 233 is a trained model trained using training data including a fundus image to which annotation information of the NPA region is added and a blood vessel image derived from the fundus image.

[0065] The classification model 234 stores a trained model used to classify input images. In this embodiment, the classification model 234 stores a trained model that receives a fundus image as input data and classifies the image into an image that may include an NPA region and an image other than the NPA region. That is, the classification model 234 is a trained model trained using a fundus image on which the presence or absence of an NPA region is annotated. Retinal diseases include diseases in which an NPA region may appear and diseases unrelated to an NPA region. By utilizing such characteristics of retinal diseases, it is possible to easily collect training data for the classification model 234.

[0066] The communication unit 240 is configured to connect the inference device 20 to a network. For example, the communication unit 240 can be realized by a LAN card, an analog modem, an ISDN modem, or the like, and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0067] Furthermore, as shown in FIG. 4, the calculation processing unit 221 includes a model acquisition unit 223, a first acquisition unit 224, a second acquisition unit 225, an inference unit 226, a classification unit 227, and an output unit 228 as functional units.

[0068] The model acquisition unit 223 acquires a trained model to be used for inference, and stores it in the trained model 233. In this embodiment, the model acquisition unit 223 acquires a trained model from the storage device 30, and stores it in the trained model 233.

[0069] The first acquisition unit 224 acquires an image to be inferred. In this embodiment, the first acquisition unit 224 acquires, from the image storage unit 231, a fundus image for inferring an NPA region.

[0070] The second acquisition unit 225 acquires a blood vessel image indicating a blood vessel region in the image acquired by the first acquisition unit 224. In this embodiment, the second acquisition unit 225 acquires a blood vessel image indicating a blood vessel region in the fundus image by inputting the fundus image to the auxiliary model 232. As described above, the auxiliary model 232 in this embodiment is a trained model trained to estimate a blood vessel region from an image of the fundus.

[0071] The inference unit 226 infers an NPA region in the fundus image based on the image acquired by the first acquisition unit 224 and the image acquired by the second acquisition unit 225. In this embodiment, the inference unit 226 inputs the fundus image acquired by the first acquisition unit 224 and a blood vessel image derived from the fundus image to the trained model 233, and acquires information indicating the NPA region in the fundus image.

[0072] In one embodiment, the inference device 20 may use a classification unit 227 as pre-processing for inference. The classification unit 227 classifies fundus images into images that may include an NPA region and other images. The classification unit 227 may classify fundus images using a classification model 234. By performing such pre-processing, it is possible to infer an NPA region only from images that may include an NPA region, and it is possible to efficiently perform highly accurate inference.

[0073] The output unit 228 outputs an inference result based on the information acquired by the inference unit 226. In this embodiment, the output unit 228 outputs an inference result based on information indicating an NPA region in a fundus image acquired by the inference unit 226. In one embodiment, the output unit 228 may output a blood vessel image used for the inference in addition to information indicating an NPA region in a fundus image.

[0074] (Learning process) The learning process of the learning device according to the embodiment of the present invention will be described in detail with reference to Fig. 5. In this embodiment, it is assumed that learning data is stored in the storage device 30 under the management of an administrator of the learning device 10 before the learning process described in Fig. 5 is performed. Note that the process shown in Fig. 5 is executed, for example, by the administrator inputting an instruction to execute a process of generating a trained model via the input unit 110.

[0075] In step S501, the learning data acquisition unit 123 of the learning device 10 acquires learning data used for learning the machine learning model 125, and stores it in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires a plurality of fundus images to which annotation information of NPA regions is added from the storage device 30, and stores them in the learning data storage unit 131. Here, the learning data acquisition unit 123 acquires fundus images including NPA regions and fundus images not including NPA regions in a 1:1 ratio from the storage device 30. In this way, it is possible to avoid a situation that is detrimental to learning, in which fundus images not including NPA regions are significantly more numerous in the learning data than fundus images including NPA regions.

[0076] Next, in step S502, the learning data acquisition unit 123 acquires a blood vessel image indicating a blood vessel region in the fundus image from the fundus image, and stores the blood vessel image in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 inputs the fundus image to the auxiliary model 132 to acquire the blood vessel image, and stores the blood vessel image in the learning data storage unit 131.

[0077] Next, in step S503, the learning unit 124 of the learning device 10 trains the machine learning model 125 by using the learning data acquired by the learning data acquisition unit 123. In this embodiment, the learning unit 124 inputs a fundus image to which annotation information has been added and a blood vessel image derived from the fundus image to the machine learning model 125, and trains the machine learning model 125.

[0078] In this embodiment, a neural network is used as an example of the machine learning model 125. In addition, in this embodiment, the machine learning model 125 has a convolution layer with a large stride compared to the initial setting of the machine learning model 125. In this way, the machine learning model 125 can learn information about a blood vessel region as information of a global range.

[0079] Furthermore, in this embodiment, the learning unit 124 trains the machine learning model 125 using a loss function that dynamically attenuates the weight of the cross entropy loss in an area where inference is easy. In this way, the learning of the machine learning model 125 can be prevented from being dominated by the learning of an area where inference is easy and does not include an NPA area, and an area where inference is difficult and includes an NPA area can be effectively trained.

[0080] When the learning of the machine learning model 125 is completed, in step S504, the model output unit 126 of the learning device 10 outputs the learned model to the storage device 30. Note that the learning unit 124 may complete the learning after training the machine learning model 125 using a predetermined number of pieces of training data, for example, or may complete the learning when the accuracy of the machine learning model 125 satisfies a predetermined condition.

[0081] (Inference processing) With reference to Fig. 6, the inference process of the inference device according to the embodiment of the present invention will be described in detail. In this embodiment, it is assumed that, before the inference process described in Fig. 6 is performed, a trained model acquired from the storage device 30 is stored in the trained model 233 under the management of an administrator of the inference device 20. It is also assumed that a fundus image to be inferred is stored in the image storage unit 231 of the inference device 20. The process shown in Fig. 6 is executed, for example, by the administrator inputting an instruction to execute the inference process via the input unit 210.

[0082] In step S601, the first acquisition unit 224 of the inference device 20 acquires an image to be inferred. In this embodiment, the first acquisition unit 224 acquires from the image storage unit 231 a fundus image for inferring an NPA region.

[0083] In step S602, the classification unit 227 of the inference device 20 classifies the fundus images acquired by the first acquisition unit 224 into images that may include an NPA region and other images. Here, the classification unit 227 classifies the fundus images using the classification model 234. By performing such preprocessing, it is possible to infer the NPA region only from images that may include an NPA region, and it is possible to efficiently perform highly accurate inference.

[0084] In step S603, for an image that may include an NPA region, the second acquisition unit 225 of the inference device 20 acquires a blood vessel image indicating a blood vessel region in the image acquired by the first acquisition unit 224. In this embodiment, the second acquisition unit 225 acquires a blood vessel image indicating a blood vessel region in the fundus image by inputting the fundus image acquired by the first acquisition unit 224 to the auxiliary model 232. As described above, the auxiliary model 232 in this embodiment is a trained model trained to estimate a blood vessel region from an image of a fundus.

[0085] In step S604, the inference unit 226 of the inference device 20 infers an NPA region in the fundus image based on the image acquired by the first acquisition unit 224 and the image acquired by the second acquisition unit 225. In this embodiment, the inference unit 226 inputs the fundus image acquired by the first acquisition unit 224 and a blood vessel image derived from the fundus image to the trained model 233, and acquires information indicating the NPA region in the fundus image.

[0086] In step S605, the output unit 228 of the inference device 20 outputs an inference result based on the information acquired by the inference unit 226. In this embodiment, the output unit 228 outputs an inference result as shown in Fig. 2 based on information indicating the NPA region in the fundus image acquired by the inference unit 226. In one embodiment, the output unit 228 may output a blood vessel image used for the inference in addition to information indicating the NPA region in the fundus image.

[0087] As described above, according to this embodiment, the learning device 10 uses a vascular image derived from a fundus image to train a machine learning model, thereby enabling the machine learning model to train more efficiently than when training is performed using only a fundus image, and as a result, learning can converge with less training data.

[0088] Furthermore, according to this embodiment, the inference device 20 can perform inference with higher accuracy by using a vascular image derived from the fundus image that is the inference target to infer the NPA area, compared to performing inference using only the fundus image. [Explanation of symbols]

[0089] 10... learning device, 110... input unit, 120... control unit, 121... arithmetic processing unit, 122... memory, 123... learning data acquisition unit, 124... learning unit, 125... machine learning model, 126... model output unit, 130... storage unit, 131... learning data storage unit, 132... auxiliary model (first trained model), 140... communication unit, 20... inference device, 210... input unit, 220... control unit, 221... arithmetic processing unit Arithmetic processing unit, 222... memory, 223... model acquisition unit, 224... first acquisition unit, 225... second acquisition unit, 226... inference unit, 227... classification unit, 228... output unit, 230... storage unit, 231... image storage unit, 232... auxiliary model (first trained model), 233... trained model (second trained model), 234... classification model, 240... communication unit, 30... storage device, N... communication network

Claims

1. a learning unit that inputs medical images with annotation information of regions of abnormal blood circulation into a machine learning model to train the machine learning model, the regions of abnormal blood circulation including non-perfusion regions and regions with neovascularization, and trains the machine learning model based at least on an order relationship between regions without abnormalities, non-perfusion regions, and regions with neovascularization; a model output unit that outputs a trained model trained by the training unit; An information processing device comprising:

2. The information processing device according to claim 1 , further comprising a learning data acquisition unit that acquires medical images to which the annotation information has been added so that a predetermined ratio of medical images including regions of abnormal blood circulation and medical images not including regions of abnormal blood circulation is achieved.

3. The information processing device according to claim 1 , wherein the learning unit trains the machine learning model using a loss function that dynamically attenuates a weight of a cross entropy loss in an area where inference is easy.

4. acquiring a medical image to which annotation information of an abnormal blood circulation region is added, the abnormal blood circulation region including a non-perfusion region and a region in which neovascularization has occurred; inputting the medical image into a machine learning model, and training the machine learning model based at least on an order relationship between a region without abnormality, a region without perfusion, and a region where neovascularization has occurred; Outputting the trained model obtained as a result of the training; How to generate a trained model including:

5. a first acquisition unit that acquires a first image including a medical image; an inference unit that infers an abnormal blood circulation region in the first image by inputting the first image into a second trained model, the abnormal blood circulation region including a non-perfusion region and a region where neovascularization has occurred, and the second trained model is a trained model that has been trained to estimate an abnormal blood circulation region in the medical image from the medical image based at least on an order relationship between a region without abnormality, a non-perfusion region, and a region where neovascularization has occurred; an output unit that outputs the result of the inference by the inference unit; An information processing device comprising:

6. The information processing device described in Claim 5, wherein the second trained model is a trained model trained using medical images to which annotation information of regions of abnormal blood circulation has been added, the medical images being acquired so that a predetermined ratio is made between medical images including regions of abnormal blood circulation and medical images not including regions of abnormal blood circulation.

7. The present invention further comprises a second acquisition unit that acquires a second image showing a vascular region in the first image by inputting the first image into a first trained model, the first trained model being a trained model trained to estimate a vascular region from a medical image; The inference unit infers the blood circulation abnormality region in the first image by inputting the first image and the second image to the second trained model.

7. The information processing device according to claim 5 or 6.

8. The information processing device according to claim 5 or 6, wherein the second trained model is a neural network and has a large stride convolutional layer.

9. A classification unit that classifies the first image into an image that may include a blood circulation abnormality region and an image other than the first image, Only images that may include the region of abnormal blood circulation are processed by the inference unit.

7. The information processing device according to claim 5 or 6.

10. acquiring a first image comprising a medical image; inferring an abnormal blood circulation region in the first image by inputting the first image into a second trained model, the abnormal blood circulation region including a non-perfusion region and a region where neovascularization has occurred, and the second trained model is a trained model trained to estimate an abnormal blood circulation region in the medical image from the medical image based at least on an order relationship between a region without abnormality, a non-perfusion region, and a region where neovascularization has occurred; Output the inferred results and The method includes:

11. On one or more computers, acquiring a first image including a medical image; A process of inferring an abnormal blood circulation region in the first image by inputting the first image into a second trained model, the abnormal blood circulation region including a non-perfusion region and a region where neovascularization has occurred, and the second trained model is a trained model trained to estimate an abnormal blood circulation region in the medical image from the medical image based at least on an order relationship between a region without abnormality, a non-perfusion region, and a region where neovascularization has occurred; A process to output the inference results and A computer-readable recording medium having a program recorded thereon for executing the above.

12. The learning unit inputs a blood vessel image, which is an estimate of a blood vessel region in the medical image from the medical image, together with the medical image into the machine learning model to train the machine learning model.

3. The information processing device according to claim 1 or 2.

13. The learning unit trains the machine learning model using a loss function that takes into account a classification error between regions having an order of a region without abnormality, a nonperfusion region, and a region where neovascularization has occurred. The information processing device according to claim 1 .

14. The learning unit trains the machine learning model using learning data to which a probability distribution that assigns a probability to other regions related to the correct answer data is added as the annotation information. The information processing device according to claim 1 .

Images