Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of inferring one or a plurality of diseases from a medical image.SOLUTION: In an information processing system in which a learning machine is connected to an inference device and a storage device over a communication network, the inference device comprises: an image acquisition unit 225 that acquires a medical image; an image feature extraction unit 227 that extracts an image feature amount from the medical image; an inference unit 226 that infers, based on the image feature amount and a classification base 233, one or a plurality of diseases that are predicted from the medical image; and an output unit 228 that outputs results inferred by the inference unit. The classification base is a set of base vectors expressing, by using vectors of a prescribed dimension, a correlation of each disease with another disease.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In the medical field, the use of machine learning is expanding, for example, in computer-assisted detection (CADe), which uses machine learning such as deep learning on photographs taken with a standard non-mydriatic fundus camera (hereinafter referred to as "fundus images") to detect diseases such as diabetic retinopathy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 142910 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent document 1 discloses a diagnostic support device that identifies areas of abnormal blood circulation in a fundus image using a trained model that has learned the relationship between the fundus image and areas of abnormal blood circulation in the fundus image, based on the fundus image, which is an image of the fundus, and areas of abnormal blood circulation identified based on a fluorescent fundus angiography image of the fundus.

[0005] However, there are over 100 types of retinal diseases, including both localized diseases in which the lesion is limited to a small area and diffuse diseases in which the lesion is widespread. Furthermore, a single lesion may be caused by multiple diseases. Therefore, it is currently difficult to apply CADe, which covers all retinal diseases, to a single fundus image and diagnose one or more retinal diseases with high accuracy.

[0006] Developing CADe specifically for each retinal disease may enable highly accurate diagnosis of each retinal disease, but the time and effort required to develop dedicated CADe for all of the more than 100 retinal diseases is immeasurable.

[0007] Furthermore, in actual diagnoses, doctors are likely to make their final decisions based not only on fundus image information but also on information regarding correlations between diseases, but current CADe systems are unable to take such relationships between diseases into account.

[0008] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and a program that are capable of inferring one or more diseases from medical images acquired by photographing a patient's examination target region. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present invention comprises an image acquisition unit that acquires medical images, an image feature extraction unit that extracts image features from the medical images, an inference unit that infers one or more diseases predicted from the medical images based on the image features and a classification basis, where the classification basis is a set of basis vectors that express, for each disease, the correlation with other diseases using a vector of a predetermined dimension, and an output unit that outputs the results inferred by the inference unit.

[0010] According to this aspect, by using a classification basis that has learned about correlations between diseases for disease inference, it is possible to infer one or more diseases from a wide variety of diseases. Furthermore, according to this aspect, by using the classification basis, it is possible to utilize information about correlations between diseases that has not been utilized in the past, and as a result, it is possible to perform inference with higher accuracy than when inference is performed using only medical images.

[0011] In the information processing device, the image feature extraction unit acquires image features by inputting medical images to a first trained model, and the first trained model may be a trained model trained to estimate image features from medical images. According to this aspect, image features with a desired accuracy can be easily acquired.

[0012] In the information processing device, the classification basis may be obtained by receiving, as input data, data embedded with label information of a plurality of medical images labeled with one or more diseases, and training a second machine learning model that outputs features learned about correlations between diseases. According to this aspect, a classification basis with desired accuracy can be easily obtained.

[0013] In the information processing device, the data received as input is graph data in which label information of multiple medical images labeled with one or more diseases is embedded as data in the form of multi-hot vectors, and the second machine learning model is a graph convolutional neural network, and Global Max Pooling (GMP) may be applied to the final convolutional layer. This aspect enables effective downsampling that preserves important image features.

[0014] In the information processing device, the graph data may have nodes corresponding to diseases or lesions, and the inference unit may infer one or more lesions in addition to one or more diseases from the medical images. According to this aspect, not only diseases but also lesions related to the diseases can be inferred from the medical images and presented as inference results. As a result, a user who uses the inference results can develop a treatment strategy that takes into account information such as lesions common to multiple diseases.

[0015] In the information processing device, the inference unit may acquire, as one or more candidate diseases, diseases for which the certainty factor for each disease obtained by inference is greater than a predetermined threshold. By binarizing the certainty factor for each disease using the predetermined threshold, it is possible to make consistent judgments about the certainty factor.

[0016] In the information processing device, the threshold may be set for each disease, for a group including multiple diseases, or a combination of each disease and a group. By setting a threshold for each disease, compared to using the same threshold for all diseases, the threshold can be set in detail according to the characteristics of the disease, such as whether the disease is relatively easy to detect or whether early detection is important. Furthermore, for example, by grouping diseases with a low prevalence rate, it is possible to accurately indicate the possibility of the presence of some disease, even though individual candidate disease names cannot be presented.

[0017] In the information processing device, the threshold may be set to maximize a predetermined evaluation criterion for the confidence level of each disease obtained by inference. According to this aspect, it is possible to minimize the difference between the environment in which the training data was collected and the environment in which the inference is actually performed.

[0018] In the information processing device, the threshold may be set to maximize the prediction accuracy of the inference unit evaluated by k-fold cross-validation of training data used to generate the first trained model. According to this aspect, even when data at the time of inference cannot be obtained, it is possible to accurately evaluate the prediction accuracy and set a threshold that maximizes the prediction accuracy.

[0019] A method according to another aspect of the present invention includes acquiring a medical image, extracting image features from the medical image, inferring one or more diseases to be predicted from the medical image based on the image features and a classification basis, where the classification basis is a set of basis vectors that represent, for each disease, the correlation with other diseases using a vector of a predetermined dimension, and outputting the inferred results.

[0020] A program according to another aspect of the present invention causes one or more computers to perform the following processes: acquiring medical images; extracting image features from the medical images; inferring one or more diseases to be predicted from the medical images based on the image features and a classification basis, where the classification basis is a set of basis vectors that express, for each disease, the correlation with other diseases using a vector of a predetermined dimension; and outputting the inferred results.

[0021] An information processing device according to another aspect of the present invention comprises: a first machine learning model that outputs, from an image, image features that indicate characteristics of the image; a second machine learning model that outputs a classification basis learned about correlations between diseases from input data generated based on label information of a plurality of medical images labeled with one or more diseases; a learning unit that trains the first machine learning model and the second machine learning model using a plurality of medical images labeled with one or more diseases, wherein the learning unit calculates a confidence level for each disease based on the image features obtained from the first machine learning model and the classification basis obtained from the second machine learning model, and trains the first machine learning model and the second machine learning model so as to minimize the error between the confidence level and the disease labeled in the medical image, wherein the classification basis is a set of basis vectors that express, for each disease, the correlation with other diseases using a vector of a predetermined dimension; a model output unit that outputs a trained model obtained by training the first machine learning model; and a basis output unit that outputs the classification basis obtained by training the second machine learning model.

[0022] According to this aspect, it is possible to obtain an image feature extraction unit that can be used to infer diseases predicted from medical images, and a classification basis that has learned about the correlation between diseases.

[0023] Another aspect of the present invention relates to a method including: acquiring a plurality of medical images labeled with one or more diseases and a plurality of medical images without the diseases; and training a first machine learning model and a second machine learning model using the medical images, wherein the first machine learning model is a machine learning model that outputs, from an image, image features that indicate characteristics of the image; and the second machine learning model is a machine learning model that outputs a classification basis learned about correlations between diseases from input data generated based on label information of a plurality of medical images labeled with one or more diseases; calculating a confidence level for each disease based on the image features acquired from the first machine learning model and the classification basis acquired from the second machine learning model; and training the first machine learning model and the second machine learning model so as to minimize an error between the confidence level and the disease labeled in the medical image; the classification basis is a set of basis vectors that express, for each disease, the correlation with other diseases using a vector of a predetermined dimension; and outputting the trained model obtained by training the first machine learning model; and outputting the classification basis obtained by training the second machine learning model. [Effects of the Invention]

[0024] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that are capable of inferring one or more diseases from medical images. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram showing a network configuration of an information processing system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a schematic diagram illustrating the processing of a learning device according to an embodiment of the present invention. [Figure 3] FIG. 2 is a schematic diagram illustrating the processing of an inference device according to an embodiment of the present invention. [Figure 4] 1 is a block diagram of a learning device according to an embodiment of the present invention. [Figure 5] 1 is a block diagram of an inference device according to an embodiment of the present invention. [Figure 6] 10 is a flowchart showing a learning process of the learning device according to one embodiment of the present invention. [Figure 7] 4 is a flowchart showing an inference process of the inference device according to one embodiment of the present invention. [Figure 8] 1 is a schematic diagram illustrating thresholds used by an inference device according to an embodiment of the present invention. [Figure 9] FIG. 2 is a schematic diagram illustrating the binarization process of an inference device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Embodiments of the present invention will be described with reference to the accompanying drawings. Note that the following embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. Furthermore, the present invention can be modified in various ways without departing from the gist of the present invention. Furthermore, those skilled in the art can adopt embodiments in which the elements described below are replaced with equivalents, and such embodiments are also included in the scope of the present invention.

[0027] (System Configuration) An overview of the present invention will be described using Figures 1, 2, and 3. Figure 1 is a diagram showing the network configuration of an information processing system according to one embodiment of the present invention. Figure 2 is a schematic diagram explaining the processing of a learning device according to one embodiment of the present invention. Figure 3 is a schematic diagram explaining the processing of an inference device according to one embodiment of the present invention.

[0028] The information processing system includes a learning device 10, an inference device 20, and a storage device 30. The learning device 10 is connected to the inference device 20 and the storage device 30 via a communication network N. The communication network N may be either a wired communication network or a wireless communication network configured with wired or wireless lines, or may be the Internet or a local area network (LAN).

[0029] The learning device 10 learns a machine learning model based on the learning data stored in the storage device 30, and stores the learned model in the storage device 30. The learning device 10 according to this embodiment includes a machine learning model, but the machine learning model may be provided in a device separate from the learning device 10.

[0030] Here, a machine learning model is a model that has a predetermined model structure and processing parameters that vary through a learning process, and that improves classification accuracy by optimizing the processing parameters based on experience gained from training data. That is, a machine learning model is a model that learns optimal processing parameters through a learning process. The algorithm of a machine learning model can be, for example, a support vector machine, a logistic regression, a neural network, or the like, but the type is not particularly limited. Machine learning models that perform this learning include those that have not yet been trained and those that have already undergone some kind of training using training data.

[0031] A trained model is a machine learning model based on an arbitrary machine learning algorithm that has been trained in advance using appropriate training data. However, a trained model does not necessarily mean that it will not undergo further training, and additional training can also be performed.

[0032] The inference device 20 uses a trained model to output output data according to the characteristics of the input data. The inference device 20 according to this embodiment performs inference using a trained model acquired from the storage device 30. Here, acquiring a trained model refers to acquiring information necessary to reproduce the function of the trained model in the inference device 20. For example, when a neural network is used as the machine learning model, acquiring a trained model refers to acquiring information regarding at least the number of layers of the neural network, the number of nodes for each layer, weight parameters for links connecting the nodes, bias parameters for each node, and the functional form of the activation function for each node.

[0033] The storage device 30 stores training data used for training the machine learning model. The storage device 30 according to this embodiment stores, as training data, fundus images labeled with one or more retinal diseases and fundus images without retinal diseases. The storage device 30 also stores a trained model output by the training device 10. While FIG. 1 illustrates the storage device 30 as a single storage device, the storage device 30 may be configured with one or more file servers. In this embodiment, fundus images labeled with one or more retinal diseases are used as an example of training data. However, in another embodiment, medical images labeled with one or more diseases in other areas of the patient to be examined may also be used as training data.

[0034] 2, the learning device 10 according to this embodiment includes a machine learning model (first machine learning model) that outputs image features indicating characteristics of a fundus image from the fundus image, and a machine learning model (second machine learning model) that outputs a classification basis learned about correlations between retinal diseases from input data generated based on label information of multiple fundus images labeled with one or more retinal diseases. The classification basis is a set of basis vectors that express, for each disease, the correlation with other diseases using vectors of a predetermined dimension.

[0035] The learning device 10 predicts retinal diseases from fundus images using image features output from the first machine learning model and classification bases output from the second machine learning model, and trains the first machine learning model and the second machine learning model so as to minimize the error between the predicted result and the retinal diseases labeled in the training data. In this embodiment, an example using a fundus image, which is an example of a medical image, will be described, but in other embodiments, images of the chest, brain, liver, etc., acquired by photographing other parts of the patient to be examined can be used.

[0036] By having the machine learning model learn not only information from fundus images but also correlations between diseases, it is possible to generate a trained model that enables highly accurate inferences that take into account correlations between diseases.

[0037] As shown in FIG. 3 , the inference device 20 according to this embodiment infers retinal diseases predicted from fundus images using a trained model obtained by training a first machine learning model and a classification basis obtained by training a second machine learning model. By using a classification basis learned about correlations between diseases to infer diseases, more accurate inference can be achieved compared to inference performed using fundus images alone. Furthermore, the inference device 20 according to this embodiment can present multiple disease candidates from a single fundus image without generating trained models corresponding to each disease. While this embodiment infers disease candidates from fundus images, which are an example of medical images, in another embodiment, diseases can be inferred from medical images, such as chest images, brain images, and liver images, acquired by photographing other areas of the patient to be examined.

[0038] (Functional configuration: learning device) Figure 4 is a block diagram of a learning device according to one embodiment of the present invention. Note that, while Figure 4 assumes a single learning device 10 and shows only the necessary functional configuration, the learning device 10 can also be configured as part of a multi-functional distributed system consisting of multiple computer systems.

[0039] The learning device 10 includes an input unit 110, a control unit 120, a storage unit 130, and a communication unit 140.

[0040] Input unit 110 is configured to accept operations from the administrator of study device 10, and can be realized by a keyboard, mouse, touch panel, or the like.

[0041] The control unit 120 includes an arithmetic processing unit 121 such as a CPU or MPU equivalent to a processor, and a memory 122 such as a RAM. The arithmetic processing unit 121 (processor) implements the functions and processes described below in the arithmetic processing unit 121 by loading a program recorded in the storage unit 130 into the memory 122 and executing it based on various inputs. This program may be stored on a computer-readable non-transitory recording medium such as a CD-ROM, or may be distributed via a network and installed on a computer. The memory 122 functions as a work memory required for the arithmetic processing unit 121 (processor) to execute the program.

[0042] The storage unit 130 is configured by a storage device such as a hard disk, and stores various programs required for executing processes in the control unit 120, data required for executing the various programs, etc. In this embodiment, the storage unit 130 preferably includes a learning data storage unit 131.

[0043] The training data storage unit 131 stores training data used for training a first machine learning model M1 and a second machine learning model M2, which will be described later. In this embodiment, the training data storage unit 131 stores fundus images labeled with one or more retinal diseases and fundus images without retinal diseases.

[0044] The communication unit 140 is configured to connect the learning device 10 to a network. For example, the communication unit 140 can be realized by a LAN card, an analog modem, an ISDN modem, or the like, and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0045] Furthermore, as shown in FIG. 4, the arithmetic processing unit 121 includes, as functional units, a learning data acquisition unit 123, a learning unit 124, an image feature extraction unit 125, a correlation feature extraction unit 126, a model output unit 127, and a basis output unit 128.

[0046] The training data acquisition unit 123 acquires training data used for training a first machine learning model M1 and a second machine learning model M2, which will be described later, and stores the data in the training data storage unit 131. In this embodiment, the training data acquisition unit 123 acquires fundus images labeled with one or more retinal diseases and fundus images without retinal diseases from the storage device 30, and stores the data in the training data storage unit 131.

[0047] The learning unit 124 trains the first machine learning model M1 and the second machine learning model M2 using the learning data acquired by the learning data acquisition unit 123. In this embodiment, as shown in Fig. 2, the learning unit 124 first inputs a fundus image labeled with one or more retinal diseases and a fundus image without retinal diseases to an image feature extraction unit 125 and a correlation feature extraction unit 126, which will be described later, respectively, and acquires a classification basis related to the correlation between retinal diseases extracted from the image features extracted from the fundus image and the label information.

[0048] Next, the learning unit 124 calculates the Hadamard product of the image features obtained from the image feature extraction unit 125 and the classification basis obtained from the correlation feature extraction unit 126, applies an activation function, and trains the first machine learning model M1 and the second machine learning model M2 so as to minimize the error between the obtained confidence levels for each of the C retinal diseases and the retinal diseases labeled in the training data. Here, C indicates the number of retinal diseases to be predicted, and is equal to the number of vectors constituting the classification basis, as described below.

[0049] The image feature extraction unit 125 receives an input of an image and extracts image feature amounts indicating characteristics of the image from the image. In this embodiment, the image feature extraction unit 125 receives an input of a fundus image and extracts image feature amounts from the fundus image using a first machine learning model M1.

[0050] The first machine learning model M1 is a machine learning model that outputs, from an image, image features that indicate characteristics of the image. In this embodiment, an example of the first machine learning model M1 will be described, in which a convolutional neural network (CNN) that receives a fundus image as input data and outputs image features that indicate characteristics of the fundus image is used. However, CNN is merely an example of the first machine learning model M1, and the learning device 10 may use other configurations as the first machine learning model M1.

[0051] In one embodiment, the first machine learning model M1 applies Global Max Pooling (GMP) to the final convolutional layer to output D-dimensional image features. In this way, applying GMP, which selects the maximum value of each channel of the feature map, enables effective downsampling that retains important image features.

[0052] The correlation feature extraction unit 126 extracts classification bases relating to correlations between diseases from a plurality of images labeled with one or more diseases. The classification bases are a set of basis vectors that express, for each disease, the correlations with other diseases using vectors of a predetermined dimension. For example, as shown in FIG. 2, in this embodiment, the classification bases are expressed as a set of basis vectors that express, for C diseases, the correlations with other diseases using vectors of D dimensions.

[0053] In this embodiment, the correlation feature extraction unit 126 receives an input of a fundus image labeled with one or more retinal diseases, generates graph data in which the label information of the fundus image is embedded as data in the form of a multi-hot vector, and extracts a classification basis using a second machine learning model M2. As shown in FIG. 2, each node of the graph data corresponds to a retinal disease. Although the graph data is depicted in a simplified form in FIG. 2, it is assumed that graph data having C nodes corresponding to C retinal diseases is generated.

[0054] The second machine learning model M2 is a machine learning model that outputs features learned about the correlations between items to be learned. In this embodiment, an example of the second machine learning model M2 will be described, using a graph convolutional neural network (GCN). The graph data generated by the correlation feature extraction unit 126 is received as input data, and a classification basis learned about the correlations between retinal diseases is output. However, a GCN is merely an example of the second machine learning model M2, and the learning device 10 may use other configurations for the second machine learning model M2. In this embodiment, the second machine learning model M2 expresses the features of nodes in d dimensions, convolves information about adjacent nodes into the features of each node according to the adjacency relationships in the graph, and outputs a D-dimensional classification basis.

[0055] When the training of the first machine learning model M1 and the second machine learning model M2 is completed, the model output unit 127 outputs the trained model obtained by training the first machine learning model M1 to the storage device 30. Note that the training unit 124 may complete the training after training the first machine learning model M1 and the second machine learning model M2 using a predetermined number of training data, or may complete the training when the accuracy of the retinal disease predicted using the first machine learning model M1 and the second machine learning model M2 satisfies a predetermined condition.

[0056] Similarly, when the training of the first machine learning model M1 and the second machine learning model M2 is completed, the basis output unit 128 outputs the classification basis obtained by training the second machine learning model M2 to the storage device 30.

[0057] (Functional configuration: Inference device) Figure 5 is a block diagram of an inference device according to one embodiment of the present invention. Note that Figure 5 assumes a single inference device 20 and shows only the necessary functional configuration, but inference device 20 can also be configured as part of a multi-functional distributed system consisting of multiple computer systems.

[0058] The inference device 20 includes an input unit 210 , a control unit 220 , a storage unit 230 , and a communication unit 240 .

[0059] The input unit 210 is configured to accept operations from the administrator of the inference device 20, and can be realized by a keyboard, mouse, touch panel, or the like.

[0060] The control unit 220 includes an arithmetic processing unit 221 such as a CPU or MPU equivalent to a processor, and a memory 222 such as a RAM. The arithmetic processing unit 221 (processor) implements the functions and processing described below in the arithmetic processing unit 221 by loading a program recorded in the storage unit 230 into the memory 222 and executing it based on various inputs. This program may be stored in a computer-readable non-transitory recording medium such as a CD-ROM, or may be distributed via a network and installed on a computer. The memory 222 functions as a work memory required for the arithmetic processing unit 221 (processor) to execute the program.

[0061] The storage unit 230 is configured by a storage device such as a hard disk, and stores various programs required for executing processes in the control unit 220, data required for executing the various programs, etc. In this embodiment, the storage unit 230 preferably includes an image storage unit 231, a trained model 232, and a classification base 233.

[0062] An image to be inferred is stored in the image storage unit 231. In this embodiment, the image storage unit 231 stores a fundus image for inferring a retinal disease.

[0063] The trained model 232 stores a trained model used for inference. In this embodiment, the trained model 232 stores a trained model that receives a fundus image as input data and outputs image features that indicate features in the fundus image. In this embodiment, an example of the trained model 232 will be described, in which a convolutional neural network (CNN) that receives a fundus image as input data and outputs image features that indicate features in the fundus image is used. However, CNN is merely one example of the trained model 232, and the inference device 20 may use other configurations for the trained model 232.

[0064] Basis vectors used for inference are stored in the classification basis 233. In this embodiment, the classification basis 233 stores a set of basis vectors that express, for each disease, the correlation with other diseases using a vector of a predetermined dimension.

[0065] The communication unit 240 is configured to connect the inference device 20 to a network. For example, the communication unit 240 can be realized by a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0066] Furthermore, as shown in FIG. 5, the arithmetic processing unit 221 includes a model acquisition unit 223, a basis acquisition unit 224, an image acquisition unit 225, an inference unit 226, an image feature extraction unit 227, and an output unit 228 as functional units.

[0067] The model acquisition unit 223 acquires a trained model to be used for inference and stores it in the trained model 232. In this embodiment, the model acquisition unit 223 acquires a trained model from the storage device 30 and stores it in the trained model 232.

[0068] The basis acquisition unit 224 acquires basis vectors to be used for inference and stores them in the classification basis 233. In this embodiment, the basis acquisition unit 224 acquires classification bases from the storage device 30 and stores them in the classification basis 233.

[0069] The image acquisition unit 225 acquires an image of an inference target. In this embodiment, the image acquisition unit 225 acquires, from the image storage unit 231, a fundus image for inferring a retinal disease.

[0070] The inference unit 226 infers a predicted retinal disease from the image acquired by the image acquisition unit 225. In this embodiment, as shown in FIG. 3 , the inference unit 226 first inputs the fundus image acquired by the image acquisition unit 225 to the image feature extraction unit 227 (described later) to acquire image features, and calculates the Hadamard product of the acquired image features and the classification basis 233. Next, the inference unit 226 binarizes the confidence levels for each of the C retinal diseases obtained by applying the activation function to the Hadamard product using a predetermined threshold, and acquires retinal diseases with confidence levels greater than the predetermined threshold as candidate diseases. In one embodiment, the inference unit 226 may pass the confidence levels for each of the C retinal diseases obtained by applying the activation function to the Hadamard product to the output unit 228 in addition to or instead of the information on the candidate diseases.

[0071] Here, the threshold used to determine candidate diseases may be a preset value, or a value input by the administrator via the input unit 210. For example, three preset thresholds may be set according to the characteristics of the facility using the inference device 20, and one of the thresholds may be set according to input from the administrator. A facility that performs medical examinations may use a low threshold to indicate a wide range of possible diseases, a general facility that performs diagnosis and treatment may use a middle threshold to indicate diseases that may be treatable, and a specialized facility that provides advanced medical care may use a high threshold to indicate diseases that may be treatable. Furthermore, the preset threshold may be adjustable according to input from the administrator. By making the threshold adjustable, more optimized inference results can be output according to the patient demographic and the facility's policy.

[0072] In one embodiment, the inference unit 226 may set a threshold for each retinal disease or for a group including multiple retinal diseases. Setting a threshold for each disease allows for more precise threshold setting based on the characteristics of each disease, such as whether the disease is relatively easy to detect or whether early detection is important, compared to using the same threshold for all diseases. However, it is difficult to individually set appropriate thresholds for all of the more than 100 types of retinal diseases. Considering the possibility that a single lesion may be caused by multiple diseases, setting an appropriate threshold for each retinal disease is virtually impossible. In such cases, combining an individual threshold for each retinal disease with a threshold for a group including multiple retinal diseases allows for accurate identification of candidate diseases. In other words, by grouping diseases with a low prevalence, it is not possible to present individual candidate disease names, but it is possible to accurately indicate the possibility of the presence of some disease.

[0073] The image feature extraction unit 227 receives an input of an image and extracts image feature amounts indicating features of the image from the image. In this embodiment, the image feature extraction unit 227 receives an input of a fundus image and extracts image feature amounts from the fundus image using the trained model 232.

[0074] The output unit 228 outputs an inference result based on the information acquired by the inference unit 226. In this embodiment, the output unit 228 outputs an inference result based on the information of the candidate disease acquired by the inference unit 226. In one embodiment, the output unit 228 may output a certainty of the disease in addition to or instead of the information of the candidate disease.

[0075] (First embodiment) (Learning process) The learning process of the learning device according to the embodiment of the present invention will be described in detail with reference to Fig. 6. In this embodiment, before the learning process described in Fig. 6 is performed, learning data is stored in the storage device 30 under the management of an administrator of the learning device 10. Note that the process shown in Fig. 6 is executed, for example, by the administrator inputting an instruction to execute a process for generating a trained model via the input unit 110.

[0076] In step S601, the learning data acquisition unit 123 of the learning device 10 acquires learning data used for training the first machine learning model M1 and the second machine learning model M2, and stores the acquired learning data in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires fundus images labeled with one or more retinal diseases and fundus images without retinal diseases from the storage device 30, and stores the fundus images in the learning data storage unit 131.

[0077] Next, in step S602, the learning unit 124 of the learning device 10 uses the learning data acquired by the learning data acquisition unit 123 to train the first machine learning model M1 and the second machine learning model M2.

[0078] The first machine learning model M1 is a machine learning model that outputs, from an image, image features that indicate characteristics of the image. In this embodiment, a convolutional neural network (CNN) that receives a fundus image as input data and outputs image features that indicate characteristics of the fundus image is used as an example of the first machine learning model M1. However, CNN is merely an example of the first machine learning model M1, and the learning device 10 may use other configurations as the first machine learning model M1.

[0079] Furthermore, in this embodiment, as shown in Fig. 2, the first machine learning model M1 applies Global Max Pooling (GMP) to the final convolutional layer to output D-dimensional image features. In this way, by applying GMP, which selects the maximum value of each channel of the feature map, it is possible to perform effective downsampling that retains important image features.

[0080] The second machine learning model M2 is a machine learning model that outputs features learned about the correlations between items to be learned. In this embodiment, a graph convolutional neural network (GCN) is used as an example of the second machine learning model M2. The GCN receives graph data as input data and outputs a classification basis learned about the correlations between retinal diseases. However, the GCN is merely an example of the second machine learning model M2, and the learning device 10 may use other configurations as the second machine learning model M2. In this embodiment, the second machine learning model M2 expresses the features of nodes in d dimensions, convolves information about adjacent nodes into the features of each node according to the adjacency relationships in the graph, and outputs a D-dimensional classification basis.

[0081] Specifically, in this embodiment, the learning unit 124 inputs a fundus image labeled with one or more retinal diseases and a fundus image without retinal diseases to the image feature extraction unit 125 and the correlation feature extraction unit 126, respectively. The image feature extraction unit 125 of the learning device 10 accepts the input of the fundus image and extracts image features from the fundus image using a first machine learning model M1. The correlation feature extraction unit 126 of the learning device 10 accepts the input of the fundus image labeled with one or more retinal diseases, generates graph data in which the label information of the fundus image is embedded as data in the form of a multi-hot vector, and extracts a classification basis using a second machine learning model M2. In this embodiment, as shown in FIG. 2 , the correlation feature extraction unit 126 generates graph data having C nodes corresponding to C retinal diseases and extracts classification bases, which are sets of basis vectors representing the correlations between the C diseases and other diseases using D-dimensional vectors, using the second machine learning model M2.

[0082] After the processing by the image feature extraction unit 125 and the correlation feature extraction unit 126 is completed, the learning unit 124 calculates the Hadamard product of the image features obtained from the image feature extraction unit 125 and the classification bases obtained from the correlation feature extraction unit 126, applies an activation function, and trains the first machine learning model M1 and the second machine learning model M2 so as to minimize the error between the obtained confidence levels for each of the C retinal diseases and the retinal diseases labeled in the training data.

[0083] When the learning of the first machine learning model M1 and the second machine learning model M2 is completed, in step S603, the model output unit 127 of the learning device 10 outputs the learned model obtained by training the first machine learning model M1 to the storage device 30. Note that the learning unit 124 may complete the learning after training the first machine learning model M1 and the second machine learning model M2 using a predetermined number of training data, or may complete the learning when the accuracy of the retinal disease predicted using the first machine learning model M1 and the second machine learning model M2 satisfies a predetermined condition.

[0084] Similarly, when the learning of the first machine learning model M1 and the second machine learning model M2 is completed, in step S604, the basis output unit 128 of the learning device 10 outputs the classification basis obtained by training the second machine learning model M2 to the memory device 30.

[0085] (inference processing) The inference processing of the inference device according to the embodiment of the present invention will be described in detail with reference to Fig. 7. In this embodiment, before the inference processing described in Fig. 7 is performed, it is assumed that the trained model 232 and classification base 233 are stored with the trained model and classification base acquired from the storage device 30 under the management of an administrator of the inference device 20. It is also assumed that the image storage unit 231 of the inference device 20 stores a fundus image to be inferred. The processing shown in Fig. 7 is executed, for example, by the administrator inputting an instruction to execute the inference processing via the input unit 210.

[0086] In step S701, the image acquisition unit 225 of the inference device 20 acquires an image to be inferred. In this embodiment, the image acquisition unit 225 acquires, from the image storage unit 231, a fundus image for inferring a retinal disease.

[0087] In step S702, the inference unit 226 of the inference device 20 inputs the fundus image acquired by the image acquisition unit 225 to the image feature extraction unit 227 to acquire image features. Specifically, the image feature extraction unit 227 accepts the input of the fundus image and extracts image features from the fundus image using the trained model 232.

[0088] In this embodiment, a convolutional neural network (CNN) that receives a fundus image as input data and outputs image features indicating the characteristics of the fundus image is used as an example of the trained model 232. As shown in Fig. 3, the trained model 232 uses Global Max Pooling (GMP) in the final convolutional layer to output D-dimensional image features. In this way, by applying GMP that selects the maximum value of each channel of the feature map, effective downsampling that retains important image features can be performed.

[0089] Next, in step S703, the inference unit 226 calculates the Hadamard product of the acquired image feature amount and the classification base 233.

[0090] Finally, in step S704, the inference unit 226 binarizes the confidence levels for each of the C retinal diseases obtained by applying the activation function to the Hadamard product using a predetermined threshold, and acquires retinal diseases having confidence levels greater than the predetermined threshold as candidate diseases. In one embodiment, the inference unit 226 may deliver the confidence levels for each of the C retinal diseases obtained by applying the activation function to the Hadamard product to the output unit 228 in addition to or instead of the information on the candidate diseases.

[0091] In step S705, the output unit 228 of the inference device 20 outputs an inference result based on the information acquired by the inference unit 226. In this embodiment, the output unit 228 outputs an inference result based on the information of the candidate disease acquired by the inference unit 226. In one embodiment, the output unit 228 may output the certainty of the disease in addition to or instead of the information of the candidate disease.

[0092] In this embodiment, an example has been described in which the correlation feature extraction unit 126 generates graph data having nodes corresponding to retinal diseases. In one embodiment, the correlation feature extraction unit 126 may generate graph data having nodes corresponding to lesions associated with diseases, such as regions of abnormal blood circulation including nonperfusion regions, which are regions where no or little blood flows, and regions of tissue degeneration, in addition to retinal diseases. In this way, it is possible to infer not only retinal diseases but also lesions associated with retinal diseases from fundus images, calculate the confidence level, and present the inference result. As a result, doctors can develop treatment strategies that take into account information such as lesions common to multiple diseases.

[0093] As described above, according to this embodiment, the learning device 10 can obtain an image feature extraction unit that can be used to infer retinal diseases predicted from fundus images, and a classification basis that has learned about the correlations between retinal diseases.

[0094] Furthermore, according to this embodiment, the inference device 20 can infer one or more retinal diseases from a wide variety of retinal diseases by using a classification basis that has learned about the correlations between retinal diseases to infer retinal diseases. Furthermore, according to this embodiment, by using the classification basis, it is possible to utilize information about the correlations between diseases that has not been utilized in the past, and as a result, it is possible to perform inference with higher accuracy than when inference is performed using only fundus images.

[0095] (Second embodiment) In the above learning process and inference process, an example has been described in which the same threshold is used for all diseases in step S704 of the inference process. In this embodiment, an example will be described in which an individual threshold for each retinal disease and a threshold for a group including multiple retinal diseases are used in combination in step S704 of the inference process. Here, the inference unit 226 uses thresholds that are individually set to 50%, 20%, and 20% for glaucoma, age-related macular degeneration, and diabetic retinopathy, respectively, and uses a threshold that is set to 10% for the group including other retinal diseases for other retinal diseases.

[0096] (Learning process) 2, the correlation feature extraction unit 126 generated graph data in which each node of the graph data corresponded to C retinal diseases and input the graph data to the second machine learning model M2. In the second embodiment, the correlation feature extraction unit 126 generated graph data corresponding to glaucoma, age-related macular degeneration, diabetic retinopathy, and other retinal diseases and input the graph data to the second machine learning model M2. That is, in the first embodiment, graph data having C nodes corresponding to C retinal diseases was generated, whereas in the second embodiment, graph data having four nodes was generated.

[0097] As described above, the second machine learning model M2 is a machine learning model that outputs features learned about the correlations between items to be learned. In the second embodiment, as in the first embodiment, the second machine learning model M2 expresses the feature amounts of nodes in d dimensions, convolves the information of adjacent nodes into the feature amounts of each node according to the adjacency relationships in the graph, and outputs a D-dimensional classification basis.

[0098] 6 is the same as that in the first embodiment, and therefore description thereof will be omitted. Note that in step S601, the learning data acquired by the learning data acquisition unit 123 is a fundus image labeled with one or more of the C retinal diseases and a fundus image without a retinal disease in the first embodiment, whereas in the second embodiment, the learning data is a fundus image labeled with one or more of four retinal diseases, namely, glaucoma, age-related macular degeneration, diabetic retinopathy, and other retinal diseases, and a fundus image without a retinal disease.

[0099] (inference processing) Of the inference processing described with reference to FIG. 7, step S704, which differs from the first embodiment, will be described in detail with reference to FIG.

[0100] In step S701, the image acquisition unit 225 acquires a fundus image for inferring a retinal disease from the image storage unit 231. In step S702, the inference unit 226 of the inference device 20 inputs the fundus image acquired by the image acquisition unit 225 to the image feature extraction unit 227 to acquire image features. Specifically, the image feature extraction unit 227 accepts the input of the fundus image and extracts image features from the fundus image using the trained model 232.

[0101] In the second embodiment, a convolutional neural network (CNN) that receives a fundus image as input data and outputs image features indicating the characteristics of the fundus image is used as an example of the trained model 232. As shown in Fig. 3, the trained model 232 uses Global Max Pooling (GMP) in the final convolutional layer to output D-dimensional image features. Next, in step S703, the inference unit 226 calculates the Hadamard product of the acquired image features and the classification base 233.

[0102] In step S704, the inference unit 226 binarizes the confidence levels for each of the four retinal diseases obtained by applying the activation function to the Hadamard product using a predetermined threshold, and acquires retinal diseases with confidence levels greater than the predetermined threshold as candidate diseases. In this embodiment, the inference unit 226 uses thresholds set individually at 50%, 20%, and 20% for glaucoma, age-related macular degeneration, and diabetic retinopathy, respectively, and uses a threshold set at 10% for a group including other retinal diseases for other retinal diseases.

[0103] 8, it is assumed that the certainty factors of glaucoma, age-related macular degeneration, diabetic retinopathy, and other retinal diseases predicted from a certain fundus image are 70%, 25%, 15%, and 15%, respectively. In such a case, the inference unit 226 acquires glaucoma, age-related macular degeneration, and other retinal diseases with certainty factors greater than the threshold as candidate diseases.

[0104] In step S705, the output unit 228 outputs the inference result based on the information on the candidate disease acquired by the inference unit 226.

[0105] Assuming that the inference unit 226 uses thresholds set individually at 80%, 30%, and 20% for glaucoma, age-related macular degeneration, and diabetic retinopathy, respectively, as shown on the right side of Figure 8, and uses a threshold set at 20% for other retinal diseases for a group including other retinal diseases, if the certainty levels of glaucoma, age-related macular degeneration, diabetic retinopathy, and other retinal diseases predicted from a certain fundus image are 70%, 25%, 15%, and 15%, respectively, the inference unit 226 will pass on information to the output unit 228 that there are no candidate diseases, since there are no retinal diseases with a certainty level greater than the threshold.

[0106] In this way, even if the confidence level of a retinal disease predicted from a certain fundus image is the same, by making it possible to adjust the threshold, it may or may not be presented as a candidate disease. By optimizing the threshold setting, it is possible to output more optimized inference results in accordance with the patient demographics and the facility's policies.

[0107] As described above, according to this embodiment, the inference device 20 sets a threshold for each disease, compared to using the same threshold for all diseases. This allows the threshold to be set in detail according to the characteristics of each disease, such as whether the disease is relatively easy to detect or whether early detection is important. On the other hand, by using a threshold for a group including multiple retinal diseases in combination with an individual threshold for each retinal disease, candidate diseases can be obtained with high accuracy. In other words, by grouping diseases with a small number of affected individuals, it is not possible to present individual candidate disease names, but it is possible to accurately present the possibility of the presence of some disease.

[0108] (Threshold optimization) As described above, in one embodiment, inference device 20 binarizes the confidence level for each of C retinal diseases using a predetermined threshold, and selects retinal diseases with confidence levels greater than the predetermined threshold as candidate diseases. This process, shown surrounded by a solid line in Figure 3, will be described in detail using Figure 9.

[0109] 9 shows an example of the first embodiment in which the same threshold value of 0.5 is used for all diseases. In this case, the inference device 20 predicts disease A and disease B with a certainty greater than the threshold as candidate diseases, and does not select disease Z with a certainty less than the threshold as a candidate disease. In the second embodiment described above, the usefulness of using a threshold value for each disease or a threshold value for a group including multiple diseases has been described.

[0110] To optimize the threshold for binarizing the obtained confidence level, we focus on the differences in the data input during learning and inference. For example, if the inference device 20 uses training data collected at a facility with a high prevalence rate during learning and a facility conducting medical examinations with a low prevalence rate during inference, the distribution of input data will differ between learning and inference. Furthermore, if the inference device 20 uses training data collected at a facility that identifies even slight abnormalities as disease during learning and a facility that identifies only obvious abnormalities as disease during inference, whether or not a candidate disease is identified will differ even if the confidence level is the same. Furthermore, because the fundus images of the training data and the fundus images to be inferred use different imaging devices and environments, even if the fundus images are taken at the same time for the same patient, different fundus images will be obtained due to the influence of the imaging device and imaging environment.

[0111] Therefore, the threshold is optimized to minimize the difference between the environment in which the learning data was collected and the environment in which inference is actually performed, as well as the imbalance in the learning data.

[0112] When data at the time of inference can be obtained, a threshold for each disease may be set so as to maximize a predetermined evaluation criterion for the confidence obtained by inputting the image to be inferred. For example, a threshold that maximizes the degree of separation between correctly binarized cases and incorrectly binarized cases may be set as the optimal threshold. Note that any evaluation criterion or algorithm may be selected for use in setting the optimal threshold. In this way, it is possible to minimize the difference between the environment in which the learning data was collected and the environment in which the inference is actually performed.

[0113] When data at the time of inference cannot be obtained, the accuracy of the inference device 20 may be evaluated by k-fold cross-validation of the training data, and a threshold value for each disease may be set to maximize the prediction accuracy. It is desirable to use a method for dividing the training data that reduces the bias in the data distribution between the training data and the evaluation data after division. By doing so, even when data at the time of inference cannot be obtained, it is possible to accurately evaluate the prediction accuracy and set a threshold value that maximizes the prediction accuracy. [Explanation of symbols]

[0114] 10... learning device, 110... input unit, 120... control unit, 121... arithmetic processing unit, 122... memory, 123... learning data acquisition unit, 124... learning unit, 125... image feature extraction unit, 126... correlation feature extraction unit, 127... model output unit, 128... basis output unit, 130... memory unit, 131... learning data memory unit, 140... communication unit, 20... inference device, 210... input unit, 220... control unit, 221... Processing unit, 222...memory, 223...model acquisition unit, 224...basis acquisition unit, 225...image acquisition unit, 226...inference unit, 227...image feature extraction unit, 228...output unit, 230...storage unit, 231...image storage unit, 232...trained model, 233...classification basis, 240...communication unit, 30...storage device, M1...first machine learning model, M2...second machine learning model, N...communication network

Claims

1. an image acquisition unit for acquiring medical images; an image feature extraction unit that extracts image features from the medical image; an inference unit that infers one or more diseases predicted from the medical image based on the image feature amount and a classification basis, the classification basis being a set of basis vectors that express, for each disease, a correlation with other diseases by a vector of a predetermined dimension; an output unit that outputs the result of the inference by the inference unit; An information processing device comprising:

2. the image feature extraction unit acquires the image feature by inputting the medical image to a first trained model; The first trained model is a trained model trained to estimate image features from medical images.

2. The information processing device according to claim 1.

3. The classification basis is obtained by receiving, as input data, data embedded with label information of a plurality of medical images labeled with one or a plurality of diseases, and training a second machine learning model that outputs features learned about correlations between diseases.

3. The information processing device according to claim 1 or 2.

4. The data received as the input data is graph data in which label information of a plurality of medical images labeled with one or a plurality of diseases is embedded as data in the form of a multi-hot vector; The second machine learning model is a graph convolutional neural network, and applies Global Max Pooling (GMP) to the final convolutional layer. The information processing device according to claim 3 .

5. the graph data has nodes corresponding to diseases or lesions; the inference unit infers one or more lesions in addition to one or more diseases from the medical image; The information processing device according to claim 4 .

6. The information processing device according to claim 1 , wherein the inference unit acquires, as one or more candidate diseases, diseases for which the certainty factor for each disease obtained by inference is greater than a predetermined threshold value.

7. The information processing device according to claim 6 , wherein the threshold is set for each disease, for a group including a plurality of diseases, or for a combination of each disease and a group.

8. The information processing device according to claim 7 , wherein the threshold value is set so as to maximize a predetermined evaluation criterion for the certainty factor for each disease obtained by inference.

9. The inference unit acquires, as one or more candidate diseases, diseases for which the certainty factor for each disease obtained by inference is greater than a predetermined threshold, The information processing device according to claim 2 , wherein the threshold is set so as to maximize a prediction accuracy of the inference unit evaluated by k-fold cross-validation of training data used in generating the first trained model.

10. A method for operating an information processing device, comprising: acquiring a medical image; extracting image features from the medical image; Inferring one or more diseases predicted from the medical image based on the image feature amount and a classification basis, the classification basis being a set of basis vectors that express, for each disease, a correlation with other diseases by a vector of a predetermined dimension; Outputting the inferred results A method comprising:

11. On one or more computers, acquiring a medical image; A process of extracting image features from the medical image; A process of inferring one or more diseases predicted from the medical image based on the image feature amount and a classification basis, wherein the classification basis is a set of basis vectors that express, for each disease, a correlation with other diseases using a vector of a predetermined dimension; A process to output the inferred results A program that executes the following.

12. a first machine learning model that outputs, from an image, an image feature that indicates a feature of the image; a second machine learning model that outputs a classification basis learned about correlations between diseases from input data generated based on label information of a plurality of medical images labeled with one or more diseases; a learning unit that trains the first machine learning model and the second machine learning model using a plurality of medical images labeled with one or a plurality of diseases, the learning unit calculating a confidence level for each disease based on the image features acquired from the first machine learning model and the classification basis acquired from the second machine learning model, and training the first machine learning model and the second machine learning model so as to minimize an error between the confidence level and the disease labeled in the medical image, the classification basis being a set of basis vectors that express, for each disease, a correlation with other diseases by a vector of a predetermined dimension; a model output unit that outputs a trained model obtained by training the first machine learning model; a basis output unit that outputs a classification basis obtained by learning the second machine learning model; An information processing device comprising:

13. acquiring a plurality of medical images labeled with one or more diseases and a plurality of medical images without the disease; a first machine learning model and a second machine learning model are trained using the medical images, wherein the first machine learning model is a machine learning model that outputs, from an image, image features that indicate characteristics of the image, and the second machine learning model is a machine learning model that outputs a classification basis learned about correlations between diseases from input data generated based on label information of a plurality of medical images labeled with one or more diseases; a certainty factor for each disease is obtained based on the image features obtained from the first machine learning model and the classification basis obtained from the second machine learning model, and the first machine learning model and the second machine learning model are trained so as to minimize an error between the certainty factor and the disease labeled in the medical image, and the classification basis is a set of basis vectors that express, for each disease, a correlation with other diseases using a vector of a predetermined dimension; outputting a trained model obtained by training the first machine learning model; outputting the classification basis obtained by training the second machine learning model; A method comprising:

Images