Information processing device, information processing method, and computer-readable recording medium

Abstract

Provided are an information processing device, an information processing method, and a computer-readable recording medium that make it possible to present, without the need of learning data in which annotation information designating pixel by pixel an abnormal region in a medical image is imparted, a region having a high probability of being an abnormal region in an image.　The present invention comprises: an image acquisition unit 224 for acquiring a first image obtained by imaging an examination target portion of a patient; an image classification unit 226 for acquiring the degree of certainty that the examination target portion in the first image is healthy and / or the degree of certainty that the examination target portion is unhealthy; an image creation unit 227 for creating a second image in which a region having contributed to classification by the image classification unit is visualized; and an output unit 228 for outputting an inference result on the basis of the degree of certainty acquired by the image classification unit and the second image created by the image creation unit.

Description

Cross - reference to related applications

[0001] This application is based on Japanese Patent Application No. 2021 - 171543 filed on October 20, 2021, the contents of which are incorporated herein by reference.

Technical Field

[0002] The present disclosure relates to an information processing apparatus, an information processing method, and a computer - readable recording medium.

Background Art

[0003] In the medical field, specialists such as medical doctors and radiologists perform annotations to specify abnormal regions in medical images, for example, by drawing contours. Based on the medical images with such annotation information, a machine learning model is trained, and using the trained model obtained by the training, feature amounts of the abnormal regions are extracted.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] In Patent Document 1, based on a fundus image which is an image of the fundus of the eye and a blood circulation abnormal region specified based on a fundus fluorescein angiography image of the fundus of the eye, a diagnostic support apparatus for specifying a blood circulation abnormal region in the fundus image using a trained model that has learned the relationship between the fundus image and the blood circulation abnormal region in the fundus image is disclosed.

[0006] In order to generate a highly accurate trained model, a large number of medical images with accurate annotation information are required. However, in order to obtain accurate annotation information, it is necessary to secure one or more radiologists with high expertise and sufficient working time, which is costly. Furthermore, when providing annotation information for diffuse diseases where lesions are spread over a wide area, it is necessary to point out abnormal areas with unclear boundaries in the image, and even highly skilled radiologists may experience inconsistencies in identifying these abnormal areas.

[0007] Therefore, the present disclosure aims to provide an information processing device, an information processing method, and a computer-readable recording medium that can present regions in an image that have a high probability of being abnormal regions, without requiring training data that has annotation information specifying abnormal regions in a medical image on a pixel-by-pixel basis.

[0008] An information processing device according to one aspect of the present disclosure includes: an image acquisition unit that acquires a first image obtained by photographing a patient's area to be examined; an image classification unit that acquires a confidence level that the area to be examined in the first image is healthy and / or unhealthy; an image creation unit that creates a second image that visualizes the region that contributed to the classification by the image classification unit; and an output unit that outputs an inference result based on the confidence level obtained by the image classification unit and the second image created by the image creation unit.

[0009] According to this embodiment, by using the confidence level acquired by the image classification unit to visualize the regions that significantly contributed to the confidence level that the area under examination is unhealthy, it is possible to present regions in the image that have a high probability of being abnormal. Furthermore, according to this embodiment, regions in the image that have a high probability of being abnormal can be identified without generating a pre-trained model trained using a large amount of training data to which annotation information specifying abnormal regions at the pixel level has been added.

[0010] In the above-described information processing device, the image classification unit inputs a first image into an inference model to obtain a confidence level that the area to be examined is healthy and / or unhealthy. The inference model may be a model that estimates whether the area to be examined contained in the first image is healthy or unhealthy. According to this embodiment, a confidence level with the desired accuracy can be easily obtained.

[0011] In the above-described information processing device, the image creation unit may create a second image by adjusting a value representing the degree of contribution to the classification by the image classification unit based on the confidence level that the area to be examined is abnormal. According to this embodiment, abnormal areas can be accurately identified in images that are predicted to be abnormal.

[0012] Other aspects of the present disclosure include obtaining a first image obtained by photographing a patient's area to be examined; obtaining a confidence level that the area to be examined in the first image is healthy and / or unhealthy; creating a second image that visualizes the region that contributed to the confidence level that the area to be examined is unhealthy; and outputting an inference result based on the obtained confidence levels and the second image.

[0013] A computer-readable recording medium according to another aspect of the present disclosure records a program that causes one or more computers to perform the following processes: acquiring a first image obtained by photographing a patient's area of ​​examination; acquiring a confidence level that the area of ​​examination in the first image is healthy and / or unhealthy; creating a second image that visualizes the region that contributed to the confidence level that the area of ​​examination is unhealthy; and outputting an inference result based on the acquired confidence levels and the second image.

[0014] An information processing device according to another aspect of this disclosure includes a learning unit that inputs images labeled as either healthy or unhealthy as ground truth data into a machine learning model to train the machine learning model, and a model output unit that outputs the trained model trained by the learning unit.

[0015] According to this embodiment, a trained model can be obtained that can be used for inferring diagnoses predicted from images. Furthermore, since images labeled as either "healthy" or "unhealthy" on an image-by-image basis can be used as training data during training, it can be collected more easily compared to conventional training data that required annotation information specifying abnormal regions on a pixel-by-pixel basis.

[0016] In the above-described information processing device, the images may include multiple images labeled as either healthy or unhealthy for multiple diseases, including diffuse diseases. According to this embodiment, a trained model can be obtained that has learned the characteristics of unhealthy individuals across multiple diseases, including diffuse diseases, in which it can be particularly difficult to identify abnormal regions.

[0017] A method relating to another aspect of this disclosure includes acquiring a plurality of images labeled as either healthy or unhealthy as ground truth data, and training a machine learning model using the images, wherein the machine learning model is a machine learning model that estimates whether the body part to be examined included in the image is healthy or unhealthy, and outputting a trained model obtained by training the machine learning model. [Effects of the Invention]

[0018] According to this disclosure, it is possible to provide an information processing device, an information processing method, and a computer-readable recording medium that can present regions in a medical image that have a high probability of being abnormal regions, without requiring training data to which annotation information specifying abnormal regions in a medical image on a pixel-by-pixel basis is attached. [Brief explanation of the drawing]

[0019] [Figure 1] This figure shows the network configuration of an information processing system according to one embodiment. [Figure 2] This is a schematic diagram illustrating the processing of a learning device according to one embodiment. [Figure 3] This is a schematic diagram illustrating the processing of an inference device according to one embodiment. [Figure 4] This is a block diagram of a learning device according to one embodiment. [Figure 5] This is a block diagram of an inference device according to one embodiment. [Figure 6] This figure shows the learning process of a learning device according to one embodiment. [Figure 7]It is a flowchart showing the inference process of an inference device according to an embodiment. [Figure 8] It is a diagram showing a function used to adjust the density of a heatmap according to an embodiment. [Figure 9] It is a schematic diagram explaining the process of a learning device according to an embodiment. [Figure 10] It is a flowchart showing the inference process of an inference device according to an embodiment.

Embodiments for Carrying Out the Invention

[0020] Embodiments of the present invention will be described with reference to the accompanying drawings. Note that the following embodiments are for facilitating the understanding of the present invention and are not for limiting the interpretation of the present invention. Further, the present invention can be variously modified without departing from its gist. Furthermore, those skilled in the art can adopt embodiments in which each element described below is replaced with an equivalent one, and such embodiments are also included in the scope of the present invention.

[0021] (System Configuration) The outline of the present disclosure will be described using FIGS. 1, 2, and 3. FIG. 1 is a diagram showing the network configuration of an information processing system according to an embodiment. FIG. 2 is a schematic diagram explaining the process of a learning device according to an embodiment. FIG. 3 is a schematic diagram explaining the process of an inference device according to an embodiment.

[0022] The information processing system includes a learning device 10, an inference device 20, and a storage device 30. The learning device 10 is connected to the inference device 20 and the storage device 30 via a communication network N. The communication network N may be either a wired communication network or a wireless communication network constituted by a wired or wireless line, and may be the Internet or a Local Area Network (LAN).

[0023] The learning device 10 trains a machine learning model based on the training data stored in the storage device 30 and stores the trained model in the storage device 30. In this embodiment, the learning device 10 includes a machine learning model, but the machine learning model may be stored in a separate device from the learning device 10.

[0024] Here, a machine learning model is a model that has a predetermined model structure and processing parameters that change through the learning process, and whose classification accuracy improves as the processing parameters are optimized based on the experience gained from the training data. In other words, a machine learning model is a model that learns the optimal processing parameters through the learning process. The algorithm of a machine learning model can be, for example, a support vector machine, logistic regression, or a neural network, but the type is not particularly limited. The machine learning model undergoing this learning includes both models that have not yet been trained and models that have already undergone some kind of learning using the training data.

[0025] A pre-trained model is a machine learning model that has been trained using appropriate training data in advance, based on any machine learning algorithm. However, a pre-trained model is not limited to further training; additional training can be performed on it.

[0026] The inference device 20 uses a trained model to output output data corresponding to the characteristics of the input data. In this embodiment, the inference device 20 performs inference using a trained model acquired from the storage device 30. Here, acquiring a trained model means acquiring the information necessary to reproduce the function of the trained model in the inference device 20. For example, when using a neural network as a machine learning model, acquiring a trained model means acquiring at least information regarding the number of layers in the neural network, the number of nodes in each layer, the weight parameters of the links connecting the nodes, the bias parameters for each node, and the functional form of the activation function for each node.

[0027] The storage device 30 stores training data used to train a machine learning model. In this embodiment, the storage device 30 stores fundus images labeled as either "healthy" or "unhealthy" as training data. The storage device 30 also stores the trained model output by the learning device 10. In Figure 1, the storage device 30 is shown as a single storage device, but the storage device 30 may be composed of one or more file servers. In this embodiment, fundus images labeled as either "healthy" or "unhealthy" are used as an example of training data, but in another embodiment, medical images obtained by photographing other areas of the patient to be examined and labeled as either "healthy" or "unhealthy" can also be used as training data.

[0028] Here, the learning device 10 according to this embodiment, as shown in Figure 2, includes a machine learning model that receives fundus images as input data and classifies them into images of a healthy eye and images of an unhealthy eye. The learning device 10 uses the machine learning model to classify the fundus images and trains the machine learning model to minimize the error between the predicted result and the ground truth data labeled in the training data. In this embodiment, an example using fundus images, which are an example of medical images, is described, but in another embodiment, images of the kidneys, livers, etc., obtained by photographing other parts of the patient's body to be examined can be used.

[0029] Since fundus images labeled as either "healthy" or "unhealthy" on an image-by-image basis can be used as training data, it can be collected more easily compared to conventional training data that required annotation information specifying abnormal regions on a pixel-by-pixel basis.

[0030] As shown in Figure 3, the inference device 20 according to this embodiment performs forward calculations using a trained model to infer whether the fundus image belongs to a healthy eye or an unhealthy eye. Next, the inference device 20 performs backpropagation from the output layer corresponding to the unhealthy eye to the convolutional layer to be visualized, calculates the contribution of each feature map to the output of the unhealthy eye, and creates a heatmap by weighting the contributions of the feature maps obtained in the forward calculations and adding them together.

[0031] By using a trained model that has learned about the characteristics of unhealthy eyes from a large number of images of healthy and unhealthy eyes, it is possible to visualize the regions that significantly contributed to the output of unhealthy eyes, thereby presenting abnormal regions in unhealthy eyes that are difficult to define clearly across multiple diseases. Furthermore, according to the inference device 20 of this embodiment, it is possible to identify regions in fundus images that have a high probability of being abnormal without generating a trained model that has been trained using a large amount of training data to which annotation information specifying abnormal regions at the pixel level has been added. In this embodiment, abnormal regions in fundus images, which are an example of medical images, are presented, but in another embodiment, abnormal regions in medical images such as images of the kidneys or liver, which are obtained by photographing other parts of the patient's body that are subject to examination, can be presented.

[0032] (Functional configuration: Learning device) Figure 4 is a block diagram of a learning device according to one embodiment. In Figure 4, only the necessary functional configuration is shown, assuming a single learning device 10. However, the learning device 10 can also be configured as part of a multifunctional distributed system consisting of multiple computer systems.

[0033] The learning device 10 includes an input unit 110, a control unit 120, a storage unit 130, and a communication unit 140.

[0034] The input unit 110 is configured to receive operations from the administrator of the learning device 10, and this can be achieved using a keyboard, mouse, touch panel, or the like.

[0035] The control unit 120 includes an arithmetic processing unit 121, such as a CPU or MPU, which corresponds to a processor, and a memory 122, such as RAM. The arithmetic processing unit 121 (processor) realizes the functions and processes described later in the arithmetic processing unit 121 by loading the program recorded in the storage unit 130 into the memory 122 and executing it based on various inputs. This program may be stored on a computer-readable non-temporary recording medium such as a CD-ROM, or distributed via a network and installed on a computer. The memory 122 functions as work memory necessary for program execution by the arithmetic processing unit 121 (processor).

[0036] The storage unit 130 is composed of a storage device such as a hard disk and stores various programs necessary for executing processing in the control unit 120, as well as data necessary for executing various programs. In this embodiment, it is desirable that the storage unit 130 has a learning data storage unit 131.

[0037] The learning data storage unit 131 stores the learning data used to train the machine learning model M described later. In this embodiment, the learning data storage unit 131 stores fundus images labeled as either "healthy" or "unhealthy" as the correct answer. In this embodiment, the learning data storage unit 131 stores multiple fundus images labeled as either "healthy" or "unhealthy" as the correct answer for multiple diseases, including diffuse diseases. However, in other embodiments, multiple fundus images labeled as either "healthy" or "unhealthy" as the correct answer for a single diffuse disease may be stored, or multiple fundus images labeled as either "healthy" or "unhealthy" as the correct answer for multiple diseases that do not include diffuse diseases may be stored.

[0038] The communication unit 140 is configured to connect the learning device 10 to a network. For example, the communication unit 140 can be implemented using a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0039] Furthermore, as shown in Figure 4, the arithmetic processing unit 121 includes a learning data acquisition unit 123, a learning unit 124, an image classification unit 125, and a model output unit 126 as functional units.

[0040] The learning data acquisition unit 123 acquires learning data to be used for training the machine learning model M described later and stores it in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires fundus images labeled as either "healthy" or "unhealthy" as the correct answer data from the storage device 30 and stores them in the learning data storage unit 131.

[0041] The learning unit 124 trains the machine learning model M using the learning data acquired by the learning data acquisition unit 123. In this embodiment, the learning unit 124 inputs fundus images labeled as either "healthy" or "unhealthy" as the correct answer data into the image classification unit 125 (described later) to train the machine learning model M.

[0042] The image classification unit 125 receives fundus image input and classifies it into images of healthy eyes and images of unhealthy eyes. In this embodiment, the image classification unit 125 receives fundus image input and uses a machine learning model M to output confidence levels for healthy eyes and unhealthy eyes.

[0043] The machine learning model M is a machine learning model that receives fundus images as input data and classifies them into images of healthy eyes and images of unhealthy eyes. In this embodiment, as an example of the machine learning model M, an embodiment using a convolutional neural network (CNN) that receives fundus images as input data and outputs image classification will be described. However, CNN is only one example of the machine learning model M, and the learning device 10 may use other configurations as the machine learning model M.

[0044] The learning unit 124 trains the machine learning model M to minimize the error between the result predicted by the machine learning model M and the ground truth data labeled on the training data.

[0045] Once the training of the machine learning model M is complete, the model output unit 126 outputs the trained model obtained by training the machine learning model M to the storage device 30. The training unit 124 may, for example, complete the training after training the machine learning model M using a predetermined number of training data, or it may complete the training when the accuracy of the classification predicted using the machine learning model M meets predetermined conditions.

[0046] (Functional configuration: Inference device) Figure 5 is a block diagram of an inference device according to one embodiment. In Figure 5, only the necessary functional configuration is shown, assuming a single inference device 20. However, the inference device 20 can also be configured as part of a multifunctional distributed system consisting of multiple computer systems.

[0047] The inference device 20 includes an input unit 210, a control unit 220, a storage unit 230, and a communication unit 240.

[0048] The input unit 210 is configured to accept operations from the administrator of the inference device 20, and this can be achieved using a keyboard, mouse, touch panel, or the like.

[0049] The control unit 220 includes an arithmetic processing unit 221, such as a CPU or MPU, which corresponds to a processor, and a memory 222, such as RAM. The arithmetic processing unit 221 (processor) realizes the functions and processes described later in the arithmetic processing unit 221 by loading the program recorded in the storage unit 230 into the memory 222 and executing it based on various inputs. This program may be stored on a computer-readable non-temporary recording medium such as a CD-ROM, or distributed via a network and installed on a computer. The memory 222 functions as work memory necessary for program execution by the arithmetic processing unit 221 (processor).

[0050] The storage unit 230 is composed of a storage device such as a hard disk and stores various programs necessary for executing processing in the control unit 220, as well as data necessary for executing various programs. In this embodiment, it is desirable that the storage unit 230 has an image storage unit 231 and a trained model 232.

[0051] The image storage unit 231 stores the image to be inferred. In this embodiment, the image storage unit 231 stores the fundus image used for diagnostic inference.

[0052] The trained model 232 stores the trained model used for inference. In this embodiment, the trained model 232 stores a trained model that receives fundus images as input data and classifies them into images of healthy eyes and images of unhealthy eyes. In this embodiment, as an example of the trained model 232, an example using a convolutional neural network (CNN) that receives fundus images as input data and classifies them into images of healthy eyes and images of unhealthy eyes will be described. However, CNN is only one example of the trained model 232, and the inference device 20 may use other configurations as the trained model 232.

[0053] The communication unit 240 is configured to connect the inference device 20 to a network. For example, the communication unit 240 can be implemented from a LAN card, an analog modem, an ISDN modem, etc., and an interface for connecting these to the processing unit via a transmission path such as a system bus.

[0054] Furthermore, as shown in Figure 5, the arithmetic processing unit 221 includes, as functional units, a model acquisition unit 223, an image acquisition unit 224, an inference unit 225, an image classification unit 226, a heatmap creation unit 227, and an output unit 228.

[0055] The model acquisition unit 223 acquires a trained model to be used for inference and stores it in the trained model 232. In this embodiment, the model acquisition unit 223 acquires a trained model from the storage device 30 and stores it in the trained model 232.

[0056] The image acquisition unit 224 acquires the image to be used for inference. In this embodiment, the image acquisition unit 224 acquires the fundus image used for diagnostic inference from the image storage unit 231.

[0057] The inference unit 225 infers a diagnosis predicted from the images acquired by the image acquisition unit 224. In this embodiment, the inference unit 225 is composed of an image classification unit 226 and a heatmap creation unit 227. First, the inference unit 225 inputs the fundus image to the image classification unit 226 and obtains the confidence level of healthy eyes and unhealthy eyes from the image classification unit 226.

[0058] The image classification unit 226 receives fundus image input and classifies it into images of healthy eyes and images of unhealthy eyes. In this embodiment, as shown in Figure 3, the image classification unit 226 receives fundus image input, performs forward calculations using the trained model 232, and outputs confidence scores for healthy and unhealthy eyes.

[0059] The heatmap creation unit 227 creates a heatmap that visualizes the regions that contributed to the classification performed by the image classification unit 226. In this embodiment, as shown in Figure 3, when the confidence scores for healthy and unhealthy eyes are output, the heatmap creation unit 227 performs backpropagation from the output layer corresponding to the unhealthy eye to the convolutional layer to be visualized, calculates the gradient of the feature map relative to the output of the unhealthy eye in order to calculate the contribution of each feature map to the output of the unhealthy eye, and takes the Global Max Pooling (GMP) of the gradient. Next, the heatmap creation unit 227 weights the feature map obtained in the forward calculation with GMP and obtains a coefficient map by adding all the feature maps together.

[0060] In one embodiment, the heatmap creation unit 227 adjusts the values ​​of each element in the coefficient map based on the confidence level of the non-healthy eye. For example, as shown in Figure 8, the heatmap creation unit 227 may set the values ​​of each element in the coefficient map to zero when the confidence level of the non-healthy eye is 0.0 to 0.3, adjust the values ​​of each element in proportion to the confidence level of the non-healthy eye when the confidence level of the non-healthy eye is 0.3 to 0.6, and set the values ​​of each element to 100%, i.e., their original values, when the confidence level of the non-healthy eye is 0.6 to 1.0. By reducing the values ​​of each element in the coefficient map when the confidence level of the non-healthy eye is low, and maintaining the values ​​of each element in the coefficient map when the confidence level is high, abnormal areas can be accurately identified in images predicted to be non-healthy eyes.

[0061] Finally, the heatmap creation unit 227 creates a heatmap by converting the coefficient map into an image using a color scale and resizing the resulting image to the size of the input image.

[0062] This embodiment describes an example of creating a heatmap using Grad-CAM (Gradient-weighted Class Activation Mapping), but heatmaps may be obtained using other visualization methods. Furthermore, this embodiment creates a heatmap that represents the degree of contribution using a color scale, but visualization maps that represent the contribution in other formats may also be created.

[0063] The output unit 228 outputs the inference result based on the information acquired by the inference unit 225. In this embodiment, the output unit 228 outputs the fundus image acquired by the image acquisition unit 224 with a heatmap superimposed on it using alpha blending.

[0064] (Learning process) Referring to Figure 6, the learning process of a learning device according to one embodiment will be described in detail. In this embodiment, it is assumed that learning data is stored in the storage device 30 under the management of the administrator of the learning device 10 before the learning process described in Figure 6 is performed. The process shown in Figure 6 is performed, for example, by the administrator inputting an instruction via the input unit 110 to execute the process of generating a trained model.

[0065] In step S601, the learning data acquisition unit 123 of the learning device 10 acquires learning data to be used for training the machine learning model M and stores it in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires fundus images labeled as either "healthy" or "unhealthy" as the correct answer data from the storage device 30 and stores them in the learning data storage unit 131. In this embodiment, the learning data acquisition unit 123 acquires multiple fundus images labeled as either "healthy" or "unhealthy" as the correct answer data for multiple diseases, including diffuse diseases, and stores them in the learning data storage unit 131. That is, if the fundus image corresponds to one or more of the multiple diseases, including diffuse diseases, the fundus image is labeled as "unhealthy" as the correct answer data.

[0066] Next, in step S602, the learning unit 124 of the learning device 10 trains the machine learning model M using the learning data acquired by the learning data acquisition unit 123. In this embodiment, the learning unit 124 inputs fundus images labeled as either "healthy" or "unhealthy" as the correct answer data into the image classification unit 125 of the learning device 10 to train the machine learning model M.

[0067] The machine learning model M is a machine learning model that receives fundus images as input data and classifies them into images of healthy eyes and images of unhealthy eyes. In this embodiment, as an example of the machine learning model M, an embodiment using a convolutional neural network (CNN) that receives fundus images as input data and outputs image classification will be described. However, CNN is only one example of the machine learning model M, and the learning device 10 may use other configurations as the machine learning model M.

[0068] The learning unit 124 trains the machine learning model M to minimize the error between the result predicted by the machine learning model M and the ground truth data labeled on the training data.

[0069] When the training of the machine learning model M is complete, in step S603, the model output unit 126 of the learning device 10 outputs the trained model obtained by training the machine learning model M to the storage device 30. The learning unit 124 may, for example, complete the training after training the machine learning model M using a predetermined number of training data, or it may complete the training when the accuracy of the classification predicted using the machine learning model M satisfies predetermined conditions.

[0070] (Inference processing) Referring to Figure 7, the inference process of an inference device according to one embodiment will be described in detail. In this embodiment, before performing the inference process described in Figure 7, it is assumed that the trained model acquired from the storage device 30 is stored in the trained model 232 under the management of the administrator of the inference device 20. Also, it is assumed that the fundus image to be inferred is stored in the image storage unit 231 of the inference device 20. Note that the process shown in Figure 7 is executed, for example, when the administrator inputs instructions to execute the inference process via the input unit 210.

[0071] In step S701, the image acquisition unit 224 of the inference device 20 acquires the image to be inferred. In this embodiment, the image acquisition unit 224 acquires the fundus image for which diagnostic inference is performed from the image storage unit 231.

[0072] In step S702, the inference unit 225 of the inference device 20 obtains the confidence scores for healthy and unhealthy eyes. In this embodiment, the inference unit 225 inputs the fundus image to the image classification unit 226 of the inference device 20 and obtains the confidence scores for healthy and unhealthy eyes from the image classification unit 226. Specifically, the image classification unit 226 receives the fundus image as input, performs forward calculations using the trained model 232, and outputs the confidence scores for healthy and unhealthy eyes.

[0073] In this embodiment, a convolutional neural network (CNN) is used as an example of a trained model 232, which receives fundus images as input data and classifies them into images of healthy eyes and images of unhealthy eyes. However, the CNN is only one example of a trained model 232, and the inference device 20 may use other configurations as the trained model 232.

[0074] Once the confidence scores for the healthy eye and the unhealthy eye are output, in step S703, the heatmap creation unit 227 of the inference device 20 performs backpropagation from the output layer corresponding to the unhealthy eye to the convolutional layer to be visualized, calculates the gradient of the feature map relative to the output of the unhealthy eye, and takes the Global Max Pooling (GMP) of the gradient in order to calculate the contribution of each feature map to the output of the unhealthy eye.

[0075] In step S704, the heatmap creation unit 227 weights the feature maps obtained in the forward calculation by GMP and obtains a coefficient map by adding all the feature maps together. Then, in step S705, the heatmap creation unit 227 adjusts the values ​​of each element of the coefficient map based on the confidence level of the non-healthy eye.

[0076] In this embodiment, as shown in Figure 8, the heat map creation unit 227 sets the value of each element of the coefficient map to zero when the confidence level of the non-healthy eye is 0.0 to 0.3, adjusts the value of each element in proportion to the confidence level of the non-healthy eye when the confidence level of the non-healthy eye is 0.3 to 0.6, and adjusts the value of each element to 100%, i.e., the original value, when the confidence level of the non-healthy eye is 0.6 to 1.0. By reducing the value of each element of the coefficient map when the confidence level of the non-healthy eye is low and maintaining the value of each element of the coefficient map when the confidence level is high, abnormal areas can be accurately identified in images predicted to be non-healthy eyes.

[0077] Furthermore, in step S706, the heatmap creation unit 227 creates a heatmap by imaging the adjusted coefficient map using a color scale and resizing the obtained image to the size of the input image.

[0078] In this embodiment, an example of creating a heatmap using Grad-CAM (Gradient-weighted Class Activation Mapping) has been described, but heatmaps may also be obtained using other visualization methods. Furthermore, in this embodiment, a heatmap is created that represents the degree of contribution using a color scale, but visualization maps that represent the contribution in other formats may also be created.

[0079] In step S707, the output unit 228 of the inference device 20 outputs an inference result based on the information acquired by the inference unit 225. In this embodiment, the output unit 228 outputs a heatmap superimposed on the fundus image acquired by the image acquisition unit 224 using alpha blending.

[0080] As described above, according to this embodiment, the learning device 10 can obtain a trained model that can be used for inferring a diagnosis predicted from fundus images. Furthermore, since fundus images labeled as either "healthy" or "unhealthy" on an image-by-image basis can be used as training data during training, it can be collected more easily compared to conventional training data that required annotation information specifying abnormal regions on a pixel-by-pixel basis.

[0081] Furthermore, according to this embodiment, the inference device 20 can visualize the regions that significantly contributed to the output of the non-healthy eye by using a trained model that has learned about the characteristics of non-healthy eyes from a large number of images of healthy and non-healthy eyes. This makes it possible to present abnormal regions in non-healthy eyes that are difficult to define clearly across multiple diseases. Moreover, according to the inference device 20 of this embodiment, it is possible to grasp regions in fundus images that have a high probability of being abnormal without generating a trained model that has been trained using a large amount of training data to which annotation information specifying abnormal regions at the pixel level has been added.

[0082] In this embodiment, an abnormal region in a fundus image, which is an example of a medical image, is shown. However, in another embodiment, abnormal regions in medical images such as images of the kidney or liver, obtained by imaging other parts of the patient's body, can be shown.

[0083] (Another embodiment) Up to this point, we have described an example in which inference processing is performed using a single trained model obtained by training one machine learning model using images labeled as either healthy or unhealthy for multiple diseases as training data. In the following embodiment, we will describe an example in which inference processing is performed using multiple trained models M-1 to MN, each of which is obtained by training one machine learning model for each of multiple diseases using images labeled as either healthy or unhealthy as training data.

[0084] As shown in Figure 9, the learning device 10 according to this embodiment includes multiple machine learning models that receive fundus images as input data and classify them into images of healthy eyes and images of unhealthy eyes (diseases). The learning device 10 classifies fundus images using each machine learning model and trains each machine learning model to minimize the error between the predicted result and the ground truth data labeled in the training data. For example, the image classification unit 125-1 is a machine learning model that learns about the characteristics of glaucoma and classifies images of healthy eyes and images of unhealthy eyes (glaucoma). The image classification unit 125-2 is a machine learning model that learns about the characteristics of diabetic retinopathy and classifies images of healthy eyes and images of unhealthy eyes (diabetic retinopathy).

[0085] The learning data storage unit 131 stores the learning data used for training multiple machine learning models M-1 to MN. In this embodiment, the learning data storage unit 131 stores fundus images labeled as either "healthy" or "unhealthy (glaucoma)" as the ground truth data used for training machine learning model M-1, fundus images labeled as either "healthy" or "unhealthy (diabetic retinopathy)" as the ground truth data used for training machine learning model M-2, ..., fundus images labeled as either "healthy" or "unhealthy (disease N)" as the ground truth data used for training machine learning model MN.

[0086] The inference device 20 according to this embodiment performs forward calculations using multiple trained models to infer whether a fundus image belongs to a healthy eye or an unhealthy eye. The inference device 20 is equipped with multiple trained models 232-1 to 232-N that have learned the features of each of multiple diseases, and obtains the confidence level of a healthy eye and the confidence level of an unhealthy eye (disease) for each disease. In this embodiment, the inference unit 225 is composed of multiple image classification units 226-1 to 226-N and multiple heatmap creation units 227-1 to 227-N corresponding to each of the multiple image classification units. First, the inference unit 225 inputs fundus images to the multiple image classification units 226-1 to 226-N and obtains the confidence level of a healthy eye and an unhealthy eye (disease) from each image classification unit 226.

[0087] Next, the inference unit 225 selects one or more image classification units 226 from among the multiple image classification units 226-1 to 226-N based on the confidence levels obtained from the multiple image classification units 226-1 to 226-N, and instructs the corresponding heatmap creation unit 227 to create a heatmap that visualizes the regions that contributed to the classification of the selected image classification unit 226. In this embodiment, the inference unit 225 selects one image classification unit 226 that has the highest confidence level for non-healthy eyes (diseases) among the confidence levels for healthy eyes and non-healthy eyes (diseases) obtained from each image classification unit 226, and causes the heatmap creation unit 227 corresponding to the selected image classification unit 226 to create a heatmap.

[0088] In this embodiment, the inference unit 225 selects one image classification unit 226. However, in other embodiments, the inference unit 225 may select one or more image classification units 226 that have a confidence level of non-healthy eyes (diseases) of a predetermined threshold, such as 0.5, or it may select a predetermined number of image classification units 226 that have a higher confidence level of non-healthy eyes (diseases).

[0089] (Learning process) The training process is the same as that described in Figure 6 for training a single machine learning model; therefore, we will omit the explanation here.

[0090] (Inference processing) Referring to Figure 10, the inference process of an inference device according to one embodiment will be described in detail. In this embodiment, before performing the inference process described in Figure 10, it is assumed that, under the management of the administrator of the inference device 20, a plurality of trained models 232-1 to 232-N acquired from the storage device 30 are stored in the trained model 232. Also, it is assumed that the fundus image to be inferred is stored in the image storage unit 231 of the inference device 20. Note that the process shown in Figure 10 is executed, for example, by the administrator inputting instructions to execute the inference process via the input unit 210.

[0091] In step S1001, the image acquisition unit 224 acquires the image to be inferred. In this embodiment, the image acquisition unit 224 acquires the fundus image used for diagnostic inference from the image storage unit 231.

[0092] In step S1002, the inference unit 225 inputs fundus images to multiple image classification units 226-1 to 226-N and obtains the confidence level of healthy eyes and unhealthy eyes (diseases) from each image classification unit 226.

[0093] When the confidence levels for healthy eyes and unhealthy eyes (diseases) are output, in step S1003, the inference unit 225 selects one or more image classification units 226-1 to 226-N from among the multiple image classification units 226-1 to 226-N based on the confidence levels obtained from the multiple image classification units 226-1 to 226-N. In this embodiment, the inference unit 225 selects the image classification unit 226 that has the highest confidence level for unhealthy eyes (diseases) among the confidence levels for healthy eyes and unhealthy eyes (diseases) obtained from each image classification unit 226.

[0094] In step S1004, the heatmap creation unit 227 performs backpropagation from the output layer corresponding to the non-healthy eye (disease) of the selected image classification unit 226 to the convolutional layer to be visualized. To calculate the contribution of each feature map to the output of the non-healthy eye (disease), the heatmap creation unit 227 calculates the gradient of the feature map with respect to the output of the non-healthy eye (disease) and takes the Global Max Pooling (GMP) of the gradient.

[0095] In step S1005, the heatmap creation unit 227 weights the feature maps obtained in the forward calculation by GMP and obtains a coefficient map by adding all the feature maps together. Then, in step S1006, the heatmap creation unit 227 adjusts the values ​​of each element of the coefficient map based on the confidence level of the non-healthy eye (disease).

[0096] In this embodiment, as shown in Figure 8, the heat map creation unit 227 sets the value of each element of the coefficient map to zero when the confidence level of the non-healthy eye (disease) is 0.0 to 0.3, adjusts the value of each element in proportion to the confidence level of the non-healthy eye (disease) when the confidence level of the non-healthy eye (disease) is 0.3 to 0.6, and adjusts the value of each element to 100%, i.e., the original value, when the confidence level of the non-healthy eye (disease) is 0.6 to 1.0.

[0097] Furthermore, in step S1007, the heatmap creation unit 227 creates a heatmap by imaging the adjusted coefficient map using a color scale and resizing the obtained image to the size of the input image.

[0098] In step S1008, the output unit 228 outputs the inference result based on the information acquired by the inference unit 225. In this embodiment, the output unit 228 outputs the fundus image acquired by the image acquisition unit 224 with a heatmap superimposed by alpha blending.

[0099] As described above, according to this embodiment, the learning device 10 can obtain multiple trained models, each trained to learn the characteristics of a specific disease, which can be used for inferring a diagnosis predicted from fundus images. Furthermore, since fundus images labeled as either "healthy" or "unhealthy" on an image-by-image basis can be used as training data during training, it can be collected more easily compared to conventional training data that required annotation information specifying abnormal regions on a pixel-by-pixel basis.

[0100] Furthermore, according to this embodiment, the inference device 20 can present abnormal regions in non-healthy eyes, which are difficult to define clearly, by visualizing the regions that significantly contributed to the output of non-healthy eyes (diseases) in the image classification unit, which has a high degree of confidence in the disease, using multiple trained models, each of which has learned the characteristics of a specific disease. Moreover, according to the inference device 20 of this embodiment, it is possible to grasp regions in fundus images that have a high probability of being abnormal regions without generating a trained model trained using a large amount of training data to which annotation information specifying abnormal regions at the pixel level has been added.

[0101] An information processing device according to one aspect of this disclosure comprises a plurality of image classification units, and one or more image classification units are selected based on the confidence levels obtained from the plurality of image classification units, and an image creation unit may create one or more second images that visualize the regions that contributed to the classification of the selected image classification units. According to this aspect, the information processing device can use the confidence levels of a plurality of different abnormal features to present regions in the image that have a high probability of being abnormal due to the selected abnormal features.

[0102] In the above-described information processing device, each of the multiple image classification units obtains a confidence level that the area to be examined is healthy and / or has a specific disease by inputting the first image into an inference model. The inference model of each image classification unit may be a model that infers whether the area to be examined included in the first image is healthy or has a specific disease. According to this embodiment, the information processing device can easily obtain a confidence level with a desired accuracy for each of the confidence levels that indicate multiple different diseases. [Explanation of Symbols]

[0103] 10...Learning device, 110...Input unit, 120...Control unit, 121...Calculation processing unit, 122...Memory, 123...Learning data acquisition unit, 124...Learning unit, 125...Image classification unit, 126...Model output unit, 130...Storage unit, 131...Learning data storage unit, 140...Communication unit, 20...Inference device, 210...Input unit, 220...Control unit, 221...Calculation processing unit, 222...Memory, 223...Model acquisition unit, 224...Image acquisition unit, 225...Inference unit, 226...Image classification unit, 227...Heatmap creation unit (image creation unit), 228...Output unit, 230...Storage unit, 231...Image storage unit, 232...Trained model, 240...Communication unit, 30...Storage device, M...Machine learning model, N...Communication network

Claims

1. An information processing device for presenting a region in a first image that has a high probability of being a specific region of the body part being examined by the examinee, An image acquisition unit that acquires the first image, An image classification unit that inputs the first image into an inference model to obtain a confidence level that the subject area of ​​the first image is healthy and / or unhealthy, wherein the inference model is a model obtained using a plurality of images in which either healthy or unhealthy is labeled as the correct answer data on an image-by-image basis, An image creation unit creates a second image that visualizes the regions that contributed to the classification of the aforementioned image classification unit, An output unit that outputs an inference result based on the confidence level obtained by the image classification unit and the second image created by the image creation unit. An information processing device equipped with the following features.

2. The information processing device comprises a plurality of image classification units, and selects one or more image classification units based on the confidence level obtained from the plurality of image classification units. The image creation unit creates one or more second images that visualize the regions that contributed to the classification of the selected image classification unit. The information processing apparatus according to claim 1.

3. Each of the multiple image classification units inputs the first image into an inference model to obtain a degree of confidence that the area to be examined is healthy and / or has a specific disease. The inference model in each image classification unit is a model that infers whether the area to be examined included in the first image is healthy or has the specific disease. The information processing apparatus according to claim 2.

5. The information processing apparatus according to claim 1 or 2, wherein the image creation unit adjusts a value representing the degree of contribution to the classification by the image classification unit based on the degree of confidence that the area to be examined is unhealthy, and creates the second image.

6. A method for presenting a region in the first image that has a high probability of being a specific area related to the body part being examined by the examinee, To obtain the first image, The method involves inputting the first image into an inference model to obtain the confidence level that the subject area of ​​the first image is healthy and / or unhealthy, wherein the inference model is a model obtained using multiple images in which either healthy or unhealthy is labeled as the correct answer data on an image-by-image basis. To create a second image that visualizes the region that contributed to the confidence that the subject area being examined was unhealthy, The inference result is output based on the acquired confidence level and the second image. A method that includes this.

7. A computer-readable recording medium containing a program that causes one or more computers to display a region in a first image that has a high probability of being a specific region relating to the body part being examined of the examinee, The process of acquiring the first image, A process to obtain the confidence level that the subject area of ​​the first image is healthy and / or unhealthy by inputting the first image into an inference model, wherein the inference model is This model is obtained using multiple images, each labeled as either healthy or unhealthy as the correct data, and involves processing and... A process to create a second image that visualizes the region that contributed to the confidence that the subject area being examined is unhealthy, A process to output an inference result based on the acquired confidence level and the second image. A computer-readable recording medium containing a program that executes a program.

8. An information processing device for generating a trained model to be used in an information processing device for presenting regions in an image that have a high probability of being a specific region related to the body part being examined by the examinee, A learning unit that inputs images labeled as either healthy or unhealthy as ground truth data on an image-by-image basis into a machine learning model to train the machine learning model, A model output unit that outputs the trained model learned by the aforementioned learning unit and An information processing device equipped with the following features.

9. The information processing apparatus according to claim 8, wherein the aforementioned images include a plurality of images labeled as either healthy or unhealthy for a plurality of diseases, including diffuse diseases.

10. A method for generating a trained model to be used in an information processing device for presenting regions in an image that have a high probability of being a specific region related to the body part being examined by a test subject, This involves acquiring multiple images, each labeled as either healthy or unhealthy, as the correct data, The method involves training a machine learning model using the aforementioned image, wherein the machine learning model estimates whether the area to be examined included in the image is healthy or unhealthy. The trained model obtained by training the aforementioned machine learning model is output. A method that includes this.

Images