Disease prediction basis display method and program

Abstract

To easily confirm a basis of prediction in the disease prediction using an inspection value and an inspection image.SOLUTION: A disease prediction basis display device predicts a disease by inputting an inspection value and an inspection image of each of multiple inspection items to a machine learning model that predicts a disease with the inspection value and the inspection image of each of the multiple inspection items as the input and has an integration layer that integrates the feature of the inspection value with the feature of the inspection image on the basis of correlation between the feature of the inspection value and the feature of the inspection image, performs correlation with the inspection item for each of the inspection items that becomes a basis of prediction with back propagation of the gradient from a node of the integration layer that has contributed to prediction, specifies a region on the inspection image that becomes the basis of prediction, and displays the inspection item and the region that become the basis of prediction and are correlated with each other in a manner of being associated with each other.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The disclosed technology relates to a disease prediction basis display method and a disease prediction basis display program. [Background technology]

[0002] In recent years, in the medical field, in order to support doctors' diagnoses, technologies have been proposed that use machine learning models to automatically predict diseases from test images such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging). Furthermore, when predicting diseases from test images using machine learning models, technologies have also been proposed that show which areas on the test image were the basis for the prediction using a heat map or the like on the test image. In this way, if the validity of the prediction results of the machine learning model can be judged from a heat map or the like, doctors can determine whether or not to refer to the prediction results of the machine learning model.

[0003] For example, a method has been proposed for cardiac segmentation using a magnetic resonance imaging system, in which a patient's heart is scanned over an extended period of time using a magnetic resonance imaging system, which detects regions indicative of the heart in the scan data using a machine learning network that is a convolutional network with a long short-term memory network, and generates an image containing information that is a function of the detected regions indicative of the heart.

[0004] In addition, a medical image processing device has been proposed that allows an observer to grasp the areas that contributed to classification in automatic classification of medical images. This device acquires captured images generated by capturing images of a living body, classifies the captured images into two or more classes, generates region images that visualize the positions of the areas that contributed to the classification in the captured images to be classified, and transmits the region images to a display device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2019 / 223725 [Patent Document 2] International Publication No. 2020 / 012872 Summary of the Invention [Problem to be solved by the invention]

[0006] When doctors judge diseases, they make a comprehensive judgment by looking at not only test images such as CT and MRI, but also test values ​​from blood tests, etc. Conventional technology can show the areas on the test images that contributed to the prediction of the disease as the basis for the prediction, but from the perspective of a comprehensive judgment that includes test items other than the test images, it is sometimes difficult to easily confirm the basis for the prediction made by the machine learning model.

[0007] In one aspect, the disclosed technology aims to make it possible to easily confirm the basis for disease prediction using test values ​​and test images. [Means for solving the problem]

[0008] In one aspect, the disclosed technology predicts a disease by inputting test values ​​and test images of a plurality of test items into a machine learning model. The machine learning model predicts a disease by inputting test values ​​and test images of a plurality of test items, and has a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images. The disclosed technology also identifies, for each of the test items that are the basis of the prediction, an area on the test image that is correlated with the test items and that is the basis of the prediction, by backpropagating a gradient from a node of the fusion layer that contributed to the prediction. The disclosed technology then displays the test items and the areas that are the basis of the prediction and are correlated with each other. Effect of the Invention

[0009] One aspect of the present invention is that it has an effect of making it possible to easily confirm the basis for disease prediction using test values ​​and test images. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the relationship between diseases and phenomena. [Diagram 2] 1A and 1B are diagrams for explaining a phenomenon that appears in an inspection image. [Diagram 3] FIG. 13 is a diagram showing an example of a relationship between an inspection item and a phenomenon. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of a heat map showing prediction grounds according to the prior art. [Diagram 5] FIG. 1 is a diagram showing an outline of the present embodiment. [Figure 6] FIG. 2 is a functional block diagram of the disease prediction basis display device. [Figure 7] FIG. 2 is a diagram illustrating an example of a configuration of a prediction model. [Figure 8] FIG. 1 is a diagram for explaining disease prediction using a prediction model and identification of test items that contributed to the prediction. [Figure 9] FIG. 2 is a schematic diagram showing an example of an inspection item CAM and an inspection image CAM. [Figure 10] 13 is a diagram for explaining calculation of prediction grounds for an inspection image correlated with an inspection item. FIG. [Figure 11] FIG. 13 is a schematic diagram showing an example of prediction grounds for an inspection image correlated with an inspection item. [Figure 12] FIG. 13 is a diagram showing an example of screen transition of a disease prediction basis screen. [Figure 13] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as a disease prediction basis display device. [Figure 14] 13 is a flowchart showing an example of a disease prediction basis display process. [Figure 15] 13 is a flowchart illustrating an example of a display control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0012] First, before explaining the details of the embodiment, a specific description will be given of a case in which, in the conventional technology, it is not easy to confirm the basis for a prediction made by a machine learning model from the perspective of a comprehensive judgment that includes inspection items other than the inspection image.

[0013] As described above, when a doctor judges a disease, the doctor makes a comprehensive judgment by looking at not only test images such as CT and MRI but also test values ​​such as blood test values. Therefore, when a disease is predicted using a machine learning model, the doctor may want to refer to and check the prediction result of the machine learning model from the viewpoint of the doctor's comprehensive judgment when checking the prediction result of the machine learning model.

[0014] Although conventional technology can show the basis of prediction by highlighting areas that contributed to disease prediction on the examination image, it cannot distinguish between multiple elements of the disease phenomenon. Therefore, conventional technology has a problem in that it is not possible to immediately check whether the prediction of the machine learning model is correct from the perspective of comprehensive judgment.

[0015] Specifically, a phenomenon appearing in a disease may involve multiple factors such as inflammation and fibrosis of an organ. For example, as shown in Fig. 1, disease X is accompanied by phenomenon A (e.g., inflammation) and phenomenon B (e.g., fibrosis), and disease Y is accompanied by phenomenon A. In addition, in an examination image of an organ, phenomenon A appears in the shaded area in the upper diagram of Fig. 2, and phenomenon B appears in the diagonal grid area in the lower diagram of Fig. 2. Note that phenomenon B appearing in the examination image is assumed to have only a subtle difference from the normal state, so that it is difficult for a doctor other than a skilled physician to recognize it.

[0016] Then, using the test values ​​of test α for testing phenomenon A, the test values ​​of test β for testing phenomenon B, and the test image as shown in Fig. 3, a machine learning model predicts a disease. Here, assume that the machine learning model predicts disease X, and a heat map as shown in Fig. 4 is obtained as the basis for the prediction. The doctor first suspects disease X or disease Y based on the result of test α and the part of the test image where phenomenon A appears, and further suspects disease X because the result of test β is somewhat high.

[0017] In the heat map shown in FIG. 4, it can be determined that phenomenon A corresponding to test α is valid as a basis for predicting disease X, but it is unclear whether phenomenon B corresponding to test β is a basis for prediction. The area highlighted in the heat map almost coincides with the area where phenomenon A appears, and it is unclear whether the area where phenomenon B appears is a basis for prediction. Therefore, the doctor closely observes the test image to find and confirm the part corresponding to phenomenon B, and is convinced that it is disease X. As such, in the conventional technology, it is sometimes difficult to easily confirm the basis for prediction by a machine learning model, such as by requiring detailed observation of the test image.

[0018] Therefore, in this embodiment, a disease is predicted from the test values ​​for each of a plurality of test items and the test image, and a visualized map of the area on the test image correlated with each test value that contributed to the prediction is displayed, as shown in Fig. 5. The disease prediction basis display device according to this embodiment will be described in detail below.

[0019] 6, test values ​​for each of a plurality of test items such as blood tests, and test image data such as CT and MRI are input to the disease prediction basis display device 10. The disease prediction basis display device 10 predicts a disease, identifies areas on the test image that are correlated with each test value that contributed to the prediction, and displays a disease prediction basis screen including the prediction result and a visualized map of the identified areas on a display device.

[0020] 6, the disease prediction basis display device 10 functionally includes a prediction unit 12, an identification unit 14, and a display control unit 16. A prediction model 20 is stored in a predetermined storage area of ​​the disease prediction basis display device 10. The prediction model 20 is an example of a "machine learning model" of the disclosed technology.

[0021] The prediction model 20 predicts diseases using test values ​​of multiple test items and test image data (pixel values ​​of each pixel) as input. The prediction model 20 may be a network model such as a deep neural network. The prediction model 20 also has a fusion layer that fuses the features of the test values ​​and the features of the test image based on the correlation between the features of the test values ​​and the features of the test image.

[0022] FIG. 7 shows an example of the configuration of the prediction model 20. The prediction model 20 includes a CNN (Convolutional Neural Network) layer, a feature map layer, an intermediate layer, a fusion layer, and an output layer. The CNN layer receives data of an inspection image, extracts features of the inspection image by convolution processing, and outputs one or more feature maps. The feature map layer includes a first portion corresponding to a feature map for a plurality of inspection items, and a second portion corresponding to a feature map for the inspection image. That is, the first portion of the feature map layer has nodes corresponding to each of the plurality of inspection items, and the inspection value of each inspection item is input to each node. In addition, the second portion of the feature map layer has nodes corresponding to each pixel of one or more feature maps output from the CNN layer, and the pixel value of each pixel of the feature map output from the CNN layer is input to each node.

[0023] The intermediate layer also includes a first part and a second part like the feature map layer, and each of the first part and the second part has a plurality of nodes. The connection between the first part of the feature map layer and the first part of the intermediate layer is a full connection, and the connection between the second part of the feature map layer and the second part of the intermediate layer is also a full connection. In FIG. 7, the connections between the layers are indicated by arrows, and the double-lined arrows indicate full connections. The fusion layer includes a plurality of nodes, and is fully connected to each node of the first part and the second part of the intermediate layer. The output layer includes nodes for the number of disease categories, and each node outputs the probability that the disease is in each category, which is predicted based on the input test value and test image, by a softmax function or the like.

[0024] Each node in each layer outputs a value calculated based on the input value and the activation function set for the node. The output value is multiplied by the weight set between the nodes, and the value becomes the input for the node in the next layer. This weight is updated to an optimal value in the machine learning of the prediction model 20. Therefore, each node in the fusion layer, to which the output from each node in the intermediate layer including the first part corresponding to the test value and the second part corresponding to the test image is input by full connection, represents which test value is multiplied by which pixel of the test image and at what ratio. In other words, each node in the fusion layer can be said to be a node where the correlation between the feature of the test value and the feature of the test image is calculated.

[0025] The prediction unit 12 inputs the test values ​​of each of the multiple test items and the test image data into the prediction model 20 to predict a disease. Specifically, in the prediction model 20, values ​​calculated at each node based on the test values ​​of each of the multiple test items and the test image data input are forward propagated through the network, and a probability is calculated at each node corresponding to a disease category in the output layer. The prediction unit 12 outputs the disease category corresponding to the node in the output layer that outputs the maximum probability as a prediction result. As shown in (1) of FIG. 8, when the probability output by the node corresponding to category X is maximum, the disease prediction result is a disease of category X (hereinafter, referred to as "disease X").

[0026] The identification unit 14 identifies, for each of the inspection items that are the basis of the prediction, an area on the inspection image that is correlated with the inspection item and that is the basis of the prediction, by backpropagating gradients from the nodes of the fusion layer that contributed to the prediction.

[0027] Specifically, the identification unit 14 identifies a predetermined number of test items that are the basis of the prediction. For example, as shown in (2) of FIG. 8, the identification unit 14 performs backpropagation of a gradient from a node indicating a prediction result (a node corresponding to disease X in the example of FIG. 8) in the output layer of the prediction model 20, and calculates a backpropagated value in the feature map layer. For example, Grad-CAM (Gradient-Class Activation Mapping) may be applied to calculate the backpropagated value. Grad-CAM is a method of giving a local explanation of a prediction for an input to a network model, and is a method of calculating which part of the input has the greatest influence on the prediction as a heat map. In other words, the higher the calculated CAM value of a node, the higher the contribution of the node to the prediction.

[0028] When Grad-CAM is applied, a CAM (hereinafter referred to as "test item CAM") for the first part of the feature map layer (part corresponding to the test item) is obtained as shown in FIG. 9. Also, a CAM (hereinafter referred to as "test image CAM") for the second part of the feature map layer (part corresponding to the test image) is obtained. In FIG. 9, each square of the test item CAM corresponds to each node of the first part of the feature map layer, i.e., each test item, and the shading of each square represents the difference in CAM value for each test item. Also, in FIG. 9, the test image CAM is shown superimposed on the test image, and the part indicated by the dashed circle represents the test image CAM, and the shading of the test image CAM represents the difference in CAM value for each pixel of the test image.

[0029] The identification unit 14 identifies a predetermined number of test items that contribute highly to the prediction for each node in the first part of the feature map layer based on a value calculated by backpropagation of the gradient from the node in the output layer. For example, the identification unit 14 selects a node whose CAM value is equal to or greater than a predetermined value, or a predetermined number of nodes whose CAM values ​​are the highest, and identifies the test items corresponding to the selected nodes as test items that contribute highly to the prediction.

[0030] In addition, the specification unit 14 calculates the contribution of each node in the fusion layer to the prediction result, as shown in (3) of FIG. 10. For example, the contribution c X The identification unit 14 calculates the contribution degree c(n) based on a value obtained by multiplying the output value from the node n by the weight between the node n and the node indicating the disease X. X (n) is the top predetermined number of nodes, or contribution degree c X Nodes with (n) equal to or greater than a predetermined value are selected, and the following process is performed on the selected nodes.

[0031] The identification unit 14 calculates an index for identifying an area of ​​the inspection image that is correlated with the inspection item on which the prediction is based, based on the values ​​obtained in each of the first and second parts of the feature map layer by backpropagating the gradient from the node selected in the fusion layer to the feature map layer. For example, the identification unit 14 applies Grad-CAM in the same manner as described above to the contribution degree c X Backpropagate the gradient from the node with the highest (n) to the node with the highest (n). As a result, the inspection item CAM and the inspection image CAM are obtained in the feature map layer in the same manner as above.

[0032] The identification unit 14 determines the contribution degree c X The test item CAM and test image CAM obtained for each node in the fusion layer selected based on (n) are aggregated with a focus on each test item, and an area on the test image that is correlated with each test item and serves as a basis for prediction is identified. Hereinafter, this area is referred to as the "prediction basis of the test image correlated with the test item."

[0033] More specifically, in the examination item CAM calculated by backpropagation of the gradient from node n of the fusion layer, the value of examination item v is defined as T_CAM_n(v). In addition, in the examination image CAM calculated by backpropagation of the gradient from node n of the fusion layer, the value of the pixel at coordinates (x, y) is defined as I_CAM_n(x, y). The identification unit 14 calculates the prediction basis I_CAM_v(x, y) of the examination image correlated with the examination item v by the following formula (1). Note that N in formula (1) is the contribution degree c X (n) is the number of nodes in the fusion layer, selected according to

[0034]

number

[0035] Fig. 11 shows a schematic diagram of a prediction basis (I_CAM_v) of an inspection image correlated with an inspection item v. In the example of Fig. 11, I_CAM_v is superimposed on the inspection image, and the portion indicated by a dashed circle represents I_CAM_v. The identification unit 14 calculates I_CAM_v for each of a predetermined number of inspection items selected based on the inspection item CAM calculated by backpropagation of the gradient from the output layer.

[0036] The display control unit 16 displays the examination items that are the basis of prediction and that are correlated with each other and the areas on the examination image in association with each other. Specifically, the display control unit 16 displays a disease prediction basis screen including a predetermined number of examination items selected based on the examination item CAM calculated by the identification unit 14 through backpropagation of the gradient from the output layer, and the examination image. Then, in response to a selection of any one of the predetermined number of examination items, the display control unit 16 displays the prediction basis (I_CAM_v) of the examination image that is correlated with the selected examination item on the examination image.

[0037] More specifically, the display control unit 16 performs initial display of a disease prediction basis screen 40 as shown in the upper diagram of FIG. 12. In the example of FIG. 12, the disease prediction basis screen 40 includes a prediction result display area 42, an examination item display area 44, and a prediction basis display area 46. The prediction result display area 42 displays the prediction result and the prediction probability by the prediction model 20. The prediction probability is the probability output by a node corresponding to the prediction result in the output layer of the prediction model 20. The examination item display area 44 displays a predetermined number of examination items selected by the identification unit 14 based on the examination item CAM calculated by backpropagation of the gradient from the output layer, and the contribution degree of the examination item. The contribution degree here is a value for a node corresponding to the relevant examination item in the examination item CAM calculated by backpropagation of the gradient from the output layer. The prediction basis display area 46 displays an examination image, and also displays an area serving as the prediction basis on the examination image in a superimposed manner. In the initial display of the disease prediction basis screen 40, only the examination image is displayed in the prediction basis display area 46. Furthermore, the specification unit 14 may superimpose an inspection image CAM calculated by backpropagation of the gradient from the output layer on the inspection image.

[0038] As shown in the middle and lower diagrams of FIG. 12, when any one of the examination items is selected in the examination item display area 44 from the initial display state, the display control unit 16 highlights the selected examination item in the examination item display area 44. In addition, the display control unit 16 superimposes I_CAM_v calculated by the specification unit 14 for the selected examination item v on the examination image displayed in the prediction basis display area 46. The middle diagram of FIG. 12 shows an example in which the examination item α is selected, and the lower diagram of FIG. 12 shows an example in which the examination item β is selected. That is, every time the selection of the examination item is changed, the display control unit 16 updates the prediction basis superimposed on the examination image in the prediction basis display area 46 to the prediction basis of the examination image correlated with the selected examination item.

[0039] The disease prediction basis display device 10 may be realized by, for example, a computer 50 shown in FIG. 13. The computer 50 includes a CPU (Central Processing Unit) 51, a GPU (Graphics Processing Unit) 52, a memory 53 as a temporary storage area, and a non-volatile storage device 54. The computer 50 also includes an input / output device 55 such as an input device and a display device, and an R / W (Read / Write) device 56 that controls reading and writing of data to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 57 that is connected to a network such as the Internet. The CPU 51, the GPU 52, the memory 53, the storage device 54, the input / output device 55, the R / W device 56, and the communication I / F 57 are connected to one another via a bus 58.

[0040] The storage device 54 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 54 as a storage medium stores a disease prediction basis display program 60 for causing the computer 50 to function as the disease prediction basis display device 10. The disease prediction basis display program 60 has a prediction process control command 62, a specific process control command 64, and a display control process control command 66. The storage device 54 also has an information storage area 70 in which information constituting the prediction model 20 is stored.

[0041] The CPU 51 reads out the disease prediction basis display program 60 from the storage device 54, loads it in the memory 53, and sequentially executes the control instructions of the disease prediction basis display program 60. The CPU 51 operates as the prediction unit 12 shown in FIG. 6 by executing the prediction process control instruction 62. The CPU 51 also operates as the specification unit 14 shown in FIG. 6 by executing the specification process control instruction 64. The CPU 51 also operates as the display control unit 16 shown in FIG. 6 by executing the display control process control instruction 66. The CPU 51 also reads out information from the information storage area 70 and loads the prediction model 20 in the memory 53. As a result, the computer 50 that has executed the disease prediction basis display program 60 functions as the disease prediction basis display device 10. The CPU 51 that executes the program is hardware. Also, a part of the program may be executed by the GPU 52.

[0042] The functions realized by the disease prediction basis display program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0043] Next, the operation of the disease prediction basis display device 10 according to this embodiment will be described. When the test values ​​for each of a plurality of test items and the data of the test image are input to the disease prediction basis display device 10, the disease prediction basis display device 10 executes a disease prediction basis display process shown in Fig. 14. The disease prediction basis display process is an example of a disease prediction basis display method of the disclosed technology.

[0044] In step S10, the prediction unit 12 acquires test values ​​and test image data for each of the multiple test items input to the disease prediction basis display device 10. Next, in step S12, the prediction unit 12 inputs the test values ​​and test image data for each of the multiple test items to the prediction model 20 to predict a disease.

[0045] Next, in step S14, the identification unit 14 performs backpropagation of gradients from nodes indicating prediction results in the output layer of the prediction model 20, and calculates the examination item CAM in the feature map layer. Then, the identification unit 14 identifies a predetermined number of examination items having a high contribution to the prediction based on the calculated examination item CAM.

[0046] Next, in step S16, the identification unit 14 calculates the contribution of each node in the fusion layer to the prediction result, backpropagates the gradient in order from the node with the highest contribution, and calculates the examination item CAM and the examination image CAM in the feature map layer. Next, in step S20, the identification unit 14 calculates the prediction basis (I_CAM_v) of the examination image correlated with each examination item using the examination item CAM, the examination image CAM, and the contribution of the nodes in the fusion layer, for example, by equation (1).

[0047] Next, in step S30, a display control process is executed. Here, the display control process will be described in detail with reference to FIG.

[0048] In step S32, the display control unit 16 performs an initial display of the disease prediction basis screen 40 using the prediction results output by the prediction model 20 and a predetermined number of test items and test images identified by the identification unit 14 in order of highest contribution.

[0049] Next, in step S34, the display control unit 16 determines whether or not any test item has been selected on the disease prediction basis screen 40. If a test item has been selected, the process proceeds to step S36, and if not, the process proceeds to step S40.

[0050] In step S36, the display control unit 16 highlights the selected examination item in the examination item display area 44. Next, in step S38, the display control unit 16 superimposes I_CAM_v calculated in step S20 for the selected examination item v on the examination image displayed in the prediction basis display area 46.

[0051] Next, in step S40, the display control unit 16 determines whether or not to end the display of the disease prediction basis screen 40, depending on whether or not a command to end the display of the disease prediction basis screen 40 has been received. If the display of the disease prediction basis screen 40 is not to be ended, the process returns to step S34, and if it is to be ended, the display control process is ended and the disease prediction basis display process is also ended.

[0052] As described above, the disease prediction basis display device according to the present embodiment predicts a disease by inputting the test values ​​of each of a plurality of test items and the pixel values ​​of the test image into the prediction model. The prediction model is a machine learning model that predicts a disease by inputting the test values ​​of each of a plurality of test items and the pixel values ​​of the test image, and has a fusion layer that fuses the test value features and the pixel value features based on the correlation between the test value features and the pixel value features. In addition, the disease prediction basis display device specifies, for each test item that is the basis of the prediction, an area on the test image that is correlated with the test item and is the basis of the prediction, by backpropagating the gradient from the node of the fusion layer that contributed to the prediction. Then, the disease prediction basis display device displays the test items that are the basis of the prediction and are correlated with each other and the areas on the test image in association with each other. This makes it possible to easily confirm the basis of the prediction in disease prediction using the test values ​​and the test image. In addition, by presenting detailed prediction basis on the test image for each test item and providing a material for judging the logical validity of the prediction, the reliability of the prediction is improved.

[0053] In the above embodiment, the calculation of I_CAM_v and display on the disease prediction basis screen are performed for a predetermined number of test items that have a high contribution to prediction, but the present invention is not limited to this. It is not essential to specify a predetermined number of test items, and all test items may be targeted.

[0054] In the above embodiment, the disease prediction basis display program is stored (installed) in advance in the storage device, but is not limited thereto. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0055] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0056] (Appendix 1) A machine learning model for predicting a disease using test values ​​and test images of a plurality of test items as input, the machine learning model having a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images, inputting the test values ​​and test images of a plurality of test items into the machine learning model to predict a disease; by backpropagating gradients from nodes of the fusion layer that contributed to the prediction, for each of the test items that are the basis of the prediction, identifying an area on the test image that is correlated with the test item and that is the basis of the prediction; displaying the test items and the regions that are correlated with each other and serve as the basis for the prediction; A disease prediction basis display method, comprising: a computer executing a process including the steps of:

[0057] (Appendix 2) the machine learning model has a feature map layer including a first portion corresponding to a feature map for the plurality of inspection items and a second portion corresponding to a feature map for the inspection image; calculating an index for identifying the region based on the value obtained in the first portion and the value obtained in the second portion by backpropagating gradients from the nodes of the fusion layer to the feature map layer; A method for displaying disease prediction basis as described in Appendix 1.

[0058] (Appendix 3) The disease prediction basis display method according to claim 2, further comprising: identifying a predetermined number of test items having a high contribution to the prediction by backpropagation of a gradient from an output layer in the machine learning model; and calculating the index for the predetermined number of the test items.

[0059] (Appendix 4) The disease prediction basis display method according to appendix 3, wherein the index is the sum of the contribution degree of the nodes in the fusion layer that contributed to the prediction, the value obtained by multiplying the value for the test item to be calculated among the values ​​obtained in the first part by the value obtained in the second part, over the number of nodes in the fusion layer that contributed to the prediction.

[0060] (Appendix 5) Displaying the predetermined number of examination items and the examination image; In response to a selection of any one of the predetermined number of test items, the region correlated with the selected test item is displayed on the test image. A method for displaying disease prediction basis according to appendix 3 or appendix 4.

[0061] (Appendix 6) The method for displaying disease prediction grounds according to any one of appendices 1 to 5, wherein the nodes in the fusion layer that contributed to the prediction are nodes in a top predetermined number of nodes in terms of contribution degree calculated based on a value obtained by multiplying the output value from the node and the weight of the node, or nodes whose contribution degree is equal to or greater than a predetermined value.

[0062] (Appendix 7) A machine learning model for predicting a disease using test values ​​and test images of a plurality of test items as input, the machine learning model having a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images, inputting the test values ​​and test images of a plurality of test items into the machine learning model to predict a disease; by backpropagating gradients from nodes of the fusion layer that contributed to the prediction, for each of the test items that are the basis of the prediction, identifying an area on the test image that is correlated with the test item and that is the basis of the prediction; displaying the test items and the regions that are correlated with each other and serve as the basis for the prediction; A disease prediction basis display program for causing a computer to execute a process including the above.

[0063] (Appendix 8) the machine learning model has a feature map layer including a first portion corresponding to a feature map for the plurality of inspection items and a second portion corresponding to a feature map for the inspection image; calculating an index for identifying the region based on the value obtained in the first portion and the value obtained in the second portion by backpropagating gradients from the nodes of the fusion layer to the feature map layer; A disease prediction basis display program described in Appendix 7.

[0064] (Appendix 9) The disease prediction basis display program according to claim 8, which identifies a predetermined number of test items that have a high contribution to the prediction by backpropagation of a gradient from an output layer in the machine learning model, and calculates the index for the predetermined number of the test items.

[0065] (Appendix 10) The disease prediction basis display program according to claim 9, wherein the index is the sum of the contribution degree of the nodes in the fusion layer that contributed to the prediction, the value obtained by multiplying the value for the test item to be calculated among the values ​​obtained in the first part by the value obtained in the second part, over the number of nodes in the fusion layer that contributed to the prediction.

[0066] (Appendix 11) Displaying the predetermined number of examination items and the examination image; In response to a selection of any one of the predetermined number of test items, the region correlated with the selected test item is displayed on the test image. A disease prediction basis display program according to appendix 9 or appendix 10.

[0067] (Appendix 12) The disease prediction basis display program according to any one of appendices 7 to 11, wherein the nodes in the fusion layer that contributed to the prediction are nodes in a top predetermined number of nodes in terms of contribution degree calculated based on a value obtained by multiplying the output value from the node and the weight of the node, or nodes whose contribution degree is equal to or greater than a predetermined value.

[0068] (Appendix 13) For a disease predicted based on the test values ​​of a plurality of test items and the test image, the test items on which the prediction is based and the test image are displayed; In response to a selection of any one of the examination items, a region on the examination image that correlates with the selected examination item and serves as a basis for prediction is displayed on the examination image. Method for displaying disease prediction evidence.

[0069] (Appendix 14) A machine learning model for predicting a disease using test values ​​and test images of a plurality of test items as input, the machine learning model having a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images, inputting the test values ​​and test images of a plurality of test items into the machine learning model to predict a disease; by backpropagating gradients from nodes of the fusion layer that contributed to the prediction, for each of the test items that are the basis of the prediction, identifying an area on the test image that is correlated with the test item and that is the basis of the prediction; displaying the test items and the regions that are correlated with each other and serve as the basis for the prediction; A non-transitory storage medium storing a disease prediction basis display program for causing a computer to execute a process including the above. [Explanation of symbols]

[0070] 10. Disease prediction evidence display device 12 Prediction Department 14 Specific part 16 Display control section 20 Predictive Models 40 Disease Prediction Basis Screen 42 Prediction result display area 44 Test item display area 46 Prediction Basis Display Area 50 Computer 51 CPU 52 GPU 53 Memory 54 Storage device 55 Input / Output Devices 56 R / W device 57 Communication I / F 58 Bus 59 Storage medium 60 Disease Prediction Basis Display Program 62 Predictive Process Control Instructions 64 Specific Process Control Instructions 66 Display Control Process Control Instructions 70 Information storage area

Claims

1. A machine learning model for predicting a disease using test values ​​and test images of a plurality of test items as input, the machine learning model having a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images, inputting the test values ​​and test images of a plurality of test items into the machine learning model to predict a disease; by backpropagating gradients from nodes of the fusion layer that contributed to the prediction, for each of the test items that are the basis of the prediction, identifying an area on the test image that is correlated with the test item and that is the basis of the prediction; displaying the test items and the regions that are the basis of the prediction and that are correlated with each other in association with each other; A disease prediction basis display method, comprising: a computer executing a process including the steps of:

2. the machine learning model has a feature map layer including a first portion corresponding to a feature map for the plurality of inspection items and a second portion corresponding to a feature map for the inspection image; calculating an index for identifying the region based on a value obtained in the first portion and a value obtained in the second portion by backpropagating a gradient from a node of the fusion layer to the feature map layer; The disease prediction basis display method according to claim 1.

3. The disease prediction basis display method according to claim 2, further comprising: identifying a predetermined number of test items having a high contribution to the prediction by backpropagation of a gradient from an output layer in the machine learning model; and calculating the index for the predetermined number of the test items.

4. The disease prediction basis display method according to claim 3, wherein the index is the sum of the contribution degree of the nodes in the fusion layer that contributed to the prediction, the value obtained by multiplying the value for the test item to be calculated among the values ​​obtained in the first part by the value obtained in the second part, for the number of nodes in the fusion layer that contributed to the prediction.

5. Displaying the predetermined number of examination items and the examination image; In response to a selection of any one of the predetermined number of test items, the region correlated with the selected test item is displayed on the test image. The method for displaying disease prediction grounds according to claim 3 or 4.

6. The node of the fusion layer that contributed to the prediction is a node having a contribution degree calculated based on a value obtained by multiplying the output value from the node and the weight of the node, which is a top predetermined number of nodes, or a node having a contribution degree equal to or greater than a predetermined value. The disease prediction basis display method according to any one of claims 1 to 4.

7. A machine learning model for predicting a disease using test values ​​and test images of a plurality of test items as input, the machine learning model having a fusion layer that fuses features of the test values ​​and features of the test images based on a correlation between the features of the test values ​​and the features of the test images, inputting the test values ​​and test images of a plurality of test items into the machine learning model to predict a disease; by backpropagating gradients from nodes of the fusion layer that contributed to the prediction, for each of the test items that are the basis of the prediction, identifying an area on the test image that is correlated with the test item and that is the basis of the prediction; displaying the test items and the regions that are the basis of the prediction and that are correlated with each other in association with each other; A disease prediction basis display program for causing a computer to execute a process including the above.

8. For a disease predicted based on the test values ​​of a plurality of test items and the test image, the test items on which the prediction is based and the test image are displayed; In response to a selection of any one of the examination items, a region on the examination image that correlates with the selected examination item and serves as a basis for prediction is displayed on the examination image. Method for displaying disease prediction evidence.

Images