Learning device, prediction device, learning method, and program

Abstract

Provided is a training device, wherein a partial image generation means generates, from an input image, a partial image smaller than the input image. A feature space generation means generates, for each of input images, a feature space to which feature amounts of a plurality of partial images are mapped. A training data generation means, on the basis of the feature space, acquires a plurality of partial training images from the plurality of partial images and generates training data. A training means uses the training data to train a prediction model which predicts a probability that a prescribed feature is included in the partial training image. A prediction means uses the trained prediction model to perform the prediction on all of or a portion of partial images included in the input image. Furthermore, the training data generation means acquires, on the basis of the predicted values for all of or a portion of the partial images in the feature space, a plurality of partial training images to be used as the training data in the next training. In this way, the partial training images are updated and the training for the prediction model is repeated by the training means.

Description

[Technical Field]

[0001] The present disclosure relates to a technique for predicting characteristic portions contained in an image. [Background technology]

[0002] There is a known technology for classifying and predicting images through deep learning using neural networks. In so-called supervised learning, a model is trained using training data in which input images are labeled. Patent Document 1 describes a method for tiling digital images of biological samples into a group of image patches and applying a classifier to classify the images. [Prior art documents] [Non-patent literature]

[0003] [Patent Document 1] Special Publication No. 2020-533725 Summary of the Invention [Problem to be solved by the invention]

[0004] When using deep learning to classify or predict images, if the image size is large and the area of ​​the image that is effective for learning is small, the probability of sampling effective data as training data decreases, resulting in inefficient learning. Also, if the image size is large and most of it is effective for learning, a large amount of similar data will be sampled as training data, reducing the diversity of the training data and potentially reducing the accuracy of the model obtained by learning.

[0005] One object of the present disclosure is to perform accurate model training while selecting regions that are effective as training data in a situation where detailed labels are not given to the entire image. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a learning device includes: a partial image generating means for generating, from an input image, a partial image smaller than the input image; a feature space generating means for generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; a training data generation means for acquiring a plurality of training partial images from the plurality of partial images based on the feature space and generating training data; a learning means for learning a prediction model that predicts the probability that a predetermined feature is included in the learning partial image using the learning data; a prediction means for performing predictions for all or some of the partial images included in the input image using a trained prediction model; Equipped with The training data generating means acquires a plurality of training partial images to be used as training data in the next training, based on predicted values ​​for all or some of the partial images in the feature space.

[0007] In another aspect of the present disclosure, a learning method includes: 1. A computer-implemented method of learning, comprising: Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; Using the trained prediction model, prediction is made for all or some of the partial images included in the input image; The generation of the learning data involves obtaining a plurality of learning partial images to be used as learning data in the next learning, based on predicted values ​​for all or some of the partial images in the feature space.

[0008] In yet another aspect of the present disclosure, program teeth, Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; Using the trained prediction model, predictions are made for all or some of the partial images included in the input image. The computer executes the process , The generation of the learning data involves obtaining a plurality of learning partial images to be used as learning data in the next learning, based on predicted values ​​for all or some of the partial images in the feature space. [Effects of the Invention]

[0009] According to the present disclosure, even in a situation where detailed labels are not given to the entire image, it is possible to accurately train a model while selecting areas that are effective as training data. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows a learning device according to a first embodiment. [Figure 2] 1 is a block diagram showing the hardware configuration of a learning device according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of the learning device. [Figure 4] FIG. 10 is an explanatory diagram of the processing of an image dividing unit and a learning data generating unit. [Figure 5] An example of a partial image is shown below. [Figure 6] An example of dividing the feature space into multiple spatial regions by grid division will be shown. [Figure 7] FIG. 10 is an explanatory diagram of a method for generating learning data. [Figure 8] 10 is a flowchart of a learning process performed by the learning device. [Figure 9]10 shows an example of the region selection probability at the end of learning. [Figure 10] FIG. 10 is a block diagram showing the functional configuration of a learning device according to a modified example. [Figure 11] FIG. 10 is a diagram illustrating a process performed by a learning device according to a modified example. [Figure 12] FIG. 2 is a block diagram showing a functional configuration of the prediction device. [Figure 13] 10 is a flowchart of a prediction process performed by the prediction device. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of a learning device according to a second embodiment. [Figure 15] 10 is a flowchart of processing by a learning device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings. <Basic principle> The present disclosure provides an apparatus for predicting predetermined image features contained in an input image, which efficiently trains a model by selecting effective partial images based on a probability distribution in a feature space. In the following embodiment, an example will be described in which a pathological tissue image of a patient receiving medication is input, and the medication effect is predicted based on the morphological features of the pathological tissue.

[0012] Specifically, during training, the training device divides an input image into multiple partial images and trains a prediction model that makes predictions for each partial image. Here, the training device maps the multiple partial images onto a feature space and divides the feature space into multiple regions (hereinafter referred to as "spatial regions") with similar morphological features. The training device then acquires partial images from the multiple spatial regions according to probability distributions assigned to the multiple spatial regions to generate training data and trains the prediction model. This allows for efficient selection of effective partial images for training from the input image to train the prediction model. Furthermore, once a trained model is obtained through training, the training device uses the trained prediction model to make predictions for each partial image included in the input image and updates the probability distributions for the multiple spatial regions based on the prediction results. The training device then acquires partial images from the spatial regions according to the updated probability distributions to generate training data and further trains the prediction model. In this way, a highly accurate prediction model is generated by repeatedly training the prediction model while updating the probability distributions assigned to the multiple spatial regions.

[0013] On the other hand, when making predictions (inferences) using a trained prediction model, the prediction device divides the input image into multiple partial images, performs predictions on the partial images using the prediction model, and integrates the prediction results for each partial image to obtain a prediction result for the input image. Furthermore, the prediction device can present parts of the input image that were important for the prediction based on the prediction results for each partial image.

[0014] First Embodiment [Learning device] 1 shows a learning device according to the first embodiment. The learning device 100 learns a prediction model based on input image data (hereinafter also referred to as "input image").

[0015] (Hardware configuration) 2 is a block diagram showing the hardware configuration of a learning device 100 according to the first embodiment. As shown in the figure, learning device 100 includes an interface (IF) 12, a processor 13, a memory 14, a recording medium 15, a database (DB) 16, and a display device 17.

[0016] The IF 12 inputs image data used for training a prediction model. The processor 13 is a computer such as a CPU (Central Processing Unit), and controls the entire learning device 100 by executing a pre-prepared program. The processor 13 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). Specifically, the processor 13 executes the training process and prediction process described below.

[0017] The memory 14 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 14 stores various programs executed by the processor 13. The memory 14 is also used as a working memory while the processor 13 is executing various processes.

[0018] Recording medium 15 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from learning device 100. Recording medium 15 records various programs executed by processor 13. When learning device 100 executes various processes, the programs recorded on recording medium 15 are loaded into memory 14 and executed by processor 13.

[0019] DB16 stores image data input via IF12. Specifically, DB16 stores data on input images used for learning by learning device 100. Display device 17 is, for example, an LCD display or projector, and displays the prediction results obtained by learning device 100. In addition to the above, learning device 100 may also be equipped with input devices such as a keyboard and mouse for the user to enter instructions and input.

[0020] (Functional configuration) 3 is a block diagram showing the functional configuration of the learning device 100. Functionally, the learning device 100 includes an image segmentation unit 21, a learning data generation unit 22, a learning unit 23, a prediction unit 24, a selection probability update unit 25, and an integration unit 26. The output of the integration unit 26 is supplied to a display device 17.

[0021] During learning, a learning dataset is prepared. In the following explanation, a set of one input image and a teacher label (hereinafter simply referred to as a "label") for that input image is referred to as learning data. A label for an input image is assigned to each input image. That is, one label is assigned to an entire input image, such that the label for one input image is a positive example and the label for another input image is a negative example. A collection of learning data for multiple input images is referred to as a learning dataset.

[0022] The learning data generation unit 22, the learning unit 23, the prediction unit 24, and the selection probability update unit 25 are configured to repeat a loop process (hereinafter referred to as a "learning loop") of generating learning data and learning a prediction model a predetermined number of times.

[0023] The image dividing unit 21 divides an input image included in a training dataset into partial images smaller than the input image. Hereinafter, the entire input image will also be referred to as the "whole image." FIG. 4 is an explanatory diagram of the processing performed by the image dividing unit 21 and the training data generating unit 22. As shown in FIG. 4, the image dividing unit 21 divides the whole image WI into multiple partial images PI and outputs them to the training data generating unit 22. In the example of FIG. 4, the whole image WI is an image of a patient's pathological tissue. However, the image dividing unit 21 only needs to generate partial images for the pathological tissue region, and does not need to generate partial images for the background region. In the example of FIG. 4, the image dividing unit 21 divides the whole image WI into multiple partial images PI by grid division. Alternatively, the whole image WI may be divided into multiple partial images PI centered on characteristic points.

[0024] The training data generation unit 22 generates training data for training a prediction model using the input multiple partial images PI. Specifically, the training data generation unit 22 first converts each of the input multiple partial images PI into a feature, performs dimensionality reduction, and maps the resulting feature onto a two-dimensional feature space. FIG. 4 shows an example in which each partial image PI is mapped onto the feature space. One partial image PI is mapped as one point on the two-dimensional feature space. For example, t-SNE (t-distributed Stochastic Neighbor Embedding), UMAP (Uniform Manifold Approximation and Projection), etc. can be used as the dimensionality reduction algorithm.

[0025] Next, the training data generation unit 22 divides the two-dimensional feature space into a plurality of regions (hereinafter also referred to as "spatial regions" to distinguish them from regions in the image). There are two methods for dividing the feature space into a plurality of spatial regions. In the first method, the training data generation unit 22 divides the feature space into a plurality of spatial regions by clustering. In the example of FIG. 4, the training data generation unit 22 divides the feature space into four spatial regions SA1 to SA4 by clustering. Note that a known clustering method such as kmeans can be used for the clustering.

[0026] Points located close to each other in the feature space mean that the image features of the partial images corresponding to those points are similar, so multiple partial images corresponding to points included in the same spatial region SA are images of morphologically similar tissues. FIG. 5 shows an example of partial images PI. In FIG. 5, if a partial image belonging to spatial region SA1 is a tissue image of stroma, it is highly likely that other partial images belonging to spatial region SA1 are also tissue images of stroma. Similarly, if a partial image belonging to spatial region SA2 is a tissue image of a glandular duct, it is highly likely that other partial images belonging to spatial region SA2 are also tissue images of glandular ducts. Furthermore, if a partial image belonging to spatial region SA4 is a tissue image of a tumor, it is highly likely that other partial images belonging to spatial region SA4 are also tissue images of stroma.

[0027] In the second method, the training data generation unit 22 divides the feature space into multiple spatial regions by simple grid division instead of clustering. Figure 6 shows an example of dividing the feature space into multiple spatial regions by grid division. In this example, the entire image WI is divided into n x m grids. In this case, each grid corresponds to one spatial region.

[0028] Next, the training data generation unit 22 acquires partial images from the plurality of spatial regions obtained by the first or second method described above, and generates training data. FIG. 7 is an explanatory diagram of the training data generation method. Note that the following description assumes that the above first method is used. First, the training data generation unit 22 sets a probability of selecting one spatial region to be used for generating training data from the plurality of spatial regions (hereinafter referred to as "region selection probability"). For example, the region selection probability of spatial region SA1 indicates the probability that spatial region SA1 is selected from four spatial regions SA1 to SA4.

[0029] 7, in the initial state, the training data generation unit 22 sets the region selection probability of each spatial region to the same value. For example, as shown in probability distribution graph 51, the training data generation unit 22 sets the region selection probability of each spatial region SA1 to SA4 to the same value. Then, the training data generation unit 22 selects one spatial region from the spatial regions SA1 to SA4 based on the set region selection probability (first selection). In the example of FIG. 7, as shown by arrow 52, ​​the training data generation unit 22 selects spatial region SA4.

[0030] Next, the training data generation unit 22 selects a predetermined number of partial images PI from the plurality of partial images PI belonging to the selected spatial region (second selection). Then, the training data generation unit 22 generates partial images (hereinafter also referred to as "patch images") by further dividing each of the selected plurality of partial images PI by grid division or the like. Then, the training data generation unit 22 acquires patch images for training (hereinafter referred to as "training patch images") from each partial image PI. Specifically, the training data generation unit 22 may acquire all patch images obtained by dividing the partial image PI as training patch images, or may select a predetermined number of these patch images by random sampling and acquire them as training patch images LI. Note that the training patch images are an example of training partial images. In this way, a plurality of training patch images LI are generated based on a plurality of partial images corresponding to the selected spatial region (spatial region SA4 in FIG. 7).

[0031] Next, the training data generation unit 22 assigns a label to each of the obtained training patch images LI. As described above, in each piece of training data included in the training dataset, a label is assigned to each input image, i.e., each entire image WI. Therefore, the training data generation unit 22 uses the label assigned to the entire image WI to which each partial image belongs as the label of each training patch image LI generated from that partial image. In this way, the training data generation unit 22 assigns labels to all training patch images LI and ends the generation of training data. Note that the training data generation unit 22 generates training data for multiple input images.

[0032] The learning unit 23 learns a prediction model using the learning data input from the learning data generation unit 22. The prediction model predicts the probability that a predetermined image feature is included in an input image (whole image WI). In this embodiment, the prediction model predicts the probability that a predetermined feature such as the above-mentioned tumor, stroma, or duct is included in a pathological tissue image of a patient, and outputs a confidence score for each feature. For example, a deep learning model such as a CNN (Convolutional Neural Network) can be used as the prediction model. The learning unit 23 performs a first learning round using the input learning data, and outputs the learned prediction model to the prediction unit 24.

[0033] The prediction unit 24 uses the trained prediction model to make predictions for all partial images that make up the input image, calculates predicted values, and outputs them to the selection probability update unit 25 and the integration unit 26. The prediction unit 24 performs this process for all input images included in the training dataset.

[0034] The selection probability update unit 25 calculates a prediction result corresponding to each spatial region based on the predicted value output by the prediction unit 24 using the trained prediction model. Specifically, the selection probability update unit 25 calculates the average value of the predicted values ​​for multiple subregions included in the spatial region SA1 as the prediction result for the spatial region SA1. For example, if the average value is equal to or greater than a predetermined threshold, the selection probability update unit 25 sets the prediction result to "+1," and if it is less than the threshold, the selection probability update unit 25 sets the prediction result to "-1." The selection probability update unit 25 similarly performs this process for the other spatial regions SA2 to SA4, generating prediction results for each spatial region.

[0035] Next, the selection probability update unit 25 updates the region selection probability of each spatial region using the prediction result for each spatial region. Specifically, the selection probability update unit 25 updates the region selection probability so that a spatial region with a high degree of confidence in the prediction result by the prediction model is more likely to be selected in the next training data.

[0036] In a preferred example, the selection probability update unit 25 calculates the selection probability D of the i-th spatial region SAi using the following equation (1): t+1 Calculate (i).

[0037]

number

[0038]

number

[0039]

number

[0040] The selection probability update unit 25 outputs the updated region selection probabilities for each spatial region to the training data generation unit 22. The training data generation unit 22 uses the input updated region selection probabilities to generate training data to be used in the next training.

[0041] In this way, the learning loop is repeated a predetermined number of times while updating the region selection probability of each spatial region. Then, when the predetermined number of learning loops is completed, the prediction unit 24 performs predictions for all partial images included in the input image using the trained model at the time of completion of learning (also called the "final model"), and outputs the predicted values ​​to the integration unit 26.

[0042] The integrating unit 26 integrates the predicted values ​​for each partial image output by the predicting unit 24 to generate a prediction result for the entire input image. For example, for each input image, the integrating unit 26 sets the average value of the predicted values ​​for all partial images that make up the input image as the predicted value for the entire input image. The integrating unit 26 then displays the predicted value for the entire input image on the display device 17 together with the input image.

[0043] In the above configuration, the image dividing unit 21 is an example of a partial image generating means, the learning data generating unit 22 is an example of a learning data generating means, the learning unit 23 is an example of a learning means, and the prediction unit 24 is an example of a prediction means.

[0044] (Learning process) 8 is a flowchart of the learning process by the learning device 100. This process is realized by the processor 13 shown in FIG. 2 executing a program prepared in advance and operating as each element shown in FIG.

[0045] First, the image dividing unit 21 receives a training dataset and divides each input image included in the training dataset into partial images (step S10). Next, the training data generating unit 22 maps the partial images for each input image into a feature space (step S11). Next, the training data generating unit 22 divides the feature space into a plurality of spatial regions (step S12). In this case, the training data generating unit 22 may divide the feature space by clustering as described above, or may divide the feature space by grid division.

[0046] Next, the training data generation unit 22 selects one spatial region based on the region selection probability set for each spatial region (step S13). Note that, in the initial state, the region selection probabilities for multiple spatial regions are set to the same value. Next, the training data generation unit 22 generates training patch images from a predetermined number of partial images belonging to the selected spatial region (step S14). Next, the training data generation unit 22 assigns a label to each training patch image and generates training data (step S15).

[0047] Next, the learning unit 23 learns a prediction model using the training data generated in step S15, and generates a trained prediction model (step S16). Next, the prediction unit 24 determines whether the learning unit 23 has learned a predetermined number of times (step S14). That is, the prediction unit 24 determines whether the above-mentioned learning loop has been repeated a predetermined number of times. If the learning unit 23 has not learned a predetermined number of times (step S17: No), the prediction unit 24 uses the trained prediction model generated in step S16 to make predictions for partial images included in all input images in the training dataset (step S18).

[0048] Next, the selection probability update unit 25 updates the region selection probability of each spatial region based on the prediction result of each spatial region by the prediction unit 24 (step S19). As a result, the spatial region with a higher reliability of the prediction result by the trained prediction model has a higher region selection probability, and is more likely to be selected in the next and subsequent generation of training data. Then, the process returns to step S13.

[0049] In this way, the learning loop of steps S13 to S19 is repeated until the learning unit 23 has performed learning a predetermined number of times. Then, when the learning unit 23 has performed learning a predetermined number of times (step S14: Yes), the learning process ends.

[0050] (Presenting the basis for model predictions) By displaying the region selection probability at the end of the learning on the display device 17 or the like, it becomes possible to present the basis for predictions made by the prediction model. FIG. 9 shows an example of the display of the region selection probability at the end of learning. In this example, for each spatial region, the tissue corresponding to that spatial region and the region selection probability for that spatial region are displayed. This display makes it possible to show the frequency with which the tissue corresponding to each spatial region was selected in the learning of the prediction model. For example, if the prediction model is a model that predicts the effect of medication on each tissue, it is possible to know which tissues have the greatest effect from medication.

[0051] (Variation) Next, a modified example of the learning device according to the first embodiment will be described. As shown in Fig. 4, the learning device 100 divides the entire pathological tissue region included in the entire image WI into a plurality of partial images PI and maps them in a feature space. This is effective in ensuring the diversity of the pathological tissues represented by the partial images PI, but since normal regions and regions clearly unrelated to cancer are also mapped, there is a possibility that unnecessary processing will increase.

[0052] Therefore, in this modified example, the learning device detects tumor cells, lymphocytes, etc. from the pathological tissue region included in the entire image WI using a tumor cell detection method, selects partial images PI with a high proportion of tumor cells, and maps them into the feature space. This narrows down the targets for mapping into the feature space to the segmented images PI of the cancer and its surroundings (regions thought to be highly related to drug efficacy), thereby reducing the amount of calculation.

[0053] FIG. 10 shows the functional configuration of a learning device 100x according to a modified example. As can be seen from a comparison with FIG. 3, in the learning device 100x, an image narrowing-down unit 28 is provided between the image dividing unit 21 and the learning data generating unit 22. The image narrowing-down unit 28 performs tumor cell detection processing on the entire image WI to detect tumor cells and the like. An existing tumor cell detection method can be used to detect tumor cells. An example of a tumor cell detection method is described in the following document. Cosatto, E., Gerard, K., Graf, HP, Ogura, M., Kiyuna, T., Hatanaka, KC, ... & Hatanaka, Y. (2021, February). A multi-scale conditional deep model for tumor cell ratio counting. In Medical Imaging 2021: Digital Pathology (Vol. 11603, pp. 31-42). SPIE.

[0054] FIG. 11(A) shows an example of tumor cell detection using an existing tumor cell detection method. In FIG. 11(A), the density of tumor cells is represented by shades of gray, with higher brightness indicating a higher proportion of tumor cells. Based on the tumor cell detection results shown in FIG. 11(A), the image filtering unit 28 selects a segmented image PI from a region with a high local peak in the tumor cell density distribution and outputs the segmented image PI to the training data generation unit 22. Specifically, as shown in FIG. 11(B), the image filtering unit 28 selects a partial image PI from a region with a high proportion of tumor cells, i.e., a region with high brightness, in FIG. 11(A), and outputs the partial image PI to the training data generation unit 22 as a target for mapping to the feature space. This reduces the computational load while improving the accuracy of the model.

[0055] [Prediction device] Next, a prediction device that performs prediction using a trained model generated by the above-described learning device will be described.

[0056] (Hardware configuration) The hardware configuration of the prediction device is basically the same as that of the learning device 100 shown in Fig. 2. However, unlike the learning device 100, the IF 12 acquires the input image to be predicted from a database or the like, and the display device 17 displays the prediction result for the input image to be predicted.

[0057] (Functional configuration) 12 is a block diagram showing the functional configuration of a prediction device 200. Functionally, the prediction device 200 includes an image dividing unit 31, a prediction unit 32, and an integration unit 33. The output of the integration unit 33 is supplied to the display device 17.

[0058] At the time of actual prediction, an input image to be predicted is prepared. In this embodiment, a pathological tissue image of a patient receiving medication is input to the image segmentation unit 31 as the input image to be predicted. The image segmentation unit 31 divides the input image, i.e., the entire image WI, into multiple partial images PI using the same method as during learning, and further divides each partial image PI into patch images (hereinafter also referred to as "prediction patch images") of the same size as the learning patch images LI. That is, the image segmentation unit 31 divides the input image to be predicted into sizes corresponding to the input to the trained prediction model. The image segmentation unit 31 then outputs the prediction patch images obtained by the division to the prediction unit 32.

[0059] The prediction unit 32 makes a prediction for the input image using the trained prediction model obtained by the above learning process. Specifically, the prediction unit 32 makes a prediction using the trained prediction model for each prediction patch image obtained by the image division unit 31, and outputs the predicted value to the integration unit 33.

[0060] The integrating unit 33 integrates the predicted values ​​calculated by the predicting unit 32 for each prediction patch image to calculate a predicted value for the entire input image, and outputs the calculated predicted value to the display device 17. The integrating unit 33 may output the calculated predicted value to an external device. This provides the probability that the input pathological tissue image contains tissue with a predetermined characteristic (such as a tumor, stroma, or duct in the previous example). The display device 17 then displays the predicted value and an important region on the input image. Furthermore, the integrating unit 33 may extract, in the prediction for the input image, a region of a partial image whose predicted value is equal to or greater than a predetermined reference value as an important region for the prediction, and display the extracted region on the display device 17. In this way, the prediction result for the input image is output.

[0061] The prediction model used by the prediction unit 32 is basically the prediction model at the end of a predetermined number of learning loops, i.e., the final model. However, since the final model does not necessarily have the highest accuracy, the prediction unit 32 may use the prediction model obtained in each learning loop with the smallest prediction error. Specifically, the prediction results of the prediction model obtained at the end of each learning loop may be compared with the learning dataset to calculate the prediction error, and the prediction model with the smallest prediction error may be adopted.

[0062] Furthermore, in the above example, one of the multiple prediction models obtained through multiple learning loops is used for prediction in the prediction unit 32, but instead, several of the multiple prediction models obtained may be used in combination. Specifically, the prediction unit 32 may perform prediction on an input image using multiple prediction models obtained through multiple learning loops, and may output a final prediction result by weighting and adding the multiple prediction results obtained. In this case, a greater weight may be assigned to the output of a prediction model with higher accuracy.

[0063] (inference processing) 13 is a flowchart of the prediction process by the prediction device 200. This process is realized by the processor 13 shown in FIG. 2 executing a program prepared in advance and operating as each element shown in FIG.

[0064] First, the image dividing unit 31 divides the input image into partial images, and further divides each partial image into prediction patch images (step S21). Next, the prediction unit 32 uses a trained prediction model obtained by the learning process to make a prediction for each prediction patch image and output a predicted value (step S22). Next, the integration unit 33 integrates the predicted values ​​for each prediction patch image and calculates a prediction result for the entire input image (step S23). Then, the display device 17 displays the prediction result (step S24). Then, the prediction process ends.

[0065] [Application example] An example in which the prediction device of this embodiment is applied to predicting the effect of medication (drug efficacy or success, hereinafter referred to as "drug efficacy") will be described in detail. In the medical field, pathological tissue images are used as input, and drug efficacy is predicted based on the images. For example, there is a method of predicting drug efficacy based on the staining rate of an immune image, which is based on an image in which cells including pathological tissue are stained. In this case, there are problems such as variations in the determination of the staining rate depending on the examiner, and it is not clear which part of the image is the characteristic part that reflects the drug efficacy.

[0066] Therefore, using the prediction device of this embodiment, a pathological tissue image is input, and a prediction model obtained by learning is used to predict drug efficacy, output a drug efficacy prediction score, and display areas that are important for determining drug efficacy.

[0067] Specifically, during training, training data is prepared in which labels indicating the presence or absence of drug efficacy are attached to pathological tissue images, and the above-mentioned training process is performed to train a drug efficacy prediction model. In this case, the method of this embodiment only requires labeling the entire pathological tissue image, so labeling is possible even if it is not clear which part of the pathological tissue image affects drug efficacy. Then, during prediction, the pathological tissue image to be predicted is input, and a drug efficacy score can be predicted using the trained prediction model. Furthermore, regions of the partial image that show high scores in the prediction process can be extracted as important regions that have a significant impact on drug efficacy and displayed on a display device.

[0068] When this embodiment is applied to predicting drug efficacy, the image segmentation unit 21, during learning and prediction, may utilize background knowledge, such as that information around the cell nucleus is particularly important for diagnosis, to generate a partial image centered on the position of the cell nucleus in the input image.

[0069] Second Embodiment 14 is a block diagram showing the functional configuration of a learning device according to the second embodiment. The learning device 70 includes a partial image generation unit 71, a feature space generation unit 72, a learning data generation unit 73, a learning unit 74, and a prediction unit 75.

[0070] FIG. 15 is a flowchart of processing by the learning device 70 of the second embodiment. The partial image generation means 71 generates, from an input image, partial images smaller than the input image (step S71). The feature space generation means 72 generates a feature space in which feature quantities of multiple partial images are mapped for each input image (step S72). The training data generation means 73 acquires multiple training partial images from the multiple partial images based on the feature space to generate training data (step S73). The learning means 74 uses the training data to train a prediction model that predicts the probability that a specific feature is included in the training partial images (step S74). The prediction means 75 uses the trained prediction model to make predictions for all or some of the partial images included in the input image (step S75). Furthermore, the training data generation means 73 acquires multiple training partial images to be used as training data in the next training based on predicted values ​​for all or some of the partial images in the feature space (step S76). In this way, the training partial images are updated, and the learning means 74 repeatedly trains the prediction model.

[0071] According to the learning device 70 of the second embodiment, even in a situation where detailed labels are not given to the entire image, it is possible to accurately learn a model while selecting areas that are effective as learning data.

[0072] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0073] (Appendix 1) a partial image generating means for generating, from an input image, a partial image smaller than the input image; a feature space generating means for generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; a training data generation means for acquiring a plurality of training partial images from the plurality of partial images based on the feature space and generating training data; a learning means for learning a prediction model that predicts the probability that a predetermined feature is included in the learning partial image using the learning data; a prediction means for performing predictions for all or some of the partial images included in the input image using a trained prediction model; Equipped with The learning data generation means is a learning device that acquires a plurality of partial learning images to be used as learning data in the next learning, based on predicted values ​​for all or some of the partial images in the feature space.

[0074] (Appendix 2) The learning data generation means a spatial domain dividing means for dividing the feature space into a plurality of spatial domains; an acquisition means for determining a selection probability of the plurality of spatial regions and acquiring the plurality of learning partial images from a plurality of partial images corresponding to the spatial regions selected in accordance with the selection probability; 2. The learning device according to claim 1, comprising:

[0075] (Appendix 3) 3. The learning device according to claim 2, wherein the learning data generation means updates the selection probabilities of the plurality of spatial regions based on predicted values ​​for all or some of the partial images included in the input image.

[0076] (Appendix 4) The learning device described in Appendix 3, wherein the learning data generation means updates the selection probabilities of the plurality of spatial regions so that the higher the confidence level of the predicted value for the partial image, the higher the selection probability of the spatial region corresponding to the partial image.

[0077] (Appendix 5) The learning device according to claim 2, wherein the spatial region division means divides the feature space into a plurality of spatial regions by mapping features of the plurality of partial images onto the feature space and clustering the distribution of the features.

[0078] (Appendix 6) 2. The learning device according to claim 1, wherein the learning data generation means uses a label previously assigned to the input image as a label for each learning partial image acquired from the input image.

[0079] (Appendix 7) The learning device according to claim 1, further comprising an output means for outputting the selection probability of the plurality of spatial regions at the end of learning of the trained prediction model. (Appendix 8) The learning device according to claim 1, wherein the feature space generation means selects a partial image with a high proportion of tumor cells from among a plurality of partial images generated from the input image and maps it to the feature space.

[0080] (Appendix 9) a partial image generating means for generating, from an input image, a partial image smaller than the input image; A prediction means for predicting the probability that a predetermined feature is included in each generated partial image using a prediction model trained by the learning device according to any one of Supplementary Notes 1 to 7; an output means for integrating the prediction results for all the partial images and outputting a prediction score indicating the probability that the predetermined feature is included in the input image; A prediction device comprising:

[0081] (Appendix 10) 1. A computer-implemented method of learning, comprising: Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; Using the trained prediction model, prediction is made for all or some of the partial images included in the input image; The generation of the training data is a learning method in which a plurality of training partial images to be used as training data in the next learning is obtained based on predicted values ​​for all or some of the partial images in the feature space.

[0082] (Appendix 11) Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; Using the trained prediction model, predictions are made for all or some of the partial images included in the input image; The generation of the learning data is based on predicted values ​​for all or some of the partial images in the feature space, and the generation of the learning data is based on the predicted values ​​for all or some of the partial images in the feature space.

[0083] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. [Explanation of symbols]

[0084] 13 processors 17 Display device 21, 31 Image division section 22 Learning data generation unit 23 Learning Department 24, 32 Prediction Section 25 Selection probability update section 26, 33 Integrated Department 100 Learning Device 200 Prediction Device

Claims

1. a partial image generating means for generating, from an input image, a partial image smaller than the input image; a feature space generating means for generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; a training data generation means for acquiring a plurality of training partial images from the plurality of partial images based on the feature space and generating training data; a learning means for learning a prediction model that predicts the probability that a predetermined feature is included in the learning partial image using the learning data; a prediction means for performing predictions for all or some of the partial images included in the input image using a trained prediction model; Equipped with The learning data generation means is a learning device that acquires a plurality of partial learning images to be used as learning data in the next learning, based on predicted values ​​for all or some of the partial images in the feature space.

2. The learning data generation means a spatial domain dividing means for dividing the feature space into a plurality of spatial domains; an acquisition means for determining a selection probability of the plurality of spatial regions and acquiring the plurality of learning partial images from a plurality of partial images corresponding to the spatial regions selected in accordance with the selection probability; The learning device according to claim 1 , comprising:

3. The learning device according to claim 2 , wherein the learning data generating means updates the selection probabilities of the plurality of spatial regions based on predicted values ​​for all or some of the partial images included in the input image.

4. The learning device according to claim 3, wherein the learning data generation means updates the selection probabilities of the plurality of spatial regions so that the higher the confidence level of the predicted value for the partial image, the higher the selection probability of the spatial region corresponding to the partial image.

5. The learning device according to claim 2 , wherein the spatial region dividing means divides the feature space into a plurality of spatial regions by mapping feature amounts of the plurality of partial images onto the feature space and clustering the distribution of the feature amounts.

6. The learning device according to claim 2 , further comprising an output means for outputting the selection probabilities of the plurality of spatial regions at the end of learning of the trained prediction model.

7. The learning device according to claim 1 , wherein the feature space generating means selects a partial image with a high proportion of tumor cells from among a plurality of partial images generated from the input image and maps the selected partial image to the feature space.

8. a partial image generating means for generating, from an input image, a partial image smaller than the input image; a prediction means for predicting the probability that a predetermined feature is included in each generated partial image by using a prediction model trained by the learning device according to any one of claims 1 to 7; an output means for integrating the prediction results for all the partial images and outputting a prediction score indicating the probability that the predetermined feature is included in the input image; A prediction device comprising:

9. 1. A computer-implemented method of learning, comprising: Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; Using the trained prediction model, prediction is made for all or some of the partial images included in the input image; The generation of the training data is a learning method in which a plurality of training partial images to be used as training data in the next learning is obtained based on predicted values ​​for all or some of the partial images in the feature space.

10. Generate a partial image from the input image that is smaller than the input image; generating a feature space in which feature amounts of a plurality of partial images are mapped for each of the input images; generating training data by acquiring a plurality of training partial images from the plurality of partial images based on the feature space; using the training data to train a prediction model that predicts the probability that a predetermined feature is included in the training partial image; causing a computer to execute a process of making predictions for all or some of the partial images included in the input image using the trained prediction model; The generation of the learning data is a program that acquires multiple learning partial images to be used as learning data in the next learning based on predicted values ​​for all or some of the partial images in the feature space.

Images