Information processing device and information processing method
The information processing device uses unsupervised learning to cluster cells in pathological images, addressing the challenge of classifying difficult cell types and enhancing prognosis prediction by maintaining feature accuracy and reducing processing time.
Patent Information
- Application Number
- JP2021163374
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-10-04
- Publication Date
- 2024-11-27
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing systems for classifying biological cells in pathological images struggle with accuracy when faced with cell types that are difficult for humans to recognize, affecting the prediction of organism prognosis.
An information processing device employs unsupervised learning to extract features from pathological images, generating clusters of cells using a combination of initial small clusters through unsupervised learning and hierarchical clustering, without relying on human-defined labels, and predicts prognosis based on these clusters.
The system effectively classifies cells into types that are challenging for humans, maintaining feature accuracy and improving prognosis prediction by reflecting difficult-to-recognize features, reducing processing time and overfitting.
Smart Images

Figure 2024164331000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device and an information processing method for outputting information related to a pathological image. [Background technology]
[0002] Patent Document 1 discloses a system in which an image of an organism's skin is input into a model generated by applying supervised learning to a set of cell images and their types, and the cells contained in the input image are classified into types such as dermal cells, epidermal cells, and immune cells. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2018 / 221625 Summary of the Invention [Problem to be solved by the invention]
[0004] The system disclosed in Patent Document 1 uses supervised learning to classify cells contained in an image into one of multiple types defined by humans as teacher data (training data). Therefore, the system disclosed in Patent Document 1 has a problem in that it cannot classify biological cells into types that are difficult for humans to recognize. Feature quantities related to classification of cells that are difficult for humans to recognize may affect the accuracy of prognosis prediction for biological organisms.
[0005] The present invention has been made in consideration of these points, and has an object to make it possible to classify cells in pathological images of living organisms into types that are difficult for humans to recognize. [Means for solving the problem]
[0006] An information processing device of a first aspect of the present invention has a feature extraction unit that extracts features of each of a plurality of cells constituting an organism in a pathology image of the organism, a cluster generation unit that generates a plurality of clusters by clustering the plurality of cells by applying unsupervised learning to the plurality of features, and an output unit that outputs information corresponding to the clusters.
[0007] The cluster generation unit may include a first clustering unit that generates a plurality of small clusters by clustering a plurality of the cells by applying the unsupervised learning, and a second clustering unit that generates a plurality of the clusters by clustering the plurality of the small clusters by performing hierarchical clustering.
[0008] The output unit may output, as the information corresponding to the cluster, information indicating a position of each of the plurality of cells and a cluster to which the cell belongs, onto the pathological image.
[0009] The output unit may output information about the cells belonging to each of the plurality of clusters as the information corresponding to the cluster, in association with each of the plurality of clusters.
[0010] The feature extraction unit may extract the feature by applying machine learning to the pathological image.
[0011] The information processing device may further include an image acquisition unit that acquires a high-resolution pathology image output by inputting the pathology image into a high-resolution model that increases the resolution of an input image and outputs the increased resolution, and the feature extraction unit may extract the features from the high-resolution pathology image.
[0012] The high-resolution model may be generated by applying machine learning to a low-resolution image of a region including cells of the organism and a high-resolution image of the region.
[0013] The information processing device may further have an image acquisition unit that acquires a color-corrected pathology image generated by performing a process to adjust the color of the pathology image to a reference value, and the feature extraction unit may extract the feature from the color-corrected pathology image.
[0014] The reference value may be a statistical value of colors of a plurality of images captured under a plurality of imaging conditions.
[0015] The reference value may be a color of a predetermined cell stained with a predetermined dye, and the image acquisition unit may match a color of an area in the pathological image corresponding to the predetermined cell to the reference value.
[0016] The feature extraction unit may extract features of learning cells, which are cells appearing in a plurality of images to be learned, from each of the plurality of images to be learned, and the cluster generation unit may generate a plurality of learning clusters by clustering the plurality of learning cells based on the features of the plurality of learning cells. The cluster generation unit may generate a classification model by supervised learning using the generated plurality of learning clusters as pseudo-ground truth labels. The cluster generation unit may store the classification model in a memory unit as a clustering model, and the cluster generation unit may use the clustering model stored in the memory unit to generate a plurality of clusters by clustering a plurality of the cells appearing in the pathology image.
[0017] The information processing device may further include a prognosis prediction unit that predicts a prognosis of the organism based on information corresponding to the cluster.
[0018] The prognosis prediction unit may predict the prognosis using a prognosis prediction model generated by applying machine learning to the relationship between information corresponding to the cluster and the prognosis.
[0019] The prognosis prediction unit may predict the prognosis when the drug is administered to the organism using the prognosis prediction model generated using the prognosis of the organism to which the drug has been administered.
[0020] The prognosis prediction unit may predict the prognosis based on the feature amounts of the cells belonging to each of the multiple clusters.
[0021] The prognosis prediction unit may predict the prognosis based on the feature amounts of the cells belonging to each of the multiple clusters in a predetermined tissue.
[0022] The prognosis prediction unit may predict the prognosis based on positions of the cells belonging to each of the multiple clusters.
[0023] The prognosis prediction unit may predict the prognosis based on data generated by combining a second pathological image and a mask image indicating the positions of the cells belonging to each of a plurality of clusters generated from the first pathological image.
[0024] The first pathology image may be an image of a specimen of the biological body stained with a first dye, and the second pathology image may be an image of the specimen stained with a second dye different from the first dye.
[0025] An information processing method of a second aspect of the present invention includes the steps of: extracting features of each of a plurality of cells constituting an organism from a pathology image of the organism, executed by a processor; generating a plurality of clusters by clustering the cells by applying unsupervised learning to the plurality of features; and outputting information corresponding to the clusters. Effect of the Invention
[0026] According to the present invention, an effect is achieved in that cells in a pathological image of an organism can be classified into types that are difficult for humans to recognize. [Brief description of the drawings]
[0027] [Figure 1] 1 is a diagram showing an overview of an information processing system according to an embodiment. [Diagram 2] 1 is a block diagram of an information processing system according to an embodiment. [Diagram 3] 4 is a schematic diagram for explaining a process performed by an image acquisition unit on a pathological image. FIG. [Figure 4] FIG. 4 is a schematic diagram for explaining a method in which a first clustering unit generates a clustering model by unsupervised learning. [Diagram 5] FIG. 13 is a schematic diagram for explaining a method in which a second clustering unit generates clusters by hierarchical clustering. [Figure 6] 13 is a schematic diagram for explaining a method in which a prognosis prediction unit generates a prognosis prediction model. FIG. [Figure 7] FIG. 1 is a schematic diagram for explaining a method for generating a prognosis prediction model for predicting prognosis using a plurality of pathological images stained with different dyes. [Figure 8] 4 is a schematic diagram of information output by an output unit; FIG. [Figure 9] FIG. 1 is a flowchart illustrating an exemplary information processing method executed by an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] [Outline of information processing system] 1 is a diagram showing an overview of an information processing system according to this embodiment. The information processing system includes an information processing device 1 and an information terminal 2. The information processing system may also include other devices such as a server and a terminal.
[0029] The information processing device 1 is a computer that clusters cells of an organism captured in a pathology image by unsupervised learning and outputs information about the generated clusters. A pathology image is a captured image for the purpose of pathological diagnosis, and is generated, for example, by capturing an image of a specimen taken from the body of an organism with a camera. The information processing device 1 transmits information to an information terminal 2 and receives information from the information terminal 2 by wired communication or wireless communication.
[0030] The information terminal 2 is a computer used by a user, such as a smartphone, a tablet terminal, or a personal computer. The user is, for example, a medical professional who uses the information processing system to analyze pathological images of patients. The information terminal 2 has a display unit such as a liquid crystal display, and an operation unit such as a keyboard or a touch panel. The information terminal 2 displays information received from the information processing device 1 on the display unit, and transmits information input using the operation unit to the information processing device 1.
[0031] An overview of the processing executed by the information processing system according to this embodiment will be described below. In response to an operation by a user, the information terminal 2 transmits a pathological image of a living organism, such as a patient, to be subjected to a pathological diagnosis to the information processing device 1 (1). The information processing device 1 acquires the pathological image transmitted by the information terminal 2.
[0032] The information processing device 1 extracts features of each of a plurality of cells constituting an organism from the acquired pathological image (2). The features are, for example, features extracted from the pathological image by a convolutional neural network (CNN).
[0033] The information processing device 1 applies unsupervised learning to the feature amounts of a plurality of cells appearing in a pathological image, thereby generating a plurality of clusters by clustering the plurality of cells. In the present application, unsupervised learning is a machine learning method that is performed without a human being manually providing correct answer data (correct answer label). In other words, the information processing device 1 clusters the plurality of cells appearing in the pathological image based on the feature amounts of the plurality of cells themselves.
[0034] The information processing device 1 outputs information corresponding to the generated clusters. For example, the information processing device 1 outputs, to the information terminal 2, information corresponding to the clusters, such as information associating the clusters with cells, and information indicating the prognosis of an organism predicted based on the clusters.
[0035] In this way, the information processing device 1 clusters a plurality of cells in a pathological image into a plurality of clusters by unsupervised learning without using a learning model generated in advance by applying supervised learning to teaching data. In supervised learning, cells are clustered into a plurality of types defined by humans, and therefore features that do not contribute to the clustering into the corresponding types tend to be lost. On the other hand, the information processing device 1 uses unsupervised learning to cluster cells based on the features of the plurality of cells themselves in the pathological image, and therefore can classify cells into types that are difficult for humans to recognize, and can generate clusters that reflect features that would be lost in supervised learning.
[0036] [Information processing system configuration] FIG. 2 is a block diagram of an information processing system according to this embodiment. In FIG. 2, arrows indicate main data flows, and data flows other than those shown in FIG. 2 may exist. In FIG. 2, each block indicates a configuration in functional units, not a configuration in hardware (device) units. Therefore, the blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between blocks via any means, such as a data bus, a network, or a portable storage medium.
[0037] The information processing device 1 includes a storage unit 11 and a control unit 12. The information processing device 1 may be configured by connecting two or more physically separate devices by wire or wirelessly. The information processing device 1 may also be configured by a cloud, which is a collection of computer resources.
[0038] The storage unit 11 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk drive, etc. The storage unit 11 stores in advance a program to be executed by the control unit 12. The storage unit 11 may be provided outside the information processing device 1, in which case data may be exchanged between the storage unit 11 and the control unit 12 via a network.
[0039] The storage unit 11 has an image storage unit 111, a cell information storage unit 112, and a cluster information storage unit 113. The image storage unit 111 stores a pathological image acquired by an image acquisition unit 121 described below. The cell information storage unit 112 stores cell information about cells extracted from the pathological image. The cluster information storage unit 113 stores cluster information about clusters generated by clustering cells. The image storage unit 111, the cell information storage unit 112, and the cluster information storage unit 113 may each be a storage area on the storage unit 11, or may be a database configured on the storage unit 11.
[0040] The control unit 12 has an image acquisition unit 121, a feature extraction unit 122, a cluster generation unit 123, a prognosis prediction unit 124, and an output unit 125. The control unit 12 is a processor such as a CPU (Central Processing Unit), and functions as the image acquisition unit 121, the feature extraction unit 122, the cluster generation unit 123, the prognosis prediction unit 124, and the output unit 125 by executing a program stored in the storage unit 11. At least a part of the functions of the control unit 12 may be executed by an electric circuit. Also, at least a part of the functions of the control unit 12 may be realized by the control unit 12 executing a program executed via a network.
[0041] The process executed by the information processing device 1 will be described in detail below. A user specifies a pathological image to be the subject of pathological diagnosis on the information terminal 2. A pathological image is an image generated by staining a biological specimen of a patient or the like using a predetermined method (e.g., hematoxylin-eosin staining) and capturing an image of the stained specimen with a camera. The information terminal 2 transmits the specified pathological image to the information processing device 1.
[0042] In the information processing device 1, the image acquiring unit 121 acquires the pathological image transmitted by the information terminal 2. The image acquiring unit 121 may acquire the pathological image stored in advance in the storage unit 11, or may acquire the pathological image stored in advance in a storage device on a network such as the Internet. The image acquiring unit 121 stores the acquired pathological image in the image storage unit 111.
[0043] Image acquisition unit 121 performs a predetermined process on the pathological image to acquire images of each of a plurality of cells constituting an organism that are shown in the pathological image. Fig. 3 is a schematic diagram for explaining the process that image acquisition unit 121 performs on the pathological image.
[0044] The image acquisition unit 121 performs a high-resolution process on the pathological image. For example, a high-resolution model that increases the resolution of an input image and outputs the increased resolution is stored in advance in the storage unit 11. The high-resolution model is generated by applying a known generative model such as a Generative Adversarial Network (GAN) to a low-resolution image of an area including cells of an organism and a high-resolution image of the area.
[0045] The image acquisition unit 121 acquires a high-resolution pathological image outputted by inputting the pathological image into a high-resolution model stored in the storage unit 11 as a high-resolution process, for example. The image acquisition unit 121 stores the high-resolution pathological image in the image storage unit 111 and uses it in subsequent processes. This enables the information processing device 1 to classify a plurality of cells shown in a pathological image into a number of clusters far exceeding the number of types that a human can recognize in unsupervised learning described below, even in the case of a low-resolution pathological image.
[0046] Furthermore, image acquiring unit 121 performs color correction processing on the pathological image. For example, image acquiring unit 121 acquires a pathological image with corrected color generated by performing processing on the pathological image to match the color to a reference value as the color correction processing.
[0047] The reference value of color is, for example, a statistical value of color of a plurality of images captured under a plurality of imaging conditions. In this case, the image acquisition unit 121 calculates statistical values such as the average color value and variance from a plurality of images captured in a plurality of different facilities or a plurality of images captured in one facility using a plurality of different equipment. The image acquisition unit 121 corrects the color of the pathological image so that the color of the entire pathological image matches the calculated statistical value.
[0048] The reference color value may be, for example, a color of a predetermined cell stained with a predetermined dye. In this case, the memory unit 11 stores in advance the color of a predetermined cell, such as a macrophage, stained with a predetermined method, such as hematoxylin-eosin staining. The image acquisition unit 121 corrects the color of the pathological image so that the color of the area in the pathological image corresponding to the predetermined cell matches the color of the predetermined cell stored in the memory unit 11.
[0049] The image acquisition unit 121 stores the color-corrected pathological image in the image storage unit 111 for use in subsequent processing. This enables the information processing device 1 to correct color differences between pathological images and suppress blurring of the clustering results due to the influence of imaging conditions, etc.
[0050] Image acquiring section 121 may perform only one of the high-resolution processing and the color correction processing on the pathological image. Image acquiring section 121 does not have to perform the high-resolution processing and the color correction processing on the pathological image.
[0051] The image acquisition unit 121 performs a segmentation process on the pathological image to acquire images of each of a plurality of cells appearing in the pathological image as cell images. For example, an instance segmentation model for extracting cells appearing in an input image is stored in advance in the storage unit 11. The instance segmentation model is generated by applying a known machine learning method such as Mask Region-Convolutional Neural Network (Mask R-CNN) to an image showing cells of an organism and the positions of the cells in the image.
[0052] For example, as a segmentation process, the image acquisition unit 121 extracts a plurality of cells appearing in the pathological image by inputting the pathological image into an instance segmentation model stored in the storage unit 11. The image acquisition unit 121 stores an image of each of the extracted cells as a cell image in the image storage unit 111, and uses the image in the subsequent processes. The cell image of each of the cells is associated with the position of the cell in the pathological image. The image acquisition unit 121 is not limited to the specific method shown here, and may generate a cell image of each of the cells appearing in the pathological image by other methods.
[0053] The cluster generating unit 123 applies unsupervised learning to a plurality of feature amounts corresponding to a plurality of cells extracted by the feature extracting unit 122, thereby generating a plurality of clusters by clustering the plurality of cells.
[0054] A method for generating multiple clusters by the cluster generating unit 123 will be described below. The cluster generating unit 123 includes a first clustering unit 123a and a second clustering unit 123b. First, the first clustering unit 123a applies unsupervised learning to the features extracted by the feature extracting unit 122 to generate in advance a clustering model that generates multiple small clusters by clustering multiple cells.
[0055] 4 is a schematic diagram for explaining a method in which the first clustering unit 123a generates a clustering model by unsupervised learning. The feature extraction unit 122 and the first clustering unit 123a generate a clustering model that extracts features and performs clustering by applying a convolutional neural network, which is a known machine learning method, to a plurality of cell images, for example, by the method described below.
[0056] The first clustering unit 123a first initializes the parameters of a convolutional neural network that outputs features from an input image with random or predetermined values. The feature extraction unit 122 inputs each of a plurality of cell images to be learned into the convolutional neural network, thereby extracting features of cells (learning cells) depicted in the cell images.
[0057] The first clustering unit 123a calculates the similarity between the plurality of cells of the feature amounts extracted by the feature amount extraction unit 122. The similarity is, for example, Euclidean distance, Manhattan distance, cosine distance, or other value that indicates the degree to which two feature amounts are similar. The first clustering unit 123a clusters the plurality of cells into a plurality of small clusters (learning clusters) based on the calculated similarity, for example, by using the k-means method. Here, the first clustering unit 123a generates a predetermined number of small clusters (for example, a number on the order of 100 to 1000 designated by a user). The first clustering unit 123a may cluster the plurality of cells into a plurality of small clusters by using other clustering methods that allow clustering based on similarity.
[0058] If this is the second or subsequent generation of small clusters, the first clustering unit 123a determines whether the clustering result of the current small cluster has changed significantly from the clustering result of the previous small cluster (for example, whether the change in the center of gravity of all small clusters is equal to or greater than a predetermined value).
[0059] When it is determined that the clustering result of the current small cluster has changed significantly from the clustering result of the previous small cluster, or when the small cluster is generated for the first time, the first clustering unit 123a assigns an ID (Identification) to each of the generated small clusters, and generates a classification model (neural network) by known machine learning using the ID as a pseudo-correct label. The classification model is configured to include a convolutional neural network for extracting features from the above-mentioned images, and the parameters of the convolutional neural network are updated by machine learning. The first clustering unit 123a inputs each of the multiple cell images again to the convolutional neural network with the updated parameters, and repeats the extraction of features and clustering into small clusters.
[0060] When it is determined that the clustering result of the current small cluster has not changed significantly from the clustering result of the previous small cluster, the first clustering unit 123a ends the machine learning and stores the final classification model as a clustering model in the storage unit 11. This enables the information processing device 1 to obtain parameters of a convolutional neural network that can extract the feature amounts of cells captured in a pathological image and perform stable clustering without the need for a human to define the feature amounts.
[0061] Furthermore, the feature extraction unit 122 may calculate features that are predetermined values representing cell characteristics, such as the cell's perimeter, area, degree of unevenness (irregularity), and density, in addition to the features extracted by machine learning. In this case, the first clustering unit 123a clusters a plurality of cells into a plurality of small clusters using a value that combines the predetermined features, such as the cell's perimeter, area, degree of unevenness (irregularity), and density, with the features extracted by machine learning. This allows the information processing device 1 to reflect in the clustering result not only features that are difficult for humans to recognize, which are extracted by machine learning, but also specific features defined by humans.
[0062] After the image acquisition unit 121 generates a plurality of cell images from a pathological image, the feature extraction unit 122 extracts features of each of the plurality of cells by inputting each of the plurality of cell images into a clustering model previously stored in the storage unit 11, and stores cell information associating the cells with the features in the cell information storage unit 112. Furthermore, the first clustering unit 123a generates a plurality of small clusters by clustering the plurality of cells based on the features using the clustering model. This allows the information processing device 1 to extract features and cluster the plurality of cell images without receiving correct answer data from a human.
[0063] Next, second clustering unit 123b performs hierarchical clustering to generate a plurality of clusters by clustering the plurality of small clusters. Fig. 5 is a schematic diagram for explaining a method in which second clustering unit 123b generates clusters by hierarchical clustering.
[0064] The second clustering unit 123b calculates a representative value (e.g., a center of gravity) representing the feature amounts of the cells belonging to each of the small clusters generated by the first clustering unit 123a. The second clustering unit 123b extracts a pair of small clusters having the smallest distance between their representative values from among the small clusters. Next, the second clustering unit 123b extracts a pair of small clusters having the smallest distance between their representative values, excluding the pair extracted from the small clusters. The second clustering unit 123b repeats the extraction of pairs of small clusters until there are no more pairs of small clusters that can be extracted.
[0065] Furthermore, the second clustering unit 123b regards the set of extracted small clusters as one small cluster, calculates a representative value again, and repeats the process of extracting sets of small clusters up to a predetermined number of hierarchical levels. The second clustering unit 123b generates a plurality of sets of small clusters finally extracted as a plurality of clusters. The second clustering unit 123b stores cluster information indicating the small clusters and the cells belonging to each cluster, and the relationship between the small clusters and the clusters (e.g., a tree diagram) in the cluster information storage unit 113.
[0066] 5, the second clustering unit 123b generates clusters 1 to k by performing two-layer hierarchical clustering on the small clusters 1 to j. The second clustering unit 123b may perform hierarchical clustering with any number of layers other than two layers.
[0067] As a result, second clustering section 123b can hierarchically cluster the multiple small clusters generated by first clustering section 123a based on the similarity of the feature amounts, and consolidate them into a small number of clusters.
[0068] In this way, the information processing device 1 generates small clusters by clustering cells using unsupervised learning, and then generates clusters by clustering the small clusters using hierarchical clustering. By using unsupervised learning that does not require correct answer data to be provided by humans, the information processing device 1 can classify cells into types that are difficult for humans to recognize, and can generate clusters that retain more features that may affect the accuracy of prognosis prediction by the prognosis prediction unit 124 described below.
[0069] On the other hand, if a large number of clusters (on the order of 100 to 1000) generated by unsupervised learning are used as is to predict prognosis, the processing time for prognosis prediction may increase and overlearning may occur, resulting in a decrease in the accuracy of prognosis prediction. In response to this, the information processing device 1 can reduce the number of clusters by hierarchical clustering, thereby reducing the processing time for prognosis prediction and suppressing a decrease in the accuracy of prognosis prediction.
[0070] The prognosis prediction unit 124 predicts the prognosis of an organism depicted in the pathological image, based on information corresponding to the clusters generated by the cluster generation unit 123. The prognosis is expressed by values such as a 5-year survival rate, a 10-year survival rate, etc.
[0071] The method by which the prognosis prediction unit 124 predicts the prognosis of an organism will be described below. The prognosis prediction unit 124 generates in advance a prognosis prediction model that outputs a prognosis predicted from the input feature quantities of the clusters.
[0072] 6 is a schematic diagram for explaining a method for generating a prognosis prediction model by the prognosis prediction unit 124. The prognosis prediction unit 124, for example, acquires a plurality of images generated by capturing images of a plurality of living things, and acquires a plurality of clusters generated by applying a clustering model stored in the storage unit 11 to each of the plurality of images, and prognosis data indicating the prognosis of the living things captured in the images.
[0073] The prognosis prediction unit 124 generates a regression model that predicts a prognosis for the input features by applying known machine learning such as random forest, support vector machine (SVM), deep learning, etc. to the features of each of the multiple clusters and the prognosis indicated by the prognosis data.
[0074] The feature amounts used for prognosis prediction include, for example, feature amounts (e.g., cell number, average cell area, density, etc.) of multiple cells belonging to each of multiple clusters generated by applying the clustering model stored in the storage unit 11 to the images of each of the multiple organisms. The prognosis prediction unit 124 stores the generated regression model in the storage unit 11 as a prognosis prediction model.
[0075] Furthermore, the prognosis prediction unit 124 may generate a prognosis prediction model that predicts a prognosis based on the feature amount of cells belonging to each of a plurality of clusters in a predetermined tissue (e.g., tumor tissue). In this case, the prognosis prediction unit 124 acquires, for example, information that specifies an area corresponding to the predetermined tissue in the image. The information that specifies the area corresponding to the predetermined tissue is, for example, a mask image (such as a binary image) that indicates an area corresponding to the predetermined tissue in the image.
[0076] The prognosis prediction unit 124 extracts features to be used for prognosis prediction for cells located within an area corresponding to a specified tissue, among cells belonging to each of a plurality of clusters. The prognosis prediction unit 124 generates a prognosis prediction model by applying known machine learning to the extracted features. The features to be used for prognosis prediction may include, for example, features of the specified tissue itself (e.g., the area, degree of irregularity, density, etc. of tumor tissue). This allows the information processing device 1 to generate a prognosis prediction model that reflects the relationship between the clusters generated using unsupervised learning and the specified tissue.
[0077] After the cluster generating unit 123 generates a plurality of clusters based on the pathological image, the prognosis predicting unit 124 extracts, for example, a feature amount used for the above-mentioned prognosis prediction for each of the plurality of clusters, and inputs the feature amount to the prognosis prediction model stored in the storage unit 11. When using information on a predetermined tissue as a feature amount used for prognosis prediction, the prognosis predicting unit 124 may, for example, accept designation of an area corresponding to the predetermined tissue in the pathological image from the user at the information terminal 2.
[0078] The prognosis prediction unit 124 determines the prognosis output by the prognosis prediction model as the prognosis predicted from the pathological image. In this way, the information processing device 1 predicts the prognosis based on the clusters generated from the pathological image using unsupervised learning that does not require correct answer data to be provided by humans, and therefore can predict the prognosis reflecting features that are difficult for humans to recognize.
[0079] Furthermore, the prognosis prediction unit 124 may generate a prognosis prediction model by applying machine learning to the feature amount of a cluster generated from an image of an organism administered with a specific drug and the prognosis of the organism. In this case, the prognosis prediction unit 124 generates a prognosis prediction model using a plurality of clusters generated for an image of an organism administered with each of a plurality of drugs and the prognosis data of the organism, and stores the prognosis prediction model in the storage unit 11 in association with the drug.
[0080] After the cluster generation unit 123 generates a plurality of clusters based on the pathological images, the prognosis prediction unit 124 inputs the feature amount extracted from each of the plurality of clusters to each of a plurality of prognosis prediction models corresponding to a plurality of drugs stored in the storage unit 11. The prognosis prediction unit 124 determines the prognosis output by the prognosis prediction model corresponding to each of the plurality of drugs as the prognosis when the drug is administered to the organism. This allows the information processing device 1 to predict the prognosis for each drug administered to the organism based on the clusters generated from the pathological images using unsupervised learning.
[0081] Furthermore, the prognosis prediction unit 124 may generate a prognosis prediction model that predicts prognosis using a feature value that combines information obtained from a plurality of images stained by a plurality of different staining methods. Fig. 7 is a schematic diagram for explaining a method for generating a prognosis prediction model that predicts prognosis using a plurality of images stained with different dyes.
[0082] The prognosis prediction unit 124 acquires a first stained image, which is an image of a biological specimen stained with a first dye, and a second stained image, which is an image of the specimen stained with a second dye different from the first dye, in association with each other. If the biological specimen is sliced into sufficiently thin slices, and the slices shown in the first stained image and the slices shown in the second stained image are close to each other, the distribution of cells in the first stained image and the distribution of cells in the second stained image can be considered to be substantially the same.
[0083] The first dye is, for example, a dye for hematoxylin-eosin staining (HE staining) capable of staining a cellular structure. The second dye is, for example, a dye for immunohistochemical staining (IHC staining) capable of staining a specific protein. The second dye may include a plurality of dyes that stain a plurality of different proteins.
[0084] Prognosis prediction unit 124 acquires, for example, a plurality of clusters generated by applying a clustering model stored in storage unit 11 to each of a plurality of first stained images, and prognosis data indicating the prognosis of an organism appearing in the first stained image. Prognosis prediction unit 124 generates, for example, a mask image (such as a binary image) indicating the positions of cells belonging to each of the plurality of clusters generated from the first stained image.
[0085] The prognosis prediction unit 124 generates tensor data by combining a mask image generated from the first stained image with a second stained image associated with the first stained image. That is, the tensor data is information indicating the relationship between the positions of cells belonging to each of a plurality of clusters obtained by staining with the first dye and the distribution of proteins obtained by staining with the second dye.
[0086] The prognosis prediction unit 124 generates a regression model that predicts a prognosis for the input tensor data by applying known machine learning to, for example, a plurality of tensor data and a prognosis indicated by the prognosis data corresponding to the tensor data. The prognosis prediction unit 124 stores the generated regression model in the storage unit 11 as a prognosis prediction model.
[0087] After the cluster generation unit 123 generates a plurality of clusters based on a first pathological image, which is a first stained image to be subjected to pathological diagnosis, the prognosis prediction unit 124 generates the above-mentioned tensor data using the first pathological image and a second pathological image, which is a second stained image of an organism depicted in the first pathological image. The prognosis prediction unit 124 predicts a prognosis by inputting the generated tensor data as a feature into a prognosis prediction model stored in the storage unit 11. This enables the information processing device 1 to predict a prognosis that reflects the feature generated by combining different staining methods.
[0088] Furthermore, the prognosis prediction unit 124 may predict the prognosis by inputting both the feature amount generated from one pathological image described in Fig. 6 and the feature amount generated from a plurality of pathological images stained by a plurality of different staining methods described in Fig. 7 to a prognosis prediction model. Furthermore, the prognosis prediction unit 124 may predict the prognosis based on other feature amounts that can be extracted from clusters generated from pathological images using unsupervised learning.
[0089] The output unit 125 outputs information corresponding to the clusters generated by the cluster generation unit 123. The output unit 125 transmits, for example, the information corresponding to the clusters to the information terminal 2. The information terminal 2 displays the information corresponding to the clusters transmitted by the information processing device 1 on a display unit.
[0090] 8(a) and 8(b) are schematic diagrams of information output by the output unit 125. The output unit 125 outputs, for example, information indicating the position of each of a plurality of cells and the cluster to which the cell belongs on the pathological image as information corresponding to the cluster. In the example of FIG. 8(a), the output unit 125 outputs information indicating the position of the cell and the cluster by labeling the position of the cell on the pathological image with a cluster name. In addition, the output unit 125 may output information indicating the position of the cell and the cluster by, for example, attaching a frame of a different color for each cluster to the position of the cell on the pathological image. This allows the information processing device 1 to notify the user of the relationship between the position of the cell on the pathological image and the cluster in an easy-to-understand manner.
[0091] Furthermore, the output unit 125 outputs information on cells belonging to each of the clusters in association with each of the clusters as information corresponding to the cluster. In the example of FIG. 8(b), the output unit 125 outputs the distribution of the areas of the cells belonging to each of the clusters in the form of a box plot. The output unit 125 may output not only the area of the cells but also the perimeter length, the degree of unevenness, the density, etc. of the cells for each cluster. Furthermore, the output unit 125 may output information on the cells belonging to the clusters in other ways such as a bar graph, a pie chart, etc. This allows the information processing device 1 to notify the user of the characteristics of the cells for each cluster in an easy-to-understand manner.
[0092] Furthermore, the output unit 125 outputs information indicating the prognosis predicted by the prognosis prediction unit 124 based on the cluster as information corresponding to the cluster, for example. The output unit 125 outputs information indicating the prognosis, such as a 5-year survival rate. The output unit 125 may also output information indicating, for example, the name of a drug and the prognosis when the drug is administered to an organism. The output unit 125 may also output the prognoses of multiple drugs in a comparative manner, for example, by displaying the names and prognoses of multiple drugs simultaneously or in sequence on the information terminal 2. This allows the information processing device 1 to notify the user of the prognosis predicted using unsupervised learning from a pathology image specified by the user.
[0093] [Flowchart of information processing method] 9 is a diagram showing a flowchart of an exemplary information processing method executed by information processing device 1 according to this embodiment. A user specifies a pathological image to be a target for pathological diagnosis on information terminal 2. Information terminal 2 transmits the specified pathological image to information processing device 1. In information processing device 1, image acquisition unit 121 acquires the pathological image transmitted by information terminal 2 (S11).
[0094] The image acquiring unit 121 executes at least one of a high resolution process and a color correction process on the pathological image (S12). For example, as the high resolution process, the image acquiring unit 121 acquires a high resolution pathological image output by inputting the pathological image into a high resolution model stored in the storage unit 11. For example, as the color correction process, the image acquiring unit 121 acquires a color-corrected pathological image generated by performing a process of adjusting the color of the pathological image to a reference value.
[0095] The image acquiring unit 121 performs a segmentation process on the pathological image to acquire images of each of the multiple cells appearing in the pathological image as cell images (S13). For example, as the segmentation process, the image acquiring unit 121 inputs the pathological image into an instance segmentation model stored in the storage unit 11 to extract the multiple cells appearing in the pathological image.
[0096] The first clustering unit 123a inputs a plurality of cell images to a clustering model previously generated by unsupervised learning, thereby generating a plurality of small clusters by clustering a plurality of cells (S14). In order to generate the clustering model, the feature extraction unit 122, for example, inputs each of the plurality of cell images to a convolutional neural network, thereby extracting features of cells depicted in the cell images. The first clustering unit 123a generates a plurality of small clusters by clustering a plurality of cells, for example, using the k-means method. The first clustering unit 123a updates parameters of the convolutional neural network for extracting features by applying machine learning to a classification model in which the generated small clusters are used as correct answer labels. The first clustering unit 123a stores the final classification model in the storage unit 11 as a clustering model.
[0097] The second clustering unit 123b performs hierarchical clustering to generate a plurality of clusters by clustering the small clusters (S15). The second clustering unit 123b generates a plurality of clusters by, for example, extracting a pair of two small clusters having the smallest distance between the representative values of the feature amounts, and repeating this hierarchically. The second clustering unit 123b stores in the cluster information storage unit 113 cluster information indicating the small clusters and the cells belonging to each cluster, and the relationship between the small clusters and the clusters.
[0098] The prognosis prediction unit 124 predicts the prognosis of the organism depicted in the pathological image based on information corresponding to the clusters generated by the cluster generation unit 123 (S16). For example, the prognosis prediction unit 124 extracts features to be used for prognosis prediction from the multiple clusters generated by the cluster generation unit 123, and inputs the features to the prognosis prediction model stored in the storage unit 11. The prognosis prediction unit 124 determines the prognosis output by the prognosis prediction model as the prognosis predicted from the pathological image.
[0099] The output unit 125 outputs information corresponding to the clusters generated by the cluster generation unit 123 (S17). The output unit 125 outputs, for example, information corresponding to the clusters, such as information associating the clusters with cells, or information indicating the prognosis predicted by the prognosis prediction unit 124 based on the clusters, to the information terminal 2.
[0100] [Effects of the embodiment] According to the information processing system of this embodiment, the information processing device 1 clusters a plurality of cells captured in a pathological image into a plurality of clusters by unsupervised learning. In supervised learning, cells are classified into a plurality of types defined by humans, and therefore features that do not contribute to the classification into the corresponding types tend to be lost. On the other hand, the information processing device 1 uses unsupervised learning, which does not require correct answer data to be provided by humans, to cluster cells based on the features of the plurality of cells captured in the pathological image, and can classify cells into types that are difficult for humans to recognize, and can generate clusters that reflect features that would be lost in supervised learning.
[0101] Furthermore, the information processing device 1 predicts the prognosis based on clusters generated from pathological images using unsupervised learning, and therefore can predict the prognosis reflecting features that are difficult for humans to recognize.
[0102] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment.
[0103] The processor of the information processing device 1 is a subject of each step (process) included in the information processing method shown in Fig. 9. That is, the processor of the information processing device 1 reads a program for executing the information processing method shown in Fig. 9 from the storage unit 11, and executes the program to execute the information processing method shown in Fig. 9. Some of the steps included in the information processing method shown in Fig. 9 may be omitted, the order of the steps may be changed, and multiple steps may be performed in parallel. [Explanation of symbols]
[0104] 1. Information processing device 11 Storage section 111 Image storage unit 112 Cell information storage unit 113 Cluster information storage unit 12 Control section 121 Image acquisition unit 122 Feature Extraction Unit 123 Cluster Generation Unit 123a First Clustering Section 123b 2nd Clustering Division 124 Prognostic Prediction Department 125 Output section
Claims
1. a feature extraction unit that extracts features of each of a plurality of cells that constitute an organism from a pathology image of the organism; A cluster generation unit that generates a plurality of clusters by clustering the plurality of cells by applying unsupervised learning to the plurality of feature amounts; an output unit that outputs information corresponding to the cluster; An information processing device having the above configuration.
2. The cluster generation unit includes a first clustering unit that generates a plurality of small clusters by clustering a plurality of the cells by applying the unsupervised learning, and a second clustering unit that generates a plurality of the clusters by clustering the plurality of the small clusters by performing hierarchical clustering. The information processing device according to claim 1 .
3. the output unit outputs, as information corresponding to the cluster, information indicating a position of each of the plurality of cells and a cluster to which the cell belongs, on the pathology image.
3. The information processing device according to claim 1 or 2.
4. the output unit outputs information on the cells belonging to each of the plurality of clusters as information corresponding to the clusters, in association with each of the plurality of clusters. The information processing device according to claim 1 .
5. The feature extraction unit extracts the feature by applying machine learning to the pathology image. The information processing device according to claim 1 .
6. The method further includes an image acquisition unit that acquires the high-resolution pathological image output by inputting the pathological image into a high-resolution model that increases the resolution of an input image and outputs the increased resolution of the pathological image, The feature extraction unit extracts the feature from the pathological image having a high resolution. The information processing device according to claim 1 .
7. The high-resolution model is generated by applying machine learning to a low-resolution image of a region including cells of the organism and a high-resolution image of the region.
7. The information processing device according to claim 6.
8. The method further includes an image acquisition unit that acquires a color-corrected pathology image generated by performing a process for adjusting the color of the pathology image to a reference value, The feature extraction unit extracts the feature from the pathology image whose color has been corrected. The information processing device according to claim 1 .
9. The reference value is a statistical value of colors of a plurality of images captured under a plurality of imaging conditions. The information processing device according to claim 8.
10. the reference value is a color of a predetermined cell stained with a predetermined dye, the image acquisition unit adjusts a color of a region in the pathological image corresponding to the predetermined cell to the reference value; The information processing device according to claim 8.
11. the feature extraction unit extracts, from each of a plurality of images to be learned, the feature of a learning cell that is a cell depicted in the image; the cluster generation unit generates a plurality of training clusters by clustering the plurality of training cells based on the feature amounts of the plurality of training cells; the cluster generation unit generates a classification model by supervised learning using the generated learning clusters as pseudo-ground truth labels; The cluster generation unit stores the classification model in a storage unit as a clustering model, The cluster generation unit generates a plurality of clusters by clustering a plurality of the cells appearing in the pathological image, using the clustering model stored in the storage unit. The information processing device according to claim 1 .
12. Further comprising a prognosis prediction unit that predicts a prognosis of the organism based on information corresponding to the cluster. The information processing device according to claim 1 .
13. The prognosis prediction unit predicts the prognosis using a prognosis prediction model generated by applying machine learning to a relationship between the information corresponding to the cluster and the prognosis. The information processing device according to claim 12.
14. The prognosis prediction unit predicts the prognosis when the drug is administered to the organism by using the prognosis prediction model generated using the prognosis of the organism to which the drug is administered. The information processing device according to claim 13.
15. The prognosis prediction unit predicts the prognosis based on the feature amounts of the cells belonging to each of the plurality of clusters. The information processing device according to claim 12 .
16. The prognosis prediction unit predicts the prognosis based on the feature amounts of the cells belonging to each of the plurality of clusters in a predetermined tissue. The information processing device according to claim 15.
17. The prognosis prediction unit predicts the prognosis based on positions of the cells belonging to each of the plurality of clusters. The information processing device according to any one of claims 12 to 16.
18. the prognosis prediction unit predicts the prognosis based on data generated by combining the second pathology image with a mask image that indicates positions of the cells belonging to each of a plurality of clusters generated from the first pathology image; The information processing device according to claim 17.
19. the first pathology image is an image of the biological specimen stained with a first dye, and the second pathology image is an image of the biological specimen stained with a second dye different from the first dye; The information processing device according to claim 18.
20. The processor executes extracting feature amounts of each of a plurality of cells constituting an organism from a pathology image of the organism; generating a plurality of clusters by clustering the plurality of cells by applying unsupervised learning to the plurality of feature amounts; outputting information corresponding to the clusters; An information processing method comprising the steps of:
Citation Information
Patent Citations
System and method for diagnostic support using pathological image of skin tissue
WO2018221625A1
