Systems and methods for image preprocessing

Abstract

A method and apparatus for a device for classifying an image are described. In an exemplary embodiment, the device applies a first convolutional neural network to segment an image into regions of interest and background regions containing information useful for classification. The device also tiles the regions of interest into a set of tiles. For each tile, the device applies a second convolutional neural network to extract a feature vector for that tile, where the features in the feature vector represent local descriptors for the tile. The device then processes the extracted feature vectors of the tile set to classify the image.

Description

[Technical Field]

[0001] [Related Applications] This application claims priority to European Patent Application No. 19305840.1, filed June 25, 2019, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Histopathological image analysis (HIA) is a critical diagnostic component in many medical fields, particularly in oncology, where this type of image analysis currently defines the gold standard metric. Recent research has sought to leverage modern machine learning (ML) advances to assist pathologists in their disease detection efforts. However, most of these techniques require local annotation masks as training data. Obtaining these local annotations is more costly than the actual diagnosis, as pathologists must spend time constructing pixel-by-pixel segmentation maps of diseased tissue at super-resolution. Consequently, HIA datasets containing annotations are very limited in size. Furthermore, these local annotations may not be available when faced with novel HIA problems, such as novel disease subtype classification, prognosis estimation, or drug response prediction. A key question for HIA is whether it is possible to design a learning architecture that achieves accurate classification without using local annotations. Good technology should enable the training of algorithms that assist pathologists during analysis and can also be used to identify previously unknown structures and regions of interest.

[0003] While histopathology provides absolute diagnostic accuracy for tumors and other diseases, it is extremely costly, requiring pathologists to concentrate for hours to reach a single diagnosis. Furthermore, for some diseases, pathologists must identify a few cells among millions of others to make a correct diagnosis, making the task akin to "finding a needle in a haystack." The error rate in histopathology is difficult to determine because it depends on the disease and tissue being examined, as well as pathologists' self-reported misdiagnoses. However, false-negative results in cancer diagnoses not only have devastating consequences for patients but also pose a significant financial risk to pathologists. Tools that can help pathologists focus their attention and efforts on the most suspicious areas can help reduce false-negative results and improve patient outcomes through more accurate diagnoses. Medical researchers have been looking to computer-aided diagnosis for decades, but a lack of computational resources and data has hindered the widespread implementation and use of such tools. Since the advent of automated digital whole slide imaging (WSI) in the 1990s, researchers have sought methods to ease pathologists' workload and improve patient outcomes through image processing algorithms. Many of these procedures focused on cell counting or segmentation for the detection of suspicious regions within WSI, rather than predicting a final diagnosis. Past methods focused on using handcrafted texture or morphological features with unsupervised techniques, such as k-means clustering or other dimensionality reduction methods, prior to classification via k-nearest neighbors or support vector machines.

[0004] Over the past decade, the fruitful development of deep learning has led to an explosion of research into automating image processing tasks. While the application of such advanced machine learning (ML) techniques to imaging tasks has been successful in many consumer applications, the adoption of such methods in the field of medical imaging has been more gradual. However, these techniques show great promise in the field of HIA. Specifically, in digital pathology using WSI, highly trained and skilled pathologists make diagnoses by reviewing digitally captured microscopic images of prepared and stained tissue samples. Digital WSI is a massive dataset consisting of images captured at multiple zoom levels. At the highest magnification, WSI can have a digital resolution exceeding 100,000 pixels in both dimensions. However, due to the significant challenges of obtaining local annotations, datasets often contain only WSI-level diagnostic labels, falling into the category of weakly supervised learning. Summary of the Invention

[0005] A method and apparatus for a device for classifying an image are described. In an exemplary embodiment, the device applies a first convolutional neural network to segment an image into a region of interest containing information useful for classification and a background region. The device also tiles the region of interest into a set of tiles. For each tile, the device applies a second convolutional neural network to extract a feature vector for that tile. The device then processes the extracted feature vectors of the tile set to classify the image.

[0006] In a further embodiment, a device extracts regions of interest from an image that correlate with a classification of the image. In this embodiment, the device tiles the region of interest of the image into a first set of tiles. For each tile, the device extracts a feature vector for that tile by applying a convolutional neural network, where features in the feature vector represent local descriptors for the tile, and calculates a score for the tile from the extracted feature vector, representing the tile's contribution to the classification of the image. For each tile, the device also selects at least one other tile having a score that verifies a given characteristic, and extracts a comparison target region in a second set of tiles that has a score between the high score and the low score of the tile in the first set of tiles and that is visually close to the at least one other tile according to a distance metric.

[0007] In yet another embodiment, a device generates a classification model. In this embodiment, the device receives a training set of images, each having an associated known classification. For each training image in the training set, the device applies a first convolutional neural network to extract a plurality of feature vectors for the training image, each feature of the plurality of feature vectors representing a local descriptor for that image. The device then trains the classification model using at least the extracted feature vectors and the associated known classification.

[0008] Other methods and apparatus are also described.

[0009] The present invention is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like reference symbols indicate similar elements and in which: [Brief explanation of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram of an embodiment of a system for classifying images by preprocessing the images using a classification model. [Figure 2] FIG. 1 is a block diagram of an embodiment of a system for training a classification model using a training image set. [Figure 3] FIG. 1 is a flow diagram of one embodiment of a process for preprocessing and classifying images. [Figure 4] FIG. 1 is a flow diagram of one embodiment of a process for sorting tiled images into tile sets. [Figure 5] FIG. 1 is a diagram of a classification system for classifying histopathological images. [Figure 6] FIG. 1 is a flow diagram of one embodiment of a process for training and validating a classification model. [Figure 7] FIG. 1 is a flow diagram of an embodiment of a process for validating a classification model. [Figure 8] FIG. 1 is a flow diagram of one embodiment of a process for extracting comparison regions from an image. [Figure 9] FIG. 1 is a block diagram of an embodiment of a system for classifying images using a classification model and local annotations. [Figure 10] FIG. 1 is a flow diagram of an embodiment of a process for classifying images using a classification model and local annotations. [Figure 11] FIG. 1 illustrates an example of a typical computer system that can be used with embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0011] A method and apparatus for a device for classifying images is described. In the following description, numerous specific details are set forth to thoroughly describe embodiments of the present invention. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In some instances, well-known components, structures and techniques have not been shown in detail in order to avoid obscuring an understanding of this description.

[0012] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0013] In the following description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. "Coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. "Connected" is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0014] The processes illustrated in the following figures are performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, etc.), software (such as may be executed on a general-purpose computer system or dedicated machine), or a combination of both. While the processes are described below in terms of several sequential operations, it should be understood that some of the operations described may be performed in a different order. Furthermore, some operations may be performed in parallel rather than sequentially.

[0015] The terms "server," "client," and "device" are intended to generally refer to data processing systems, rather than to specifically refer to a particular form factor of a server, client, and / or device.

[0016] As described above, histopathological image analysis can rely on local annotations by experts before the image can be analyzed, which is time-consuming and labor-intensive. To solve this problem, images can be analyzed using an image processing pipeline without using local annotations. In one embodiment, local annotations are metadata (e.g., text, markings, numbers, and / or other types of metadata) that apply to a portion of an image but not to the entire image. For example, in one embodiment, local annotations can be markings of regions of interest within an image, such as a tissue image. Exemplary local annotations include markings that outline or otherwise identify portions of an image, such as a tumor region of an image, a stromal region of an image, identification of a cell type within an image, or identification of a biological structure composed of multiple cells within an image. In contrast, "global annotations," as referred to herein, refer to metadata that applies to the entire image. Exemplary global annotations include labels that identify the entire image, data regarding how the image was acquired, labels that identify characteristics of the subject from whom the image was acquired, such as labels indicating the age, sex, diagnosis, etc. of the subject from whom the image was acquired, and / or other data that applies to the entire image. In some embodiments, the global annotations may indicate the presence, type, grade, or stage of a tumor known or understood to be present in the subject from whom the image was obtained. In other embodiments, the global annotations may indicate known characteristics of the subject from whom the image was obtained, such as survival time (e.g., survival time after sample acquisition represented in the image) or response to a given treatment. In some embodiments described herein, an image containing global annotations may be used if no local annotations are present.

[0017] The pipeline is based on first subdividing a large image (e.g., WSI) into smaller images, such as 224x224 pixel images, detecting regions of interest in these images, and performing classification using Otsu's method. Therefore, the classification operates on smaller images, which are computationally much less expensive. These smaller images are fed into a ResNet convolutional neural network to extract a feature vector from each smaller image, which is a local descriptor of this smaller image. From the extracted feature vectors, a score is calculated for each smaller image as a local tile-level (instance) descriptor. The top- and bottom-level instances are used as inputs to a multi-layer perceptron (MLP) to perform classification on them. While this solution produces good results, its overall efficiency leaves room for improvement.

[0018] A method and apparatus for a device for classifying images is described that improves upon the above pipeline. In one embodiment, the device classifies images using one or more neural network models to determine a label for the images. In this embodiment, the images may be large images for which it is computationally impractical to process the entire image using a neural network model alone. Among other things, the device reduces the amount of computational resources (e.g., time and / or memory requirements) required to perform image classification tasks on these large images. This reduction in resources further improves the performance of the device in performing image classification tasks. Additionally, the device is capable of classifying entire slide images, even if the images are too large to fit into the memory of graphics processing units typically used to train machine learning models.

[0019] In a further embodiment, the device reduces the dimensionality of the data, thus resulting in better generalization error and greater efficiency in terms of model accuracy.

[0020] According to one embodiment, a device classifies at least one input image by applying a first convolutional neural network to segment the image between at least one region of interest containing information useful for classification and at least one background region not containing information useful for classification. The device then tiles the at least one region of interest of the image into a set of tiles. The device also applies a second convolutional neural network to extract a feature vector for each tile, where these features are local descriptors for the tile. The device then processes the extracted tile feature vectors to classify the image. In one embodiment, the device segments the input image to process a smaller number of tiles, thereby avoiding processing the entire image.

[0021] In one embodiment, the first convolutional network is a semantic segmentation neural network that classifies pixels of an input image into one of two categories: (a) regions of interest and (b) background regions. Furthermore, tiling can be performed by applying a fixed tiling grid to the image so that the tiles have a predetermined size. At least one zoom level can be applied to the obtained tiles. For example, in one embodiment, multiple zoom levels can be applied to the tiles to combine tiles from different zoom levels. Furthermore, the device can arbitrarily randomly sample tiles and / or pad the tile set with blank tiles so that the tile set contains a fixed number of tiles.

[0022] In further embodiments, the second convolutional neural network may be a residual neural network, such as a ResNet50 residual neural network or a ResNet101 residual neural network with the last layer removed using the previous layer as the output, or a VGG neural network. This second convolutional neural network may be a pre-trained neural network, which allows the use of state-of-the-art advanced neural networks without the need for a large image database and computational resources to train the neural network.

[0023] In one embodiment, the device can calculate at least one score for a tile from the extracted feature vector, with each tile score representing the tile's contribution to the classification of the image. The device can use these tile scores to sort the set of tile scores, select a subset of the tile scores based on their value and / or rank within the sorted set, and apply a classifier to the retained tile scores to classify the image. The device can further apply this classification to multiple input images, in which case the device can aggregate groups of corresponding tiles from different input images.

[0024] In another embodiment, the device may also aggregate clusters of adjacent tiles. In this embodiment, aggregating the clusters of tiles may include concatenating tiles of the cluster, selecting a single tile from the cluster according to a given criterion, using the cluster as a multidimensional object, or aggregating values ​​by an average or max-pooling operation, etc. The device may also apply an autoencoder to the extracted feature vector to reduce the dimensionality of the features. In one embodiment, the image may be a histopathology slide, the region of interest may be a tissue region, and the classification of the image may be a diagnostic classification.

[0025] In another embodiment, if local annotations, such as the presence of a tumor in a region of a slide, are available, these annotations can be taken into account using a hybrid technique. To do this, the device can train a machine learning model for two simultaneous tasks: (1) local prediction of the presence of a macroscopic feature on each tile (e.g., the presence of a tumor or other type of macroscopic feature), and prediction of a global label set. On the one hand, the device (or multiple devices) can process the 128 feature set using a complex architecture that includes the classification system described above. On the other hand, the device applies a convolutional neural network to convert the features of the N tiles into an N*128 feature vector. Based on this vector, the device trains the convolutional neural network to predict the presence or absence of a tumor (or other macroscopic feature) for each tile. The device can take both the output of the prediction and the N*128 feature vector and apply a weighted pooling operation to the concatenation of these two vectors to obtain the 128 feature vector for the input image. The device concatenates the output of the classification model with the obtained 128 features and attempts to predict a global label set for the image (e.g., survival rate, tumor size, necrosis, and / or other types of predictions) based on this vector. The model loss includes both global and local predictions. In this embodiment, adding information derived from local annotations to the computational flow can improve overall model performance.

[0026] Figure 1 is a block diagram of one embodiment of a system for classifying images by preprocessing and classifying images using a preprocessing model and a classification model. In Figure 1, system 100 includes a client 118 coupled to a preprocessing device 102 coupled to a classification device 108. In one embodiment, preprocessing device 102 and classification device 108 receive input images and output an image classification using preprocessing model(s) 106 and classifying model(s) 112. In this embodiment, the image classification can be one of a different label, a number selected from a range of numbers, and / or other types of classification.

[0027] In one embodiment, each of the client 118, the preprocessing device 102, and the classification device 108 can be a personal computer, laptop, server, mobile device (e.g., smartphone, laptop, personal digital assistant, music playback device, gaming device, etc.), and / or any device capable of processing images. In one embodiment, each of the client 118, the preprocessing device 102, and / or the classification device 108 can individually be a physical or virtual device. In one embodiment, the client 118 can be a device used by a user, such as a medical professional who wishes to determine a patient's prognosis from the images, to determine the classification of the images. In one embodiment, the images can be large images, on the order of gigabytes or more. For example, in one embodiment, the images can be digital images of histopathology slides (e.g., WSIs), which can be on the order of several gigabytes or more. In one embodiment, the image is a histopathology slide, but in other embodiments, the image may be another type of image (e.g., a satellite image, an astronomical image, a wind turbine image, another type of medical image, and / or any other type of large image).

[0028] In a further embodiment, the preprocessing device 102 segments the image into regions of interest and background regions. In this embodiment, extracting regions of interest from the input image can reduce the amount of computation required to classify the input image. For example, in one embodiment, because histopathology slides (or other types of images) can contain one or more empty regions of the image that contain little or no tissue, it is useful to employ so-called "tissue detection" or "foreign body detection" methods to assess whether regions of the slide contain any tissue. More generally, when the goal is to classify large images, this involves identifying regions of interest within the image and distinguishing them from background regions. These regions of interest are regions of the image that contain valuable information for the classification process. Background regions are regions of the image that contain little or no valuable information and can be considered noise for the task at hand. Image segmentation is further described in FIG. 3 below.

[0029] In one embodiment, the preprocessing device 102 can use the segmented images to tile the image. In this embodiment, tiling is the process of dividing the image into a group of image subsets. The image subsets can be of fixed size or variable size. Tiling can also be based on regions of interest generated from the image segmentation described above. Tiling is further described in FIG. 3 below.

[0030] In a further embodiment, the preprocessing device 102 determines image features using a convolutional neural network. In this embodiment, the preprocessing device 102 calculates a feature vector for each tile in the tile set. These feature vectors can be used to classify the image by a classifier. Feature determination is further described in FIG. 3 below. In one embodiment, the preprocessing device 102 includes a preprocessing module 104 that uses one or more preprocessing models 106 to perform the functions of the preprocessing device 102.

[0031] In one embodiment, the preprocessing device 102 can send the tile set and feature vector to the classification device 108, which uses the tile set, feature vector, and classification model 112 to classify the input image 114. In this embodiment, the classification device 108 calculates a tile score for each tile in the tile set, sorts the tile set, and selects a subset of tiles from the tile set. The classification device 112 uses the subset of tiles to classify an image from the subset of tiles and outputs an image classification 114. In one embodiment, the image classification is sent to the client 118. In another embodiment, the image classification 114 can be stored in a repository (possibly along with the input image), and a client can retrieve the input image 116 and / or the image classification 114. Classifying images using tiles is further described in FIG. 3 below. In one embodiment, the functions of either the preprocessing device 102 or the classification device 108 can be performed by other devices, or one device can perform the functions of the preprocessing device 102 and the classification device 108. In one embodiment, the classification device 108 includes a classification module 110 that uses one or more classification models 112 to perform the functions of the classification device 108 .

[0032] 1, a client preprocesses and classifies images using preprocessing device 102 and / or classification device 108. In one embodiment, a set of input images with known characteristics may be used to train each of preprocessing model 106 and / or classification model 112. For example, in one embodiment, classification model 112 may be trained using the set of input images and a known set of labels for each input image. Additionally, the set of input images may also include information that can be used to train an image segmentation model.

[0033] 2 is a block diagram of one embodiment of a system 200 for training a classification model using a training image set 216. In FIG. 2, the system 200 includes a preprocessing device 202 coupled to a training device 206. In one embodiment, the preprocessing device 202 receives the image training set 216 and generates a feature vector set for each image in the image training set 216. A training device 208 can receive this feature vector set and train a classification model 214 using input labels 212 associated with the image training set 216. This training is further described in FIG. 6 below.

[0034] As described above, the preprocessing model 102 and the classification device 108 can preprocess and classify images. FIG. 3 is a flow diagram of one embodiment of a process 300 for preprocessing and classifying images. In one embodiment, either the preprocessing device 102 and / or the classification device perform the functions of process 300. In FIG. 3, process 300 begins at block 302 by receiving an image, one or more models, and other inputs. In one embodiment, the input image may be a histopathology image, a satellite image, an astronomical image, a wind turbine image, another type of medical image, and / or any other type of large image. In this embodiment, the model is one or more models that can be used to preprocess and / or classify the image. For example, in one embodiment, the model may include an image segmentation model used to segment the image and a classification model used to classify the image. Additionally, the other inputs may include weights and other input parameters used by process 300.

[0035] Process 300 segments the input image at block 304. In one embodiment, process 300 segments the input image into regions of interest and background regions. In this embodiment, process 300 can extract regions of interest from the input image by segmenting the image. In this embodiment, extracting regions of interest from the input image can reduce the amount of computation required to classify the input image. For example, in one embodiment, a histopathology slide (or other type of image) can contain one or more empty regions of the image that contain little or no tissue; therefore, it is useful to employ so-called "tissue detection" or "foreign body detection" methods to assess whether a region of the slide contains any tissue. More generally, when the goal is to classify large images, this involves identifying regions of interest within the image and distinguishing them from background regions. These regions of interest are regions of the image that contain valuable information for the classification process, while background regions are regions of the image that contain little or no valuable information and can be considered noise for the task at hand. A variety of different types of image segmentation schemes can be used to accomplish this task. For example, in one embodiment, an image can be segmented using Otsu's method, which is a simple thresholding method based on the intensity histogram of the image. In this embodiment, Otsu's method has shown fairly good results when the image contains two classes of pixels that follow a bimodal distribution, such as foreground and background pixels, or more specifically, tissue and non-tissue. However, it has been found that this method does not perform well for complex images when the histogram of intensity levels cannot be assumed to have a bimodal distribution. Therefore, a more robust method needs to be used to improve the overall efficiency of the method.

[0036] In another embodiment, to increase the robustness of image segmentation and enable it to handle complex images, images can be segmented using a semantic segmentation neural network, such as the U-NET semantic segmentation neural network, SegNet, DeepLab, or another type of semantic segmentation neural network. In this embodiment, a semantic segmentation neural network can be used that does not rely on a particular distribution of intensity histograms. Furthermore, such a neural network enables image segmentation that takes into account multi-channel images, such as RGB images. Thus, segmentation can also exploit image semantics rather than relying solely on the histogram of pixel intensities. In one embodiment, the semantic segmentation neural network is trained to segment tissue from background, for example, to distinguish stained or unstained tissue from the background. In another embodiment, the semantic segmentation neural network can be trained for different types of images to distinguish the foreground from the background of those types of images.

[0037] In a further embodiment, the original image can be downsampled to reduce the computational cost of the image segmentation step. As explained further below, in one embodiment, part of the image analysis is performed at the tile level (which are subsections of the image), and using semantic segmentation on a downsampled version of the image does not degrade the quality of the segmentation. This allows the use of downsampled images without actually degrading the quality of the segmentation. As a result, to obtain the segmentation mask for the original full-resolution image, the segmentation mask generated by the neural network can simply be upscaled.

[0038] In another embodiment, another advantage of using the U-NET segmentation neural network is that this network type was developed for biomedical image segmentation and therefore fits the typical constraints of biomedical data: small datasets with very high dimensionality. Indeed, the U-NET segmentation neural network is a model with few parameters to train and can be trained with few training examples. Furthermore, in another embodiment, using data augmentation techniques on the training data, this architecture can achieve very good results, allowing for more training examples to be obtained from the same training set.

[0039] Process 300 tiles the images into an image tile set in block 306. In one embodiment, process 300 uses tiling to enhance image preprocessing capabilities. For example, in one embodiment, using a tiling method is useful in histopathological analysis due to the large size of whole slide images. More broadly, when dealing with specialized images such as histopathological slide images or satellite images, or other types of large images, the resolution of image sensors used in these fields can grow as quickly as the capacity of the random access memory associated with the sensors. This large image size makes it difficult to store a group of images, or even a single image, in a computer's random access memory. This difficulty is exacerbated when attempting to store these large images in the dedicated memory of a graphics processing unit (GPU). In this situation, processing an image slide or any other similarly sized image as a whole becomes computationally intractable.

[0040] In one embodiment, process 300 addresses this challenge by tiling the image (or region of interest) by dividing the original image (or region of interest) into smaller, more manageable images called tiles. In one embodiment, the tiling operation is performed by applying a fixed grid to the entire slide image and using a segmentation mask generated by the segmentation method to select tiles that contain tissue or any other type of region of interest for the subsequent classification process. Furthermore, further selection methods, such as random subsampling, which retains only a fixed number of slides, can be used to reduce the number of tiles to be processed.

[0041] For example, in one embodiment, process 300 divides the region of interest into tiles of a fixed size (e.g., each tile having a size of 224x224 pixels). Alternatively, the tile size can be smaller or larger. In this example, the number of tiles generated depends on the size of the detected foreign object and can vary from several hundred tiles to 50,000 or more tiles. In one embodiment, the number of tiles is limited to a fixed number (e.g., 10,000 tiles), which can be set based at least on computation time and memory requirements.

[0042] Process 300 can also apply zoom levels to tiles generated by the grid, where the zooming is adapted so that all tiles have the same dimensions. The zoom selection is a trade-off between detail and context, with higher zoom levels allowing process 300 to extract more detailed information within the tile, while lower zoom levels preserve the context of the selected area. Process 300 extracts tiles at multiple zoom levels to benefit from multiple zoom levels, allowing the model to extract more information from a given slide. These multiple zoom levels can be combined by concatenating multiple tiles at different zoom levels with the same center location.

[0043] Process 300 extracts feature vectors from the tile set at block 308. In one embodiment, feature extraction aims to construct derivatives that are intended to be informative and non-redundant from tiles formed from the input image to facilitate subsequent learning and generalization steps. In one embodiment, the extracted features are in the form of numerical vectors, also called local descriptors.

[0044] In one embodiment, process 300 can use any feature extraction neural network, such as a ResNet-based architecture (e.g., ResNet-50, ResNet-101, ResNetX), a Visual Geometry Group (VGG) neural network, an Inception neural network, an autoencoder for unsupervised feature extraction, a custom-made neural network designed specifically for the task, or a non-neural network feature extractor such as SIFT or CellProfiler. Furthermore, the feature extraction neural network used can be pre-trained because it has been trained on large datasets and therefore has optimal generalization accuracy.

[0045] In one embodiment, process 300 uses a ResNet-50 neural network because it can provide features well suited for image analysis without requiring excessive computational resources. For example, in one embodiment, ResNet-50 can be used for histopathological image analysis. In this example, because simple convolutional neural network architectures have the worst accuracy when the number of layers is too large, the ResNet-50 neural network relies on residual blocks, which allow the neural network to be deeper and further improve accuracy. In one embodiment, because the ImageNet dataset is a truly general-purpose image dataset, the weights of the ResNet-50 neural network can be weights used for feature extraction from pre-training on the ImageNet dataset. In one embodiment, using a neural network pre-trained on a large, independent image dataset can yield good features regardless of the type of image, even if the input images are specialized, such as histopathological images (or other types of images). In this embodiment, process 300 extracts 2,048 features per tile using a ResNet-50 convolutional neural network. For example, if 10,000 tiles are extracted, process 300 generates a 2,048 x 10,000 matrix. Furthermore, if process 300 is run with many images as input, it generates a tensor with dimensions of number of images x number of features / tiles x number of tiles.

[0046] In one embodiment, to extract features for a given slide, process 300 processes each selected tile through a ResNet-50 neural network, which outputs a feature vector for that tile. In this embodiment, the feature vector may be a vector of dimension 2048. Process 300 may also apply an autoencoder to the feature vector to further reduce the dimensionality, for example, reducing the dimensionality of the feature vector to 256 (or another dimension). In one embodiment, an autoencoder can be used when the machine learning model is susceptible to overfitting. For example, in one embodiment, process 300 may reduce a feature vector of length 2048 to a feature vector of length 512. In this example, process 300 could optionally use an autoencoder with a single hidden layer architecture (of 512 neurons). This prevents the model from overfitting by discovering multiple anomalous features in the training dataset, while also reducing computational time and memory requirements. In one embodiment, the classification model is trained on a small subset of image tiles, for example, 200 randomly selected tiles from each slide (out of a total of 411,400 tiles).

[0047] Process 300 can optionally perform a zero-padding operation on the feature vectors to derive the minimum number of features. In this embodiment, process 300 can perform zero-padding to add feature vectors to the feature vector set for the image if the number of feature vectors is below the minimum number of feature vectors. In this embodiment, each zero-padded feature vector has a null value.

[0048] Process 300 scores each tile at block 310. In one embodiment, process 300 converts each feature vector into one or more scores using any of a number of connected neural networks. In one embodiment, process 300 can convert the feature vector into a single score using a fully connected neural network, or into multiple scores representing different characteristics of the tile using one fully connected neural network that outputs different scores or multiple fully connected neural networks that each output a different score. These scores associated with a tile are sorted, and a subset of tiles is selected for image classification. In one embodiment, this subset of tiles can be the tiles with the top R scores and bottom R scores, the tiles with the top R scores, the tiles with the bottom R scores, and / or a weighted combination of any of these scores. Finally, these scores are concatenated into an image score vector, which can be considered an input for image classification.

[0049] For example, in one embodiment, process 300 can use a convolutional 1D layer to create a score for each tile. In the example described above with a feature vector of length 2,048, the convolutional layer performs a weighted sum between all 2,048 features of the tile to obtain this score, and the weights of this sum are learned by the model. Furthermore, because process 300 uses a convolutional 1D layer with a bias of zero, zero-padded tiles have a score of zero and are therefore references to tiles that provide no information, respectively. Process 300 then selects the highest and lowest R scores and uses these as inputs for classification, as described below. This architecture ensures which tiles are used to make predictions and, therefore, how process 300 predicts outcomes. Tile scoring is further described in FIG. 4 below.

[0050] At block 312, process 300 uses the tile scores to classify the image and predict one or more global labels for the image. In one embodiment, process 300 uses the image score vector as input to a dense multilayer neural network that provides the desired classification. This classification can be any task of associating a label with data provided as input to the classifier. In one embodiment, using a classifier trained for histopathology slide image inputs allows the classifier to label histopathology slides or any other type of image provided as input without having to process the entire image, which can be computationally prohibitive, because the input data is derived through the entire pipeline. For example, in one embodiment, these labels can be any type of label, such as a binary value representing the prognosis of a given condition, a score such as a survival prediction or a treatment response prediction, a numerical label representing a probability or a prediction of a physical quantity, and / or a scalar label as described above, or a vector, matrix, or tensor of such labels representing structured information. For example, in one embodiment, process 300 outputs a continuous risk score as a classification of a histopathological input image of mesothelioma, which can be used to determine the subject's estimated survival time. In this example, different continuous output scores for different subjects can be plotted against similar risk scores derived from multiple mesothelioma subjects with known survival times to determine the individual subject's estimated survival time. In one embodiment, process 300 uses a multilayer perceptron (MLP) with two fully connected layers of 200 neurons and 100 neurons with sigmoid activation. In this embodiment, the MLP is used as the core of a prediction algorithm that converts tile scores into label(s). In one embodiment, process 300 predicts a single label (e.g., a risk score) for an image, while in another embodiment, process 300 can predict multiple global labels for an image. In one embodiment, process 300 can perform multi-task learning to predict multiple global labels.For example, in one embodiment, a classification model (e.g., MLP and / or other models described elsewhere) can be trained to predict multiple labels simultaneously (e.g., survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions) in a multi-task learning setting.

[0051] As can be seen in FIG. 3 , images can be classified based on at least a set of tile scores derived from image tile feature vectors generated from a neural network. FIG. 4 is a flow diagram of one embodiment of a process 400 for scoring images tiled into tile sets. In one embodiment, a process such as block 310 of process 300 performs process 400 for scoring images. In FIG. 4 , process 400 begins by receiving a tile set at block 402. In one embodiment, this tile set is the tile set generated at block 306 of FIG. 3 . The tile set also includes feature vectors for each tile in the tile set. In block 404, process 400 calculates a tile score for each tile using each tile's associated feature vector. For example, in one embodiment, process 400 can use a convolutional 1D layer to create a score for each tile. In the example described above with a feature vector length of 2,048, this convolutional layer performs a weighted sum between all 2,048 features of the tile to obtain this score, and the weights of this sum are learned by the model. Furthermore, since convolutional 1D layers are unbiased, zero-padding tiles have a score of zero and are therefore tile references that carry no information whatsoever.

[0052] Process 400 sorts the tile set in block 406. In one embodiment, process 400 sorts the tile set to determine the top R and / or bottom R scores in block 408 below. Process 400 selects a subset of tiles in block 408, which is used in a subsequent classification step. In one embodiment, this subset of tiles is sorted into the top R scores.top Highest scores and lowest R bottom Lowest score, top R top Highest score, lowest R bottom The tiles may be tiles containing the lowest scores, and / or a weighted combination of any of these scores. top and / or R bottom The range of values ​​of R can be the same or different. top and / or R bottom The range can be a static numeric range (e.g., 10, 20, 100, or some other numeric value), a fitted range, a percentage, a label (e.g., small, large, or some other label), set via a user interface component (a slider, a user input, and / or another type of user interface component), and / or some other value. In one embodiment, process 400 also concatenates these scores into an image score vector, which can be considered as input for image classification.

[0053] In one embodiment, when reviewing histopathology whole slide images (or slides), a patient (or subject) can be associated with multiple slides taken with different stains, in different locations on the same specimen, from multiple organs, or at different time points. In this embodiment, slides from a single patient can be aggregated in multiple ways. In one embodiment, process 300 can concatenate slides to form larger slides that are processed in the same or similar ways as regular slides (segmentation, tiling, feature extraction, and classification).

[0054] In a further embodiment, process 300 can treat multiple slides as three-dimensional images. This is particularly useful when the slides are multiple contiguous slices of the same tissue. In this case, a 3D convolutional neural network (CNN) can be used for the feature extraction step to obtain the maximum information from a given set of slides. Furthermore, by applying the segmentation, tiling, and feature extraction steps to multiple slides and retaining only a certain number of features to match the input dimensions used by the classifier, the selected features can be, for example, the N largest and M smallest features for each tile. This approach is particularly suitable when the slides being assembled are from the same batch of slides but using different stains.

[0055] In another embodiment, process 300 can aggregate feature vectors by clustering nearby tiles according to a distance metric calculated on the image or feature and calculating the mean, maximum, or weighted sum of these associated feature vectors. This allows process 300 to significantly reduce the dimensionality of the problem by starting with a large number of tiles (e.g., 10,000 tiles) and aggregating them into a small number of clusters (e.g., 200), which not only reduces computation time but also reduces overfitting, resulting in a better generalization error for the model. Specifically, a good distance metric for selecting tiles to aggregate is, for example, a 1 mm distance on a slide. 2 is the Euclidean distance across the entire slide image aggregating all tiles contained in the patch.

[0056] In another embodiment, a slide may not contain enough useful tissue to extract tiles to which the feature extraction step should be applied, and thus provide features to the classifier. In this case, the classifier input is zero-padded, i.e., for each missing tile, a feature consisting entirely of zeros is added to the actual features computed by the feature extractor.

[0057] As discussed above, one example of image classification is for classifying histopathological images. In this example, the calculated label is a patient prediction based on the input histopathological slide image. FIG. 5 is an illustration of a classification system 500 for histopathological image classification. In FIG. 5, the classification system 500 includes a tiling function 502, a tile sampling function 504, a feature extraction function 506, a tile scoring function 508, a tile sorting function 510, and a classification (here, survival prediction) function 512. In one embodiment, the tiling function 502 receives a whole slide image 514 and outputs a preprocessed tile set 516. In this embodiment, the number of tiles for an image can be on the order of 10,000 tiles. In further embodiments, an image can have more or fewer tiles. In one embodiment, to reduce computational complexity, the classification system 500 samples the tiles 504 to reduce the number of tiles 518 used in the neural network calculations. In one embodiment, the classification system 500 randomly samples the tiles 504 or with some other type of sampling mechanism. For example, in one embodiment, the classification system 500 randomly samples the tiles 504 to reduce the number of tiles from on the order of 10,000 tiles to on the order of a few thousand tiles (e.g., 3000 tiles).

[0058] In one embodiment, the classification system 500 performs a feature extraction function 506 on the sampled tiles. In this embodiment, the classification system 500 extracts features on the sampled tiles using a convolutional neural network (e.g., using ResNet-50 or another type of feature extractor described above), which results in a matrix of local descriptors 520. Additionally, in one embodiment, the classification system 500 scores 522 the tiles based on at least the tile feature vectors 508. In this embodiment, the classification system 500 generates a score for each image tile using a 1D convolutional neural network. The classification system 500 further sorts the tile scores 510, and the sorted tiles are used by an MLP regressor to generate a predicted survival probability 512. In one embodiment, each of these functions performed by the classification system 500 is the same as or similar to the functions described above in FIG. 3.

[0059] In one embodiment, an investigator can use the selection of tiles to determine correlations between patterns in the image and associated global labels associated with the image or associated with entities associated with the image (e.g., a patient, a subject in a satellite image or a wind turbine image, etc.). In this embodiment, for a particular type of image, an investigator can review selected tiles from the image with known (or unknown) labels to determine correlations. For example, in one embodiment, an investigator can review selected tiles from histopathological images processed by process 300 and compare the selected tiles with corresponding image labels or other corresponding data indicative of survival time to discover trends or correlations between survival time and the histopathological image features indicated by the selected tiles.

[0060] As mentioned above, in one embodiment, the method is shown for use in histopathological image analysis, which can assist a pathologist in their disease detection task. However, processes 300 and / or 400 can also be adapted for any type of image processing problem where running a classifier on the entire image is computationally intractable. One such use case is for classifying high-resolution images, for example, from satellite imagery. In this example, a semantic segmentation neural network and classifier (e.g., an MLP regressor and / or other model(s)) are trained to classify high-resolution satellite imagery.

[0061] As described above, process 300 uses training model(s) to determine label(s) for each input image. In one embodiment, process 300 uses training models for image segmentation, convolutional neural network scoring, and classification. In this embodiment, each of these training models is trained for a particular category of images (e.g., histopathology image slides, satellite images, and / or other types of image categories). In a further embodiment, some of the models used in FIG. 3 are trained end-to-end, and these models are trained together. In this embodiment, some of the models used in FIG. 3, such as the MLP used to classify images and the one-dimensional convolutional neural network used to score tiles, can be trained together. Other models used in FIG. 3 can also be trained separately on different training sets (e.g., ResNet models, U-NET, and / or other types of models). FIG. 6 is a flow diagram of one embodiment of a process 600 for training and validating classification models. In one embodiment, the classification model may include one or more individual models (e.g., MLP and / or one-dimensional convolutional neural networks) used in the classification process described in Figure 3. In Figure 6, process 600 begins by receiving a training image set at block 602. In one embodiment, the training image set is used to train a classification model for a particular category of images (and / or other models used in process 300). For example, in one embodiment, the image set may be histopathology slide images labeled to be associated with a particular type of disease, such as cancer. Process 600 receives a model at block 604. In one embodiment, the model is a classification model, such as the MLP model and one or more other models described above.

[0062] Process 600 executes a processing loop (blocks 606-610) to generate a set of feature vectors for a set of training images. Process 600 extracts feature vectors for images in the training set at block 608. In one embodiment, process 600 extracts the feature vectors as described above in FIG. 3. For example, in one embodiment, process 600 uses a ResNet-50 convolutional neural network, as described above in FIG. 3, to determine the feature vector for each tile of the tiled segmented images. In one embodiment, process 600 generates a set of feature vectors for the training images. Process 600 can also perform data augmentation during training of the method to improve generalization error. This data augmentation can be performed by applying various transformations to the tiles, such as rotation, translation, cropping, adding noise to the image, changing the intensity of certain colors, or changing the contrast. The process loop ends at 610.

[0063] At block 612, process 600 trains model(s) using the extracted feature vectors of the training image set and the input labels of the training image set. In one embodiment, process 600 jointly trains a one-dimensional convolutional neural network and an MLP classification model using the labels of the training image set. In this embodiment, process 600 iteratively trains model(s) by computing a score set for the training images, predicting labels, determining the difference between the predicted labels and the input labels, and optimizing the model(s) based on the difference (e.g., calculating new weights for the model(s)) until the difference is within a threshold. In one embodiment, process 600 trains a model to predict a single label (e.g., a risk score) for an image, but in another embodiment, process 600 can also be trained to predict multiple global labels for an image. In one embodiment, process 600 can be trained to predict multiple global labels by performing multi-task learning. For example, in one embodiment, a classification model (e.g., MLP and / or other model(s) described elsewhere) can be trained to predict multiple labels (e.g., survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions) simultaneously in a multi-task learning setting. Process 600 validates the classification model at block 614 to determine the adequacy of the training. Validation is further described in FIG. 7.

[0064] In FIG. 6 , process 600 trained a classification model used to classify images. How good the classification model is can be checked by validating the classification model using a training image set as input and calculating one or more labels. FIG. 7 is a flow diagram of one embodiment of a process for validating a classification model. In FIG. 7 , process 700 begins by receiving a validation image set at block 702. In one embodiment, the validation image set is the same as the training set. In another embodiment, the validation set can be different from the training image set. For example, in an embodiment, a labeled image set of a particular type of image (e.g., histopathology of a particular disease) can have some images selected for use in training the model and other images from this set that can be used to validate the trained model. Process 700 receives a model used to classify the validation image set at block 704. In one embodiment, the model is a classification model, such as an MLP model and / or other model(s) described elsewhere.

[0065] Process 700 executes a processing loop (blocks 706-714) to generate a set of image labels for the validation image set. Process 700 extracts feature vectors for images in the validation image set at block 708. In one embodiment, process 700 extracts the feature vectors as described above in FIG. 3. For example, in one embodiment, process 700 determines a feature vector for each tile of the tiled segmented images using a ResNet-50 convolutional neural network as described above in FIG. 3. In one embodiment, the process generates a set of feature vectors for the validation image set. Process 700 generates a score set for the validation image set at block 710 using the feature vector set. In one embodiment, process 700 generates a score set for the images using a convolutional 1D layer as described above in FIG. 4 to create a score for each tile. In this embodiment, process 700 selects a subset of tiles of the images, and this subset of tiles is used to generate the tile scores. Process 700 then classifies each image using the trained classification model at block 712. In one embodiment, process 700 classifies images in the validation image set using a multi-layer perceptron (MLP) with two fully connected layers of 200 neurons and 100 neurons with sigmoid activation. The process loop ends at 716.

[0066] Process 700 can use the classification of the validation image set to compare the generated classification with known classifications of the images in the validation image set to determine the accuracy of the model(s) used for such image set. In one embodiment, any type of metric that compares the difference or distance between labels can be used. For example, in one embodiment, if the prediction task is a binary task, process 700 can use the area under the receiver operating characteristic curve (ROC-AUC) to determine the comparison between the generated classification and known classifications of the images in the validation image set. In another embodiment, if the binary labels are imbalanced, process 700 can use the area under the precision recall curve (PR-AUC) to determine the comparison between the generated classification and known classifications of the images in the validation image set. In another embodiment, when predicting survival, process 700 can use a concordance index (c-index) to determine the comparison between the predicted risk order of the data points and the known order.

[0067] In addition to predicting the label of an image as described above in FIG. 1 , the preprocessing device 102 and / or the classification device 108 can further extract a comparison region of the image based on the tile scores of the image. In this embodiment, the comparison region differs from the region of interest as described in FIG. 3 because there can be tiles that are similar (e.g., visually similar) to one or more tiles with the minimum or maximum scores. For example, in one embodiment, the tiles of the comparison region can be tiles that have an average score (e.g., a score that is not the minimum or maximum of the calculated tile scores for the image) and are similar to tiles from the maximum or minimum scoring tiles. FIG. 8 is a flow diagram of one embodiment of a process 800 for extracting a comparison region from an image. In FIG. 8 , process 800 begins by receiving an image and model(s) at block 802. In one embodiment, the image is an image that can be classified by model(s) (e.g., a histopathological image, a satellite image, etc.), and the model(s) are used to select the comparison region. In this embodiment, the model may include a model used to segment the image, extract feature vectors, and / or score feature vectors as described above in FIG. 3. Process 800 segments the image at block 804. In one embodiment, process 800 segments the image as described in block 304 of FIG. 3. Process 800 tiles the image at block 806. In one embodiment, process 800 tiles the image as described in block 306 of FIG. 3. Process 800 extracts feature vectors for the image at block 808. In one embodiment, process 800 extracts a feature vector for each image tile as described in block 308 of FIG. 3. Process 800 scores the image tiles using the extracted feature vectors at block 810. In one embodiment, process 800 scores the image tiles as described in block 310 of FIG. 3. The processing loop ends at block 812.

[0068] The process 800 selects image tiles at block 814 using at least the tile score. In one embodiment, the tiles are used to determine the comparison region. In one embodiment, the process 800 selects tiles based on at least the tile score and the rank of the tile score. In this embodiment, the process 800 selects tiles based on the top scores (R top ) and / or subnumber (R bottom ) can be selected. In one embodiment, R top and R bottom represent the highest and lowest tile scores. In one embodiment, R top and / or R bottom The range of values ​​of R can be the same or different. top and / or R bottom The range can be a static numeric range (e.g., 10, 20, 100, or some other numeric value), a fitted range, a percentage, a label (e.g., small, large, or some other label), set via a user interface component (a slider, user input, and / or another type of user interface component), and / or some other value. Alternatively, process 800 can select a set of one or more tiles above a first threshold of tile scores or below a second threshold tile score, where the first threshold can be the same or different. In a further embodiment, process 800 randomly selects one or more tiles based on a probability derived from at least the corresponding tile scores. In this embodiment, using random selection based on a probability derived from at least the corresponding tile scores allows for the selection of tiles with outlying tile scores, meaning that there is a higher probability of selecting tiles with higher or lower scores, rather than strictly selecting tiles with the top N or bottom N tile scores.

[0069] Process 800 extracts comparison regions at block 816. In one embodiment, process 800 uses the selected tile(s) from block 814 above to extract comparison regions that are visually close to at least one of the selected tiles according to a distance metric associated with the selected tile(s) with an average score. For example, in one embodiment, the distance metric used by process 800 to evaluate visual proximity between two tiles may be an L2 norm calculated on the extracted features of the two tiles and / or an L2 norm calculated on these two tiles.

[0070] As described above, the models used for classification and tile selection do not require or use local annotations of the images made by experts (e.g., pathologists). In another embodiment, if local annotations are present in the image, these local annotations can be used to improve the predictive use of the model(s). Thus, in one embodiment, if local annotations, such as the presence of a tumor in a region of a slide, are available, these annotations can be taken into account using a hybrid technique. To do this, the device can train a machine learning model for two simultaneous tasks: (1) local prediction of the presence of macroscopic features on each tile (e.g., the presence of a tumor or other type of macroscopic feature), and (2) prediction of a global label set. On the one hand, the device (or multiple devices) can process the 128 feature set using a complex architecture that includes the classification system described above in FIG. 3. On the other hand, the device applies a convolutional neural network to convert the features of the N tiles into an N*128 feature vector. Based on this vector, the device trains a convolutional neural network to predict local annotation information, such as the presence or absence of a tumor, for each tile. The device can take both the output of the prediction and the N*128 feature vector and apply a weighted pooling operation to the concatenation of these two vectors to obtain a 128 feature vector for the input image. The device concatenates the output of the classification model with the obtained 128 features and attempts to predict a global label set for the image (e.g., survival rate, tumor size, necrosis, and / or other types of predictions) based on this vector. The model loss includes both global and local predictions. In this embodiment, adding information derived from local annotations to the computational flow can improve overall model performance.

[0071] Figure 9 is a block diagram of one embodiment of a system 900 for classifying images using classification models and local annotations. In Figure 9, the system 900 can use both slide-level processing 932 and tile-level processing 930. In one embodiment, the tile-level processing 932 processes tiles using local annotations to improve global label predictions for the entire system 900. In one embodiment, the tile-level processing 932 processes tiles and feature vectors as described above in Figure 3. Additionally, the results of each of the slide-level processing 930 and tile-level processing 932 can be combined and further processed to determine various predictions, as described below.

[0072] In one embodiment, system 900 receives an image having N tiles and corresponding feature vector sets derived from an input image using tiling and feature vector generation operations. For example, in one embodiment, system 900 receives a tiled image and corresponding feature vector sets generated as described above in FIG. 3 . In one embodiment, system 900 receives N tiles with feature vector sets of 256 features per feature vector, while in other embodiments, the feature vector sets may include feature vectors with fewer or more features. System 900 then sends the feature vector sets to slide-level processing 930 and tile-level processing 932. In this embodiment, each feature vector for tile-level processing 932 is reduced from 256 features to 128 features. In one embodiment, the feature vectors are reduced using a one-dimensional convolutional neural network 916. In this embodiment, the one-dimensional convolutional neural network 916 is used to extract interesting features for each tile. The 1D convolutional neural network 916 can be applied once, multiple times, or not at all. In one embodiment, the number of features per feature vector is reduced by half, while in another embodiment, the number of feature vectors remains the same, can decrease, and / or increase by different amounts. In a further embodiment, the system 900 applies a 1D convolutional neural network 928 to extract interesting features for each tile before sending the feature vector set to slide-level processing 930.

[0073] In one embodiment, tile-level processing 932 begins by receiving the feature vector set 908 generated by the one-dimensional convolutional neural network 916 described above. Furthermore, tile-level processing 932 uses the local annotations to train a second one-dimensional convolutional neural network on the feature vector set 908. In this embodiment, the second one-dimensional convolutional neural network can then be used to predict the presence of macroscopic features 910 in each tile for input images that do not include local annotations (e.g., the presence or absence of a tumor in each tile, the presence of other macroscopic medical features (e.g., inflammation, etc.), and / or the presence of other types of features). If local annotations are present in some or all of the tiles, the N predictions can be supplemented or replaced with predictions derived from the available local annotations. For example, in one embodiment, if a pathologist determines the partial or complete outline or identification of a tumor on an image, N predictions for the corresponding N tiles can be derived from the indication or absence of a tumor in each image tile. In the absence of local annotations, this trained one-dimensional convolutional neural network can be used to create N predictions 910. These N predictions 910 can be combined with the convolved feature vector set 908 by weighted polling of the N predictions and the convolved feature vector set 908 to create a new feature vector 912. In one embodiment, the new feature vector 912 is 128 features. In this embodiment, weighted pooling is used to weight the feature vector set 908 with the N predictions 910. In other embodiments, the new feature vector 912 can have more or fewer features.

[0074] At the same time, slide level processing 930 receives original feature vector set 902 and performs feature vector processing as described above in Figure 3 using dense layer 922 to generate feature set 906. In one embodiment, feature vector set 906 is 128 features, while in other embodiments, feature vector set 906 may include more or fewer features.

[0075] In one embodiment, the system has two different feature vectors: feature vector 912 generated by tile-level processing 930 and feature vector 906 generated by slide-level processing 932. The system can combine these two feature vectors by concatenating them into a resulting feature vector 914, which can be used to predict one or more global labels 928. For example, in one embodiment, for a medical image slide, one or more global labels can be predicted, such as survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions. In one embodiment, by using feature vectors from both slide-level processing 930 and tile-level processing 932, the resulting feature vector 914 can have influences from local annotations, which can be used for prediction.

[0076] As described above, system 900 can use local annotations, if available, to further improve the predictive capabilities of the classification model(s). FIG. 10 is a flow diagram of one embodiment of a process 1000 for classifying an image using a classification model and local annotations. In FIG. 10, process 1000 begins at block 1002 by receiving an image, local annotations, and a global label. In one embodiment, the local annotations can be data and / or metadata that can describe and / or indicate the presence or absence of macroscopic features in the input image. For example, in one embodiment, the local annotations can indicate the presence of tumor or tumor-like tissue in a particular region of the image. Process 1000 tiles the image to generate a set of feature vectors at block 1004. In one embodiment, process 1000 generates a feature vector for each tile in the image. In this embodiment, each feature vector can include 256 features. For example, in one embodiment, process 1000 generates tiles and feature vectors as described above in FIG. 3.

[0077] At this point, process 1000 can take two paths: for slide-level processing and for tile-level processing. For slide-level processing, process 1000 applies classification model(s) to generate a feature vector of 128 features at block 1006. In one embodiment, process 1000 applies classification model(s) to the feature vector set as described in block 308 of FIG. 3. Execution proceeds to block 1012 below.

[0078] Process 1000 begins at block 1008 with tile-level processing. Process 1000 predicts local labels for each image tile at block 1008. In one embodiment, process 1000 applies a one-dimensional convolutional neural network to extract interesting features for a set of 256 feature vectors. Process 1000 may also use the local annotations to train a second convolutional neural network to generate a set of predictions for each tile. Process 1000 uses the local predictions as model features at block 1010. At this point, process 1000 has generated a feature vector of 128 features from the tile-level processing of blocks 1008 and 1010. Execution proceeds to block 1012 below.

[0079] Process 1000 combines features from blocks 1006 and 1010 to create a composite feature vector of 256 features at block 1012. In one embodiment, process 1000 creates the composite feature vector by concatenating two feature vectors from blocks 1006 and 1010. Alternatively, the composite feature vector can be created using other means. Process 1000 also predicts a global label at block 1012 using the composite feature vector. In one embodiment, process 1000 can predict different global labels (e.g., survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions for medical image slides). In one embodiment, process 1000 predicts the global label by scoring tiles to create predictions as described above in blocks 310 and 312 of FIG. 3 .

[0080] FIG. 11 illustrates an example of a data processing system 1100 that can be used with an embodiment of the present invention. For example, system 1100 can be implemented including preprocessing device 102 and / or classification device 108, as shown in FIG. 1 above. Note that while FIG. 11 illustrates various components of a computer system, it is not intended to depict any particular architecture or manner of interconnecting the components, as such details are not germane to the present invention. It will also be understood that network computers and other data processing systems or other consumer electronic devices having fewer or more components can also be used with the present invention.

[0081] As shown in FIG. 11 , a computer system 1100 in the form of a data processing system includes a bus 1103 coupled to one or more microprocessors 1105, a ROM (read-only memory) 1107, a volatile RAM 1109, and a non-volatile memory 1111. The microprocessor 1105 may include one or more CPUs, GPUs, special purpose processors, and / or combinations thereof. The microprocessor 1105 may read instructions from the memories 1107, 1109, and 1111 and execute those instructions to perform the operations described above. The bus 1103 interconnects these various components and also interconnects these components 1105, 1107, 1109, and 1111 to peripheral devices such as a display controller and display device 9111 and input / output (I / O) devices, which may be a mouse, keyboard, modem, network interface, printer, and other devices known in the art. Typically, the input / output devices 915 are coupled to the system through an input / output controller 1113. Volatile RAM (Random Access Memory) 1109 is typically implemented as dynamic RAM (DRAM), which requires continuous power to refresh or maintain the data in the memory.

[0082] Typically, mass storage 1111 is a magnetic hard drive, magneto-optical drive, optical drive, DVD RAM, flash memory, or other type of memory system that retains data (e.g., large amounts of data) even after power is removed from the system. Typically, mass storage 1111 is also random access memory, although this is not required. While FIG. 11 shows mass storage 1111 as a local device directly coupled to the rest of the data processing system, it will be understood that the present invention may also utilize non-volatile memory off-system, such as a network storage device coupled to the data processing system through a network interface, such as a modem, an Ethernet interface, or a wireless network. Bus 1103 may include one or more buses connected to each other through various bridges, controllers, and / or adapters as are known in the art.

[0083] Portions of the above may also be implemented using logic circuitry, such as special-purpose logic circuitry, microcontrollers, or other forms of processing cores that execute program code instructions. Accordingly, the processes taught by the above description may also be performed using program code, such as machine-executable instructions, that cause a machine executing the instructions to perform a particular function. A "machine" in this context may be a machine that translates intermediate-form (or "abstract") instructions into processor-specific instructions (e.g., an abstract execution environment such as a "virtual machine" (e.g., a Java Virtual Machine), an interpreter, a Common Language Runtime, a high-level language virtual machine, etc.), and / or electronic circuitry (e.g., "logic circuitry" implemented with transistors) located on a semiconductor chip, such as a general-purpose processor and / or a special-purpose processor, designed to execute instructions. The processes taught by the above description may also be performed by (in place of or in combination with a machine) electronic circuitry designed to perform a process (or a portion thereof) without the execution of program code.

[0084] The present invention also relates to apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer program may be stored on a computer-readable storage medium such as, but not limited to, a floppy disk, optical disk, any type of disk including CD-ROM and magneto-optical disk, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical card, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0085] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, etc.

[0086] An article of manufacture can be used to store the program code. The article of manufacture storing the program code can be embodied as, but not limited to, one or more memories (e.g., one or more flash memories, random access memories (static, dynamic, or other)), optical disks, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code can also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a data signal embodied in a propagation medium (e.g., via a communications link (e.g., a network connection)).

[0087] The preceding detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0088] It should be noted, however, that these and similar terms should all be associated with the appropriate physical quantities and are merely convenient notations given to these quantities. As is apparent from the above description, unless specifically stated otherwise, descriptions utilizing terms such as "segmenting," "tiling," "receiving," "computing," "extracting," "processing," "applying," "augmenting," "normalizing," "pre-training," "sorting," "selecting," "aggregating," "sorting," and the like throughout the description should be understood to refer to operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (e.g., electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computer system.

[0089] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the described operations. The required structure for a variety of these systems will appear from the description below. Also, the present invention is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the invention as described herein.

[0090] The foregoing description describes only some exemplary embodiments of the present invention, and those skilled in the art will readily recognize from such description, the accompanying drawings, and the claims that various modifications can be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0091] 500 Classification System 502 Tiling 504 tile sampling 506 Feature Extraction 508 Tile Scoring 510 Tile Sorting 512 predicted survival rate 514 Whole slide images and true viability 516 pre-processed tiles (total of ~10,000 tiles per slide) 518 sampled tiles (~3,000 sampled tiles per slide) 520 local descriptor matrix (3,000 x 2048 scalar) 522 tile score (3,000 scalar score)

Claims

1. 1. A method for classifying an input image, comprising: applying a first convolutional neural network to segment the input image into regions of interest and background regions containing information useful for classification; tiling the region of interest into a set of tiles; for each tile, extracting a feature vector for said tile by applying a second convolutional neural network, wherein features in said feature vector represent local descriptors of said tile, and a score calculated from said feature vector is associated with each tile of said tile set, said score representing the contribution of the tile to the classification of said input image; The method comprises: processing the extracted feature vectors of the tile sets to classify the input image based on a first subset of the tile sets having the highest scores and a second subset of the tile sets having the lowest scores; Including, method.

2. 2. The method of claim 1, wherein the first convolutional neural network is a semantic segmentation neural network that classifies each pixel of the input image as one of a region of interest and a background region.

3. 3. The method of claim 2, wherein the semantic segmentation neural network is selected from the group consisting of U-NET, Fully Convolutional Networks for Semantic Segmentation, SegNet, and DeepLab.

4. The method of claim 1 , wherein the tiling comprises applying a fixed tiling grid to at least the region of interest, each of the tile sets having a predetermined size.

5. The method of claim 1 , wherein a zoom level is applied to the tileset.

6. The method of claim 1 , wherein multiple zoom levels are applied to the tilesets, and the tilesets at different zoom levels are combined.

7. 2. The method of claim 1 , wherein the tiling further comprises expanding the tile set by performing an operation on the tile set, the operation being at least one of rotating, translating, cropping, adding noise to the input image, modifying the intensity of one or more colors, or changing the contrast of the input image.

8. 2. The method of claim 1 , wherein the tiling further comprises normalizing tiles in the tile set for each input image by performing at least one of random sampling of the tile set and padding the tile set with blank tiles so that the tile set contains a constant number of tiles for each input image.

9. 2. The method of claim 1, wherein the second convolutional neural network is a residual neural network selected from the group consisting of a ResNet-type residual neural network, a VGG neural network, an autoencoder for unsupervised feature extraction, and an Inception neural network.

10. 10. The method of claim 1, further comprising pre-training weights of the second convolutional neural network on at least one of a different dataset and task, wherein at least one output layer is removed from the second convolutional neural network.

11. The process comprises: For each tile, calculating the score associated with the tile from the extracted feature vector, the score representing the contribution of the tile to the classification of the input image; sorting the set of scores; selecting the first subset having the highest scores and the second subset having the lowest scores based on at least one of the values ​​and ranks of the tiles within the set of sorted scores; applying a classifier to a subset of the scores to classify the image; The method of claim 1 , comprising:

12. The method of claim 1 , wherein the tiling further comprises aggregating groups of corresponding tiles from different images of a plurality of images, the plurality of images comprising the input image.

13. The method of claim 1 , wherein the tiling further comprises aggregating clusters of tiles from the tile set within a certain distance according to a distance metric calculated in at least one of the input image or feature map.

14. Aggregating a cluster of tiles concatenating tiles within said cluster of tiles; selecting a single tile from the cluster of tiles based on at least a given criterion, using the cluster of tiles as a multidimensional object for the feature vector extraction; calculating the mean value of the feature vectors of tiles within the cluster of tiles; calculating at least a maximum or minimum value of the feature vectors of tiles within the cluster of tiles; The method of claim 13 , comprising at least one of:

15. The method of claim 1 , wherein the feature vector extraction further comprises applying an autoencoder to the extracted feature vector to reduce the dimensionality of the features of the feature vector.

16. The method of claim 1 , wherein the input image is a histopathology slide and the region of interest is a tissue region.

17. The method of claim 1 , wherein the classification of the input image is used to perform at least one of a diagnostic classification, a survival prediction, and a response to treatment prediction.

18. The method of claim 1 , wherein the classification of the input image comprises predicting at least one global label and a local label.

19. 2. The method of claim 1 , wherein the processing of the extracted feature vectors includes using one or more local annotations associated with the input image incorporated as weights in a weighted combination of feature vectors derived from the extracted feature vectors that provide additional information for the classification of the input image.

20. The method of claim 1 , wherein the classification of the input image includes predicting multiple global labels in a multi-task environment.

21. 1. A non-transitory machine-readable medium having executable instructions for causing one or more processing units to perform a method for classifying an input image, the method comprising: applying a first convolutional neural network to segment the input image into regions of interest and background regions containing information useful for classification; tiling the region of interest into a set of tiles; for each tile, extracting a feature vector for said tile by applying a second convolutional neural network, wherein features in said feature vector represent local descriptors of said tile, and a score calculated from said feature vector is associated with each tile of said tile set, said score representing the contribution of the tile to the classification of said input image; The method comprises: processing the extracted feature vectors of the tile sets to classify the input image based on a first subset of the tile sets having the highest scores and a second subset of the tile sets having the lowest scores; 1. A machine-readable medium comprising:

22. 22. The machine-readable medium of claim 21, wherein the tiling comprises applying a fixed tiling grid to at least the region of interest, and wherein the each of the tile sets has a predetermined size.

23. 22. The machine-readable medium of claim 21 , wherein the tiling further comprises normalizing a number of tiles in the tile set for each input image by performing at least one of random sampling of the tile set and padding the tile set with blank tiles so that the tile set contains a constant number of tiles for each input image.

24. 22. The machine-readable medium of claim 21, wherein the second convolutional neural network is a residual neural network selected from the group consisting of a ResNet-type residual neural network, a VGG neural network, an autoencoder for unsupervised feature extraction, and an Inception neural network.

25. 22. The machine-readable medium of claim 21, further comprising pre-training weights of the second convolutional neural network on at least one of a different dataset and task, wherein at least one output layer is removed from the second convolutional neural network.

26. The process comprises: For each tile, calculating the score associated with the tile from the extracted feature vector, the score representing the contribution of the tile to the classification of the input image; sorting the set of scores; selecting the subset of scores based on at least one of the values ​​and ranks of tiles within the sorted set of scores; applying a classifier to a subset of the scores to classify the input image; 22. The machine-readable medium of claim 21, comprising:

27. 22. The machine-readable medium of claim 21, wherein the tiling further comprises aggregating groups of corresponding tiles from different images of a plurality of images, the plurality of images comprising the input image.

28. 22. The machine-readable medium of claim 21, wherein the tiling further comprises aggregating clusters of tiles from the tileset within a certain distance according to a distance metric calculated on at least one of the input image or feature map.

29. 22. The machine-readable medium of claim 21, wherein the feature vector extraction further comprises applying an autoencoder to the extracted feature vector to reduce the dimensionality of the features of the feature vector.

30. 22. The machine-readable medium of claim 21, wherein the input image is a histopathology slide and the region of interest is a tissue region.

31. 22. The machine-readable medium of claim 21, wherein the classification of the input image is used to perform at least one of a diagnostic classification, a survival prediction, and a response to treatment prediction.

32. 22. The machine-readable medium of claim 21, wherein the classification of the input image comprises predicting at least one global label and a local label.

33. 22. The machine-readable medium of claim 21 , wherein the processing of the extracted feature vector includes using one or more local annotations associated with the input image incorporated as weights in a weighted combination with a feature vector derived from the extracted feature vector that provides additional information for the classification of the input image.

34. 22. The machine-readable medium of claim 21, wherein classifying the input image includes predicting multiple global labels in a multi-task environment.

35. 1. A method for generating a classification model, comprising: receiving a set of training images, each of the set of training images having an associated known classification; For each training image in the set of training images: extracting a plurality of feature vectors of the training image by applying a first convolutional neural network, wherein the extracting comprises tiling a region of interest of the training image into a tile set, each of the plurality of feature vectors corresponding to a tile from the tile set, and a score calculated from the feature vectors associated with each tile, the score representing the contribution of the tile to the classification of the input image; The method comprises: training the classification model using at least a first subset of the tile sets having the highest scores and a second subset of the tile sets having the lowest scores and the associated known classifications; A method comprising:

36. 36. The method of claim 35, further comprising validating the classification model using at least a validation image set.

37. The verification may include: receiving the verification image set, each verification image in the verification image set having an associated known classification; For each verification image in the verification image set: extracting a plurality of feature vectors of the verification image by applying a first convolutional neural network, each feature of the plurality of feature vectors representing a local descriptor of the verification image; The method comprises: For each verification image in the verification image set: generating a verification score set for the verification image using at least the plurality of feature vectors; generating a classification for the validation image using at least the validation score set and the classification model; comparing the classification of the verification image to the associated known classification; 37. The method of claim 36, comprising:

38. The method of claim 37, wherein the comparison of the classifications of the validation images is performed using at least one of a comparison of the area under the receiver operating characteristic curve (ROC-AUC), a comparison of the area under the precision-recall curve (PR-AUC), or a concordance index (c-index) comparison.

39. 36. The method of claim 35, wherein the classification model is a multi-layer perceptron with two fully connected layers.

40. 36. The method of claim 35, wherein the training set images of the validation set are one of digital whole slide images (WSI).

41. 36. The method of claim 35, wherein the first convolutional neural network is a ResNet50 neural network.

42. 36. The method of claim 35, wherein the tiling comprises applying a fixed tiling grid to at least the region of interest, each of the tile sets having a predetermined size.

43. 36. The method of claim 35, wherein the training of the classification model comprises calculating the score for each tile of the tile set using at least a convolutional 1D layer and the tile's corresponding feature vector.

44. 36. The method of claim 35, wherein the method further comprises, for each of the training images in the training set of training images, applying a second convolutional neural network to segment the training image into regions of interest containing information useful for classification and background regions.

45. 45. The method of claim 44, wherein the second convolutional neural network is a U-NET neural network.

46. 36. The method of claim 35, wherein the classification model comprises one or more separate models.

47. 36. The method of claim 35, wherein the classification model comprises at least one of a multi-layer perceptron model and a one-dimensional convolutional neural network model.

48. 1. A machine-readable medium having executable instructions for causing one or more processing units to perform a method for generating a classification model, the method comprising: receiving a set of training images, each of the set of training images having an associated known classification; For each training image in the set of training images: extracting a plurality of feature vectors of the training image by applying a first convolutional neural network, wherein the extracting comprises tiling a region of interest of the training image into a tile set, each of the plurality of feature vectors corresponding to a tile from the tile set, and a score calculated from the feature vectors associated with each tile, the score representing the contribution of the tile to the classification of the input image; The method comprises: training the classification model using at least a first subset of the tile sets having the highest scores and a second subset of the tile sets having the lowest scores and the associated known classifications; 1. A machine-readable medium comprising:

49. 49. The machine-readable medium of claim 48, further comprising validating the classification model using at least a validation image set.

50. The verification may include: receiving the verification image set, each image in the verification image set having an associated known classification; For each verification image in the verification image set: extracting a plurality of feature vectors of the verification image by applying a first convolutional neural network, each feature of the plurality of feature vectors representing a local descriptor of the verification image; The method comprises: For each verification image in the verification image set: generating a verification score set for the verification image using at least the plurality of feature vectors; generating a classification for the validation image using at least the validation score set and the classification model; comparing the classification of the verification image to the associated known classification; 50. The machine-readable medium of claim 49, comprising:

51. The machine-readable medium of claim 50, wherein the comparison of the classifications of the validation images is performed using at least one of a comparison of the area under the receiver operating characteristic curve (ROC-AUC), a comparison of the area under the precision-recall curve (PR-AUC), or a concordance index (c-index) comparison.

52. 49. The machine-readable medium of claim 48, wherein the classification model is a multi-layer perceptron having two connected layers.

53. 49. The machine-readable medium of claim 48, wherein the training images of the training set or the validation set are one of digital whole slide images (WSI).

54. 49. The machine-readable medium of claim 48, wherein the first convolutional neural network is a ResNet50 neural network.

55. 49. The machine-readable medium of claim 48, wherein the tiling comprises applying a fixed tiling grid to at least the region of interest, each of the tile sets having a predetermined size.

56. 56. The machine-readable medium of claim 55, wherein the training of the classification model comprises calculating the score for each tile of the tile set using at least a convolutional 1D layer and the tile's corresponding feature vector.

57. 49. The machine-readable medium of claim 48, wherein the method further comprises, for each of the training images in the training set of training images, applying a second convolutional neural network to segment the training image into regions of interest containing information useful for classification and background regions.

58. 58. The machine-readable medium of claim 57, wherein the second convolutional neural network is a U-NET neural network.

59. 49. The machine-readable medium of claim 48, wherein the classification model comprises one or more separate models.

60. 49. The machine-readable medium of claim 48, wherein the classification model comprises at least one of a multi-layer perceptron model and a one-dimensional convolutional neural network model.

Images