System and method for image pre-processing

Abstract

A method and apparatus of a device classifying an image are described. In an exemplary embodiment, the device segments the image into a region of interest and a background region that includes information useful for classification by applying a first convolutional neural network. Further, the device tiles the region of interest into a set of tiles. For each tile, the device extracts a feature vector for the tile by applying a second convolutional neural network, where a feature of the feature vector represents a local descriptor of the tile. Further, the device processes the extracted feature vectors of the set of tiles to classify the image.

Description

[0001] Related applications

[0002] This application claims priority to European Patent Application No. EP 19305840.1, filed on June 25, 2019. The entire contents of the aforementioned priority application are incorporated herein by reference. Background Technology

[0003] Histopathological image analysis (HIA) is a crucial element of diagnosis in many medical fields, particularly in oncology, where this type of image analysis currently defines the gold standard. Recent work has attempted to leverage modern advancements in machine learning (ML) to assist pathologists in disease detection tasks, but most of these techniques require localized annotation masks as training data. Obtaining these local annotations is even more costly than the original diagnosis, as pathologists must spend time assembling pixel-by-pixel segmentation maps of diseased tissue at extremely high resolution. Consequently, the size of annotated HIA datasets is very limited. Furthermore, such local annotations may not be available when faced with novel problems in HIA, such as new disease subtype classifications, prognostic estimations, or drug response predictions. A key question for HIA is: can we design learning architectures that achieve accurate classification without local annotations? A successful technique should be able to train algorithms to assist pathologists during analysis and also be used to identify previously unknown structures and regions of interest.

[0004] In fact, while histopathology is the gold standard for diagnosis in oncology and other diseases, it is extremely expensive, requiring pathologists to spend a significant amount of time to make a single diagnosis. Furthermore, because a correct diagnosis of some diseases requires pathologists to identify a few cells from millions, these tasks are akin to "finding a needle in a haystack." Hard figures on the error rate of histopathological diagnoses are difficult to obtain, as it depends on the disease and tissue in question, as well as the pathologist's self-reporting of diagnostic errors. However, false negatives in cancer diagnoses not only have disastrous consequences for patients but also pose incredible financial risks to pathologists. Any tool that can help pathologists focus their attention and effort on the most suspicious areas can help reduce false negatives and improve patient outcomes through more accurate diagnoses. For decades, medical researchers have been seeking computer-aided diagnosis, but a lack of computational resources and data has hindered the widespread implementation and use of such tools. Since the advent of automated digital whole-slide image (WSI) capture in the 1990s, researchers have been searching for ways to reduce the workload of pathologists and improve patient outcomes through image processing algorithms. Instead of predicting the final diagnosis, many of these procedures focus on segmentation, for cell counting, or for detecting suspicious areas in the WSI. Historical approaches focus on using handcrafted texture or morphological features in combination with unsupervised techniques such as K-means clustering or other dimensionality reduction techniques before classification via k-nearest neighbors or support vector machines.

[0005] The remarkable advancements in deep learning over the past decade have led to an explosion in research into automating image processing tasks. While the application of this advanced machine learning (ML) technique in image tasks has been successful for many consumer applications, its adoption in medical imaging has been more gradual. However, these techniques show remarkable promise in the field of HIA. Specifically, in digital pathology using WSI (Weighted System Indication), highly trained and skilled pathologists examine digitally captured microscopic images from prepared and stained tissue samples for diagnosis. Digital WSI is a massive dataset consisting of images captured at multiple zoom levels. At maximum magnification, WSI can have a digital resolution of up to 100,000 pixels in both dimensions. However, due to the difficulty in obtaining local annotations, the dataset may only contain diagnostic labels at the WSI level, thus falling into the category of weakly supervised learning. Summary of the Invention

[0006] A method and apparatus for classifying images are described. In an exemplary embodiment, the apparatus segments an image into regions of interest and background regions that include information useful for classification by applying a first convolutional neural network. Furthermore, the apparatus blocks the regions of interest into a set of patches. For each patch, the apparatus extracts a feature vector of the patch by applying a second convolutional neural network, wherein the features of the feature vector represent local descriptors of the patch. Furthermore, the apparatus processes the extracted feature vectors from the set of patches to classify the image.

[0007] In another embodiment, the device extracts target regions of an image relevant to its classification. In this embodiment, the device divides the regions of interest in the image into a first set of tiles. For each tile, the device further extracts a feature vector by applying a convolutional neural network. The features of the feature vector represent local descriptors of the tile, and a tile score is calculated based on the extracted feature vector, where the tile score represents the tile's contribution to the image classification. Furthermore, for each tile, the device selects at least one other tile with a score validating a given attribute and extracts a comparison target region. This comparison target region is a second set of tiles with scores between high and low in the first set of tiles and visually close to the at least one other tile according to a distance metric.

[0008] In yet another embodiment, the device generates a classification model. In this embodiment, the device receives a set of training images, wherein each training image in the set has an associated known classification. For each training image in the set, the device further extracts multiple feature vectors of the training image by applying a first convolutional neural network, wherein each feature of the multiple feature vectors represents a local descriptor of the image. Furthermore, the device trains the classification model using at least the extracted feature vectors and the associated known classification.

[0009] Other methods and apparatus are also described. Attached Figure Description

[0010] The invention is illustrated by way of example rather than limitation in the accompanying drawings, in which the same reference numerals denote similar elements.

[0011] Figure 1 This is a block diagram of an embodiment of a system for classifying images by preprocessing them using a classification model.

[0012] Figure 2 This is a block diagram of one embodiment of a system for training a classification model using a training image set.

[0013] Figure 3 This is a flowchart of an example of image preprocessing and classification.

[0014] Figure 4 This is a flowchart of an example of a process for classifying images that have been divided into sets of tiles.

[0015] Figure 5 This is a diagram of a classification system used for classifying histopathological images.

[0016] Figure 6 This is a flowchart of one embodiment of the process of training and validating a classification model.

[0017] Figure 7 This is a flowchart of one embodiment of the process for validating a classification model.

[0018] Figure 8 This is a flowchart of an example of the process of extracting and comparing target regions from an image.

[0019] Figure 9 This is a block diagram of an embodiment of a system that uses classification models and local annotations to classify images.

[0020] Figure 10 This is a flowchart of an example of image classification using a classification model and local annotation.

[0021] Figure 11 An example of a typical computer system that can be used in conjunction with the embodiments described herein is illustrated. Detailed Implementation

[0022] A method and apparatus for classifying images are described. Numerous specific details are set forth in the following description to provide a thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention can be practiced without these specific details. In other instances, well-known components, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

[0023] The phrase "in one embodiment" or "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with that embodiment that may be included in at least one embodiment of the invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment.

[0024] In the following description and claims, the terms “coupled” and “connected”, and their derivatives, may be used. It should be understood that these terms are not intended to be synonyms with each other. “Coupled” is used to indicate two or more elements that may or may not be in direct physical or electrical contact with each other, cooperate with each other, or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled to each other.

[0025] The processes depicted in the diagrams below are performed by processing logic, which may include hardware (e.g., circuit systems, special-purpose logic, etc.), software (e.g., running on a general-purpose computer system or a special-purpose machine), or a combination of both. While these processes are described below according to some sequential operations, it should be recognized that some of the described operations may be performed in a different order. Furthermore, some operations may be performed in parallel rather than sequentially.

[0026] The terms “server,” “client,” and “device” are intended to refer to data processing systems in general, rather than to specific form factors used for servers, clients, and / or devices.

[0027] As mentioned above, histopathological image analysis can rely on expert local annotation before image analysis, which is slow and laborious. To overcome this problem, an image processing pipeline can be used to analyze images without the need for local annotation. In one embodiment, local annotation is metadata (e.g., text, tags, numbers, and / or another type of metadata) applied to a portion of an image rather than the entire image. For example, in one embodiment, local annotation may be a marker of a region of interest in an image, such as a histological image. Exemplary local annotations include markers that delineate or otherwise identify a portion of an image (e.g., a tumor region of an image, a stromal region of an image), identifiers of cell types within an image, identifiers of biological structures composed of multiple cells in an image, etc. In contrast, references to "(one or more) global annotations" in the specification refer to metadata applied to the entire image. Exemplary global annotations include labels that identify the entire image, data about how the image was acquired, labels that identify characteristics of the subject from whom the image was derived, such as labels indicating the age, sex, diagnosis, etc., of the subject from whom the image was derived, and / or any other data applicable to the entire image. In some embodiments, global annotations may indicate the presence, type, grade, or stage of a tumor known or understood to exist in the subject from whom the image was derived. In other embodiments, global annotations may indicate known characteristics of the subject from whom the image is derived, such as survival duration (e.g., survival duration after the sample represented in the image has been acquired) or response to a given treatment. In some embodiments described herein, images containing global annotations may be used without local annotations.

[0028] The pipeline initially focuses on segmenting large images (e.g., WSI) into smaller images, such as 224×224 pixel images, and detecting regions of interest in these images using the Otsu method for classification. This classification is therefore suitable for small images and is significantly less computationally expensive. These smaller images are fed into a ResNet convolutional neural network to extract a feature vector from each small image, which is a local descriptor for that small image. A score is calculated for each small image from the extracted feature vectors, serving as a local patch-level (instance) descriptor. The top and bottom instances are used as input to a multilayer perceptron (MLP) for classification. This solution delivers good results, but its overall efficiency still needs improvement.

[0029] A method and apparatus are described to improve upon the aforementioned pipeline and classify images. In one embodiment, the apparatus uses one or more neural network models to classify an image to determine its label. In this embodiment, the image may be a large image, where processing the entire image using only a neural network model is computationally impractical. In particular, the apparatus reduces the amount of computational resources (e.g., time and / or memory requirements) required to perform image classification tasks on these large images. This reduction in resources further improves the performance of the apparatus when performing image classification tasks. Furthermore, the apparatus can classify entire slices of images, even if such images are too large to fit into the memory of graphics processing units typically used to train machine learning models.

[0030] In another embodiment, the device reduces the dimensionality of the data, thereby providing better generalization error and being more efficient in terms of model accuracy.

[0031] According to one embodiment, the device classifies at least one input image by segmenting the image between at least one region of interest containing information useful for classification and at least one background region not containing information useful for classification, using a first convolutional neural network. The device further divides the at least one region of interest into a set of patches. Furthermore, the device extracts a feature vector for each patch by applying a second convolutional neural network, where features are local descriptors of the patch. The device then processes the extracted feature vectors of the patches to classify the image. In one embodiment, by segmenting the input image, the device processes a reduced number of patches and avoids processing the entire image.

[0032] In one embodiment, the first convolutional network is a semantic segmentation neural network that classifies pixels of the input image into one of two categories: (a) regions of interest; and (b) background regions. Furthermore, a tiling step can be performed by applying a fixed tile grid to the image, such that the tiles have a predetermined size. Additionally, at least one level of scaling can be applied to the obtained tiles. For example, and in one embodiment, multiple scaling levels can be applied to the tiles and tiles with different scaling levels can be combined. Furthermore, the device can optionally randomly sample the tiles and / or fill the tile set with blank tiles, such that the tile set includes a given number of tiles.

[0033] In another embodiment, the second convolutional neural network can be a residual neural network, such as a ResNet50 residual neural network, a ResNet101 residual neural network that uses the previous layer as output and removes the last layer, or a VGG neural network. This second convolutional neural network can be a pre-trained neural network, allowing the use of state-of-the-art advanced neural networks without requiring a large-scale image database and computational resources to train it.

[0034] In one embodiment, the device can compute at least one score for a patch based on the extracted feature vector, where each patch score represents the patch's contribution to image classification. Using the patch scores, the device can sort the set of patch scores and select a subset of patch scores based on the values ​​of the patch scores and / or their rank in the sorted set; and apply a classifier to the retained patch scores to classify the image. The device can also apply this classification to multiple input images, where it can aggregate corresponding groups of patches from different input images.

[0035] In an alternative embodiment, the device may also aggregate clusters of adjacent tiles. In this embodiment, aggregating tile clusters may include tiles from cascaded clusters, selecting individual tiles from clusters using clusters as multidimensional objects according to given criteria, or aggregating values, for example, through mean or max pooling operations. Furthermore, the device may apply an autoencoder to the extracted feature vectors to reduce the dimensionality of the features. In one embodiment, the image may be a histopathological slice, a region of interest as a tissue area, or a classification of an image for diagnostic purposes.

[0036] In an alternative embodiment, when local annotations are available (such as the presence of a tumor in a slice region), these annotations can be taken into account using a hybrid technique. To this end, the device can train a machine learning model for two concurrent tasks: (1) local prediction of the presence of macroscopic attributes (e.g., the presence of a tumor or other types of macroscopic attributes) on each tile and prediction of a global set of labels. The device (or multiple devices) can use a complex architecture that, on the one hand, involves the aforementioned classification system to handle a set of 128 features. On the other hand, the device applies a convolutional neural network to transform the features of N tiles into N*128 feature vectors. Based on this vector, the device trains a convolutional neural network to predict the presence of a tumor (or some other macroscopic attribute) for each tile. The device can obtain both the predicted output and the N*128 feature vectors and apply a weighted pooling operation to the concatenation of these two vectors to obtain 128 feature vectors for the input image. The device concatenates the output of the classification model with the obtained 128 features and attempts to predict a global set of labels for the image (e.g., survival rate, tumor size, necrosis, and / or other types of predictions) based on this vector. The model's loss involves both global and local predictions. In this embodiment, the overall model performance can be improved by adding information derived from local annotations to the computation process.

[0037] Figure 1 This is a block diagram of one embodiment of a system for classifying images by using preprocessing and classification models. Figure 1 In this system, system 100 includes a client 118 coupled to a preprocessing device 102, which in turn is coupled to a classification device 108. In one embodiment, the preprocessing device 102 and the classification device 108 receive an input image and output an image classification using one or more preprocessing models 106 and one or more classification models 112. In this embodiment, the image classification can be one of different labels, numbers selected from a range of numbers, and / or other types of classification.

[0038] In one embodiment, each of the client 118, preprocessing device 102, and classification device 108 may be a personal computer, laptop, server, mobile device (e.g., smartphone, laptop, personal digital assistant, music player, gaming device, etc.) and / or any device capable of processing images. In one embodiment, the client 118, preprocessing device 102, and / or classification device 108 may each be a physical or virtual device independently. In one embodiment, the client 118 may be a device used by a user (such as a medical professional wishing to determine a patient's prognosis from an image) to determine image classification. In one embodiment, the image may be a large image on the order of gigabytes or larger. For example, and in one embodiment, the image may be a digital image of a histopathological slide (e.g., WSI), where the image may be on the order of gigabytes or larger. While in one embodiment the image is a histopathological slide, in alternative embodiments the image may be another type of image (e.g., satellite imagery, astronomical imagery, wind turbine imagery, another type of medical image, and / or any other type of large image).

[0039] In another embodiment, the preprocessing device 102 segments the image into regions of interest and background regions. In this embodiment, extracting regions of interest from the input image reduces the computational load required to classify the input image. For example, and in one embodiment, because histopathological slides (or other types of images) may include one or more blank areas of the image with little or no tissue, it is useful to introduce so-called “tissue detection” or “material detection” methods to assess whether the regions of the slide contain any tissue. More generally, when the goal is to classify large images, it is relevant to identify regions of interest in the image and distinguish them from background regions. These regions of interest are image regions that contain information valuable for classification processing. Furthermore, background regions are areas of the image that contain little or no valuable information, where background regions can be considered noise for the task at hand. Image segmentation is described below. Figure 3 Further details are provided below.

[0040] In one embodiment, for a segmented image, the preprocessing device 102 can block the image. In this embodiment, block segmentation is the process of dividing an image into a set of image subsets. The image subsets can be of a fixed size or can be of different sizes. Furthermore, block segmentation can be based on regions of interest generated from the image segmentation described above. The following... Figure 3 The text further describes the block division.

[0041] In another embodiment, the preprocessing device 102 uses a convolutional neural network to determine the features of the image. In this embodiment, the preprocessing device 102 computes a feature vector for each tile in the tile set. These feature vectors can be used to classify the image using a classifier. (See below...) Figure 3 The characteristics to be defined are further described below. In one embodiment, the preprocessing device 102 includes a preprocessing module 104 to perform the functions of the preprocessing device 102 using one or more preprocessing models 106.

[0042] In one embodiment, preprocessing device 102 may send a set of tiles and feature vectors to classification device 108, where classification device 108 uses the set of tiles, feature vectors, and classification model 112 to classify the input image 114. In this embodiment, classification device 108 calculates a tile score for each tile in the set of tiles, sorts the set of tiles, and selects a subset of tiles from the set. Using the tile subsets, classification device 112 classifies the images in the tile subsets and outputs image classification 114. In one embodiment, image classification is sent to client 118. In an alternative embodiment, image classification 114 (and possibly along with the input image) may be stored in a repository, where the client can retrieve input image 116 and / or image classification 114. The following... Figure 3 The document further describes the use of tiles to classify images. In one embodiment, the functions of the preprocessing device 102 or the classification device 108 may be performed by another device, or alternatively, one device may perform the functions of both the preprocessing device 102 and the classification device 108. In one embodiment, the classification device 108 includes a classification module 110 to perform the functions of the classification device 108 using one or more classification models 112.

[0043] like Figure 1 As described above, the client uses preprocessing device 102 and / or classification device 108 to preprocess and classify images. In one embodiment, each of the preprocessing model 106 and / or classification model 112 can be trained using a set of input images with known attributes. For example, and in one embodiment, classification model 112 can be trained using the set of input images and a set of known labels for each input image. Furthermore, the set of input images may also include information that can be used to train the image segmentation model.

[0044] Figure 2 This is a block diagram of one embodiment of a system 200 for training a classification model using a training image set 216. Figure 2In this system, system 200 includes a preprocessing device 202 coupled to training device 206. In one embodiment, preprocessing device 202 receives an image training set 216 and generates a set of feature vectors for each image in the image training set 216. Training device 208 may receive the feature vector sets and use input labels 212 associated with the image training set 216 to train a classification model 214. The following... Figure 6 The training process is further described in the text.

[0045] As described above, the preprocessing model 102 and the classification device 108 can preprocess and classify images. Figure 3 This is a flowchart of one embodiment of a process 300 involving image preprocessing and classification. In one embodiment, a preprocessing device 102 and / or a classification device perform the functions of process 300. Figure 3 In this embodiment, processing 300 begins by receiving an image, one or more models, and other inputs at box 302. In one embodiment, the input image may be a histopathological image, satellite image, astronomical image, wind turbine image, another type of medical image, and / or any other type of large image. In this embodiment, the model is one or more models that can be used to preprocess and / or classify the image. For example, and in one embodiment, the model may include an image segmentation model for segmenting the image and a classification model for classifying the image. Furthermore, other inputs may include weights and other input parameters used by processing 300.

[0046] At box 304, process 300 segments the input image. In one embodiment, process 300 segments the input image into regions of interest and background regions. In this embodiment, by segmenting the image, process 300 can extract regions of interest from the input image. In this embodiment, extracting regions of interest from the input image can reduce the computational cost required to classify the input image. For example, and in one embodiment, histopathological slides (or other types of images) may include one or more blank regions of an image with little or no tissue, so it is useful to introduce so-called “tissue detection” or “material detection” methods to assess whether the regions of the slide contain any tissue. More generally, when the goal is to classify large images, it is relevant to identify regions of interest in the image and distinguish them from background regions. These regions of interest are image regions that contain information valuable for the classification process, and background regions are regions of the image that contain little or no valuable information, where background regions can be considered noise for the task at hand. To achieve this task, various different types of image segmentation schemes can be used. For example, and in one embodiment, the Otsu method can be used to segment the image, where the Otsu method is a simple thresholding method based on the image intensity histogram. In this embodiment, the Otsu method has shown very good results for image segmentation when the image contains two classes of pixels following a bimodal distribution (e.g., foreground pixels and background pixels, or more specifically, organized and unorganized). However, this method performs poorly on complex images when the intensity level histogram cannot be assumed to have a bimodal distribution. More robust techniques are needed to improve the overall efficiency of this method.

[0047] In another embodiment, to improve the robustness of image segmentation and enable the handling of complex images, a semantic segmentation neural network can be used for image segmentation, such as the U-NET semantic segmentation neural network, SegNet, DeepLab, or another type of semantic segmentation neural network. In this embodiment, a semantic segmentation neural network that does not rely on a specific distribution in the intensity histogram can be used. Furthermore, using such a neural network allows image segmentation to consider multi-channel images, such as RGB images. Therefore, segmentation depends not only on the histogram of pixel intensity but also on the semantics of the image. In one embodiment, the semantic segmentation neural network is trained to segment tissue from the background, such as distinguishing stained or unstained tissue from the background. In an alternative embodiment, for different types of images, the semantic segmentation neural network can be trained to distinguish the foreground from the background of that type of image.

[0048] In another embodiment, the original image can be downsampled to reduce the computational cost of the image segmentation step. As will be further described below, in one embodiment, some image analysis is performed at the tile level (which is a sub-part of the image), and using semantic segmentation on the downsampled version of the image does not degrade the segmentation quality. This allows the use of downsampled images without actually reducing the segmentation quality. Then, to obtain the segmentation mask of the original full-resolution image, it is only necessary to upscale the segmentation mask generated by the neural network.

[0049] In another embodiment, a further advantage of using the U-NET segmentation neural network is that this type of network has been developed for biomedical image segmentation, thus meeting the typical constraints of biomedical data with small datasets of very high dimensionality. In fact, the U-NET segmentation neural network is a model that requires very few parameters to train, allowing the network to be trained with fewer training samples. Furthermore, in another embodiment, data augmentation techniques applied to the training data can produce very good results with this architecture, allowing for more training examples to be obtained from the same training set.

[0050] Processing 300 divides the image into a set of image tiles at box 306. In one embodiment, processing 300 uses tile division to increase the ability to preprocess images. For example, and in one embodiment, the tile division approach is helpful for histopathological analysis due to the large size of whole-slice images. More broadly, when processing specialized images such as histopathological slices, satellite images, or other types of large images, the resolution of image sensors used in these fields can grow as rapidly as the capacity of the random access memory associated with the sensor. As image size increases, it becomes difficult to store batches of images within a computer's random access memory, and sometimes even a single image. This difficulty is further complicated if one attempts to store these large images in the dedicated memory of a graphics processing unit (GPU). This situation makes processing whole image slices or any other images of similar size computationally intractable.

[0051] In one embodiment, image (or region of interest) processing 300 addresses this challenge by dividing the original image (or region of interest) into smaller, more manageable images (referred to as tiles). In one embodiment, the tile-slicing operation is performed by applying a fixed grid to the full-slice image, using a segmentation mask generated by a segmentation method, and selecting tiles containing tissue or any other type of region of interest for subsequent classification processing. To further reduce the number of tiles to be processed, additional selection methods, such as random subsampling, can be used to retain only a given number of slices.

[0052] For example, and in one embodiment, process 300 divides the region of interest into tiles of a fixed size (e.g., each tile has a size of 224 × 224 pixels). Alternatively, the tile size can be smaller or larger. In this example, the number of tiles generated depends on the size of the detected substance and can range from several hundred tiles to 50,000 or more tiles. In one embodiment, the number of tiles is limited to a fixed number (e.g., 10,000 tiles) that can be set at least based on computation time and memory requirements.

[0053] Furthermore, Process 300 can apply scaling levels to tiles generated from a mesh, where the scaling operation is adjusted so that each tile has the same dimensions. The choice of scaling is a trade-off between detail and context: higher scaling levels allow Process 300 to extract more detailed information from the tiles, while lower scaling levels preserve the context of the selected area. To benefit from multi-level scaling, Process 300 can extract tiles under multiple scaling levels, enabling the model to extract more information from a given slice. These multi-level scaling can be combined by concatenating multiple tiles with different scaling levels and the same center location.

[0054] At box 308, process 300 extracts feature vectors from the set of tiles. In one embodiment, feature extraction constructs derived values ​​from the tiles created from the input image that are designed to provide information and are non-redundant, thereby facilitating subsequent learning and generalization steps. In one embodiment, the extracted features are in the form of numerical vectors, also known as local descriptors.

[0055] In one embodiment, the processing 300 can use any feature extraction neural network, such as ResNet-based architectures (ResNet-50, ResNet-101, ResNetX, etc.), Visual Geometric Group (VGG) neural networks, Inception neural networks, autoencoders for unsupervised feature extraction, custom neural networks specifically designed for the task, or even non-neural network feature extractors such as SIFT or CellProfiler. Furthermore, the feature extraction neural networks used can be pre-trained because they are trained on very large datasets and therefore have optimal generalization accuracy.

[0056] In one embodiment, processing 300 uses a ResNet-50 neural network because this network can provide features well-suited for image analysis without requiring excessive computational resources. For example, in one embodiment, ResNet-50 can be used for histopathological image analysis. In this example, the ResNet-50 neural network relies on residual blocks, which allow the neural network to be deeper while still improving its accuracy, as simple convolutional neural network architectures may achieve the worst accuracy when the number of layers grows too large. In one embodiment, the weights of the ResNet-50 neural network can be weights used for feature extraction, derived from pre-training on the ImageNet dataset, as this dataset is a truly general-purpose image dataset. In one embodiment, using a neural network pre-trained on a large, independent image dataset provides good features independent of image type, even when the input image is specialized, such as for histopathological images (or other types of images). In this embodiment, processing 300 uses a ResNet-50 convolutional neural network to extract 2,048 features per patch. For example, if Process 300 extracts 10,000 tiles, then Process 300 generates a 2,048 × 10,000 matrix. Furthermore, if Process 300 is performed with multiple images as input, then Process 300 generates a tensor with the following dimensions: number of images × number of features / number of tiles × number of tiles.

[0057] In one embodiment, and to extract features from a given slice, process 300 processes each selected patch, traversing the ResNet-50 neural network, and outputs a feature vector for that patch. In this embodiment, the feature vector may be a vector of dimension 2048. Furthermore, process 300 may apply an autoencoder to the feature vector to further provide dimensionality reduction and reduce the dimension of the feature vector to, for example, 256 (or another dimension). In one embodiment, an autoencoder may be used when the machine learning model may be prone to overfitting. For example, and in one embodiment, process 300 may reduce the length of the 2048 feature vectors to a feature vector of length 512. In this example, process 300 may optionally use an autoencoder comprising a single hidden layer architecture (of 512 neurons). This prevents the model from overfitting by finding a few singular features in the training dataset and also reduces computation time and required memory. In one embodiment, the classification model is trained on a small subset of image patches, for example, on 200 patches randomly selected from each slice (411,400 patches in total).

[0058] Process 300 may optionally perform a zero-padding operation on the feature vectors to derive a minimum number of features. In this embodiment, if the number of feature vectors is less than the minimum number of feature vectors, then process 300 may perform zero-padding to add the feature vectors to the feature vector set of the image. In this embodiment, each zero-padding feature vector has a null value.

[0059] At box 310, process 300 scores each tile. In one embodiment, process 300 uses a connected neural network to reduce each feature vector to one or more scores. In another embodiment, process 300 can use a fully connected neural network to reduce the feature vector to a single score, or use a fully connected neural network that outputs various scores, or multiple fully connected neural networks that each output different scores, to reduce the feature vector to multiple scores representing various characteristics of the tile. These scores associated with a tile are sorted, and a subset of tiles is selected for image classification. In one embodiment, the subset of tiles can be tiles with a top R highest scores and a bottom R lowest scores, a top R highest scores, a bottom R lowest scores, and / or any weighted combination of these scores. Finally, these scores are concatenated into an image score vector that can be used as input for image classification.

[0060] For example, and in one embodiment, processing 300 can use a convolutional 1D layer to create a score for each tile. In the example described above with a feature vector length of 2,048, the convolutional layer performs a weighted summation across all 2,048 features of the tile to obtain the score, where the weights of the sum are learned by the model. Furthermore, because processing 300 uses a convolutional 1D layer with zero bias, zero-filled tiles have zero scores, and therefore, each zero-filled tile is a reference to a completely informationless tile. Processing 300 selects the highest and lowest R scores and uses them as input for the classification described below. This architecture ensures which tiles are used for prediction, and therefore how processing 300 predicts the results. Tile scores are shown below. Figure 4 Further details are provided below.

[0061] At box 312, processing 300 uses tile scores to classify the image to predict one or more global labels for the image. In one embodiment, processing 300 uses an image score vector as input to a dense, multi-layer neural network to provide the desired classification. This classification can be any task that associates labels with data input to a classifier. In one embodiment, a trained classifier is used for histopathological slide image input. Since the input data is derived from the entire pipeline, the classifier is able to label the histopathological slide or any other type of image given as input without processing the entire image, which can be computationally prohibitively expensive. For example, and in one embodiment, the labels can be any kind of label, such as: binary values ​​representing the prognosis of a given pathology; numerical labels representing predictions of scores, probabilities, or physical quantities, such as survival predictions or responses to treatment predictions; and / or scalar labels as previously described or vectors, matrices, or tensors of such labels representing structured information. For example, and in one embodiment, processing 300 can output a continuous risk score as a classification of the mesothelioma histopathological input image, which can be used to determine the estimated survival duration of a subject. In this example, different consecutive output scores for different subjects can be plotted against similar risk scores from multiple mesothelioma subjects with known survival durations to determine the estimated survival time of an individual test subject. In one embodiment, processing 300 uses a multilayer perceptron (MLP) with two fully connected layers having 200 and 100 neurons with sigmoid activation, respectively. In this embodiment, the MLP serves as the core of a prediction algorithm that transforms tile scores into one or more labels. While in one embodiment, processing 300 predicts a single label (e.g., risk score) for an image, in alternative embodiments, processing 300 may predict multiple global labels for an image. In one embodiment, processing 300 may perform multi-task learning to predict multiple global labels. For example, and in one embodiment, a classification model (e.g., an MLP and / or other models described elsewhere) may be trained to predict multiple labels simultaneously in a multi-task learning setting (e.g., survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions).

[0062] As from Figure 3 As can be seen, images can be classified based on at least a set of patch scores derived from the feature vectors of image patches generated from neural networks. Figure 4 This is a flowchart of one embodiment of a process 400 for scoring an image that has been divided into a set of tiles. In one embodiment, the process 300, such as at box 310, performs process 400 to score the image. Figure 4In this process, 400 begins by receiving a set of tiles at box 402. In one embodiment, the set of tiles is... Figure 3 The generated tile set is shown in box 306. Furthermore, the tile set includes the feature vector of each tile in the tile set. At box 404, processing 400 uses the associated feature vector of each tile to compute a tile score. For example, and in one embodiment, processing 400 may use a convolutional 1D layer to create a score for each tile. In the example described above with a feature vector length of 2,048, the convolutional layer performs a weighted summation across all 2,048 features of the tile to obtain the score, where the weights of the sum are learned by the model. Furthermore, since the convolutional 1D layer is unbiased, zero-padded tiles have a score of zero and are therefore references to completely informationless tiles.

[0063] At box 406, process 400 sorts the set of tiles. In one embodiment, process 400 sorts the set of tiles to determine the top R and / or bottom R scores in box 408 below. Process 400 selects a subset of tiles at box 408, where this subset of tiles is used in a later classification step. In one embodiment, this subset of tiles may be those with the top R... top The highest score and the bottom R bottom The lowest score, the top R top The highest score, bottom R bottom A tile representing the lowest score and / or any weighted combination of these scores. In one embodiment, R top and / or R bottom The range of values ​​for R can be the same or different. Furthermore, R... top and / or R bottom The range can be a static numerical range (e.g., 10, 20, 100, or some other number) suitable for a range, percentage, label (e.g., small, large, or some other label), and / or some other value, set via a user interface component (slider, user input, and / or other type of user interface component). In one embodiment, the process 400 additionally concatenates these scores into an image score vector, which can be used as input for image classification.

[0064] In one embodiment, when studying whole-section images (or sections) of histopathological tissue, a patient (or subject) can be associated with multiple sections obtained from different locations within the same sample, from multiple organs, or at different time points with various staining methods. In this embodiment, sections from a single patient can be aggregated in various ways. In one embodiment, processing 300 can cascade sections to create larger sections that will be processed in the same or similar manner as normal sections (segmentation, dicing, feature extraction, and classification).

[0065] In another embodiment, processing 300 can process multiple slices into a 3D image. This is particularly useful when the slices are multiple consecutive slices of the same tissue. In this case, a 3D convolutional neural network (CNN) can be used for the feature extraction step to extract the most information from a given set of slices. Furthermore, by applying segmentation, tiling, and feature extraction steps to multiple slices and retaining only a given number of features to match the input dimension used by the classifier, the selected features can be, for example, the N largest features and M smallest features for each patch. This approach is particularly suitable when the multiple slices to be assembled are a batch of identical slices but using multiple colorings.

[0066] In another embodiment, process 300 can cluster nearby tiles based on a distance metric computed on the image or features, and aggregate their associated feature vectors by computed mean, maximum, or weighted sum of feature vectors. This allows process 300 to significantly reduce the dimensionality of the problem by starting with a large number of tiles (e.g., 10,000 tiles) and aggregating them into a small number of clusters (e.g., 200), which reduces computation time but also reduces overfitting and imparts better generalization error to the model. More specifically, a good distance metric for selecting tiles to aggregate is the Euclidean distance in the whole slice image, such as aggregating all tiles contained in a 1 mm² block on the slice.

[0067] In another embodiment, the slice may not contain enough useful tissue to extract patches on which the feature extraction step is applied and thus fed to the classifier. In this case, the classifier's input is zero-padded, meaning that for each missing patch, features consisting only of zeros are added to the true features computed by the feature extractor.

[0068] As mentioned above, one example of image classification is the classification of histopathological images. In this example, the calculated label is a predicted value based on the patient's input histopathological slide image. Figure 5 This is an illustration of Classification System 500, used for classifying histopathological images. Figure 5In this system, classification system 500 includes functions for patching 502, patch sampling 504, feature extraction 506, patch scoring 508, patch ranking 510, and classification (in this case, survival prediction) 512. In one embodiment, patching function 502 receives a full-slice image 514 and outputs a preprocessed set of patches 516. In this embodiment, the number of patches in the image can be on the order of 10,000 patches. In another embodiment, the image can have more or fewer patches. In one embodiment, and to reduce computational complexity, classification system 500 samples patch 504 to reduce the number of patches 518 used in neural network computation. In one embodiment, classification system 500 samples patch 504 randomly or using some other type of sampling mechanism. For example, in one embodiment, classification system 500 randomly samples patch 504 to reduce the number of patches from the order of 10,000 patches to the order of several thousand patches (e.g., 3,000 patches).

[0069] In one embodiment, classification system 500 performs feature extraction 506 on sampled patches. In this embodiment, classification system 500 uses a convolutional neural network (e.g., using ResNet-50 or another type of feature extraction mechanism described above) to extract features on the sampled patches, thereby generating a local descriptor matrix 520. Furthermore, in one embodiment, classification system 500 scores the patches 522 based at least on the patch feature vector 508. In this embodiment, classification system 500 uses a 1D convolutional neural network to generate a score for each image patch. Classification system 500 additionally sorts the patch scores 510, where an MLP regressor uses the sorted patches to generate a predicted survival 512. In one embodiment, each of these functions performed by classification system 500 is related to the above... Figure 3 The functions described in the text are the same or similar.

[0070] In one embodiment, researchers can use tile selection to determine the correlation between patterns in an image and relevant global labels associated with the image or entities associated with the image (e.g., images of patients, subjects on satellites, or wind turbines). In this embodiment, for a specific type of image, researchers can examine selected tiles from images with known (or unknown) labels to determine correlations. For example, and in one embodiment, researchers can examine selected tiles from histopathological images processed by Process 300 and compare the selected tiles with corresponding image labels or other corresponding data indicating survival time length to discover or correlate trends between histopathological image features indicated by the selected tiles and survival.

[0071] As previously described, and in one embodiment, the method is illustrated for histopathological image analysis, where the task can assist pathologists in disease detection. However, processing 300 and / or 400 can be applied to any type of image processing problem where running a classifier across the entire image is computationally difficult. For example, one such use case could be classifying high-resolution images from satellite imagery. In this example, a semantic segmentation neural network and a classifier (e.g., an MLP regressor and / or (one or more) other models) are trained to classify high-resolution satellite images.

[0072] As described above, the process 300 uses trained models(s) to determine labels(s) for each input image(s). In one embodiment, the process 300 uses trained models for image segmentation, convolutional neural network scoring, and classification. In this embodiment, each of these trained models is trained for a specific category of images (e.g., histopathological image slices, satellite images, and / or other types of image categories). In another embodiment, Figure 3 Some of the models used are trained end-to-end, meaning the models are trained together. In this embodiment, Figure 3 Some of the models used can be trained together, such as an MLP for image classification and a one-dimensional convolutional neural network for tile scoring. Furthermore, Figure 3 Other models used (e.g., ResNet, U-Net, and / or other types of models) can be trained separately on different training sets. Figure 6 This is a flowchart of one embodiment of the process 600 for training and validating a classification model. In one embodiment, the classification model may include methods for... Figure 3 The classification process described herein refers to one or more individual models (e.g., MLPs and / or one-dimensional convolutional neural networks). Figure 6 In this process, 600 begins by receiving a set of training images at box 602. In one embodiment, the set of training images is used to train a classification model (and / or other models used in process 300) for a specific category of images. For example, and in one embodiment, the set of images may be histopathological slide images that have been labeled as being associated with a specific type of disease (such as cancer). At box 604, process 600 receives a model. In one embodiment, the model is a classification model, such as an MLP model and one or more of the other models described above.

[0073] Process 600 executes a processing loop (blocks 606-610) to generate a set of feature vectors for the training image set. At block 608, process 600 extracts feature vectors from the images in the training set. In one embodiment, process 600 is as described above. Figure 3The feature vector extraction described above. For example, and in one embodiment, process 600 uses a ResNet-50 convolutional neural network to determine the feature vector as described above. Figure 3 The feature vector of each patch of the image segmentation method described herein. In one embodiment, process 600 generates a set of feature vectors for the training image. Furthermore, process 600 may perform data augmentation during training of the method to improve generalization error. Such data augmentation can be accomplished by applying various transformations to the patches (such as rotation, translation, cropping, adding noise to the image, modifying the intensity of specific colors, or changing the contrast).

[0074] The processing loop ends at position 610.

[0075] At box 612, processing 600 trains one or more models using extracted feature vectors from a training image set and input labels from the training image set. In one embodiment, processing 600 trains a one-dimensional convolutional neural network and an MLP classification model together using the labels from the training image set. In this embodiment, processing 600 iteratively trains one or more models by: calculating a set of scores for the training images, predicting labels, determining the difference between the predicted labels and the input labels, and optimizing one or more models based on the difference (e.g., calculating new weights for one or more models) until the difference is within a threshold. While in one embodiment processing 600 trains a model to predict a single label (e.g., a risk score) for an image, in alternative embodiments processing 600 may be trained to predict multiple global labels for an image. In one embodiment, processing 600 may be trained to perform multi-task learning to predict multiple global labels. For example, and in one embodiment, a classification model (e.g., an MLP and / or other models described elsewhere) can be trained to predict multiple labels simultaneously in a multi-task learning setting (e.g., survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions). To determine the adequacy of the training, process 600 validates the classification model at box 614. Figure 7 The verification process is further described in the text.

[0076] exist Figure 6 In this study, a classification model was trained using 600 images for image classification. The quality of the classification model can be checked by validating it using the training image set as input and calculating one or more labels. Figure 7 This is a flowchart of one embodiment of the process for validating a classification model. Figure 7In this process, 700 begins by receiving a set of validation images at block 702. In one embodiment, the set of validation images is the same as the training set. In another embodiment, the set of validation images may be different from the training set. For example, and in one embodiment, a set of images already labeled for a specific type of image (e.g., histopathology of a disease) may have some images selected for training the model, and other images from that set used to validate the trained model. At block 704, 700 receives a model for classifying the set of validation images. In one embodiment, the model is a classification model, such as an MLP model and / or one or more other models described elsewhere.

[0077] Process 700 executes a processing loop (blocks 706-714) to generate an image label set for the verification image set. At block 708, process 700 extracts feature vectors for the images in the verification image set. In one embodiment, process 700 is as described above. Figure 3 The feature vector extraction described above. For example, and in one embodiment, process 700 uses a ResNet-50 convolutional neural network to determine the feature vector as described above. Figure 3 The feature vector of each patch of the image is segmented as described above. In one embodiment, the process generates a set of feature vectors for a set of verification images. Process 700 uses the set of feature vectors to generate a set of scores for the set of verification images at box 710. In one embodiment, process 700 generates a set of scores for the image by creating a score for each patch using a convolutional 1D layer, as described above. Figure 4 As described above. In this embodiment, process 700 selects a subset of images, where this subset of patches is used to generate patch scores. Furthermore, process 700 classifies each image using a trained classification model at box 712. In one embodiment, process 700 uses a multilayer perceptron (MLP) with two fully connected layers having 200 and 100 neurons with sigmoid activation to classify images in the validation image set. This processing loop ends at 716.

[0078] Using the classification of the validation image set, process 700 can compare the known image classifications in the validation image set with the generated classifications to determine the accuracy of one or more models for this type of image set. In one embodiment, any type of metric that compares differences or distances between labels can be used. For example, and in one embodiment, when the prediction task is a binary task, process 700 can use the area under the receiver operating characteristic curve (ROC-AUC) to determine the comparison between the known classifications and the generated classifications of images in the validation image set. In another embodiment, when the binary labels are imbalanced, process 700 can use the area under the precision-recall curve (PR-AUC) to determine the comparison between the known classifications and the generated classifications of images in the validation image set. In yet another embodiment, when predicting survival, process 700 can use the consistency index (c-index) to determine the comparison between the predicted risk ranking of data points and the known ranking.

[0079] In addition to the above Figure 1 In addition to predicting the image label as described, the preprocessing device 102 and / or classification device 108 can also extract the comparison target region of the image based on the image's tile score. In this embodiment, the comparison target region is different from... Figure 3 The area of ​​interest described herein is considered because there may be tiles that are similar (e.g., visually similar) to one or more tiles with the lowest or highest scores. For example, and in one embodiment, the comparison target area tile may be a tile with an average score (e.g., not the lowest or highest score among the calculated tile scores of the image) and similar to a tile in the highest or lowest score area. Figure 8 This is a flowchart of an embodiment of a process 800 for extracting and comparing target regions from an image. Figure 8 In this process, processing 800 begins by receiving an image and one or more models at box 802. In one embodiment, the image is an image that can be classified by one or more models (e.g., histopathological images, satellite images, etc.), and the one or more models are used to select comparison target regions. In this embodiment, the models may include models for segmenting the image, extracting feature vectors, and / or scoring the feature vectors, as described above. Figure 3 As described in [the document]. At box 804, image segmentation is processed 800 times. In one embodiment, processing 800 times is as described in [the document]. Figure 3 The image is segmented as described in box 304. At box 806, process 800 divides the image into blocks. In one embodiment, process 800 is as described in... Figure 3 The image is divided into blocks as described in block 306. At block 808, process 800 extracts the feature vector of the image. In one embodiment, process 800 is as described in... Figure 3 Feature vectors are extracted for each image patch as described in box 308. Process 800 uses the extracted feature vectors to score the image patches at box 810. In one embodiment, process 800 performs the process as described in... Figure 3 The image tiles are scored as described in box 310. This processing loop ends at box 812.

[0080] At box 814, process 800 uses at least tile scores to select image tiles. In one embodiment, the tile is used to determine a comparison target region. In one embodiment, process 800 selects tiles based at least on tile scores and the ranking of those tile scores. In this embodiment, process 800 can select high-number (R) tiles. top ) and / or low numbers (R) bottom The fraction of ). In one embodiment, R top and R bottom Represents the highest and lowest tile scores. In one embodiment, R top and / or R bottom The range of values ​​for R can be the same or different. Furthermore, R... top and / or R bottom The range can be a static numerical range (e.g., 10, 20, 100, or some other number) and / or value suitable for a range, percentage, label (e.g., small, large, or other label), and / or set via a user interface component (slider, user input, and / or another type of user interface component). Alternatively, processing 800 can select a set of one or more tiles that are above a first threshold of tile scores or below a second threshold of tile scores, where the first threshold can be the same or different. In another embodiment, processing 800 randomly selects one or more tiles based at least on probabilities derived from at least corresponding tile scores. In this embodiment, using random selection based on probabilities derived from at least corresponding tile scores allows for the selection of tiles with peripheral tile scores, meaning there is a higher probability of selecting tiles with higher or lower scores, rather than strictly selecting tiles with the top N tile scores or the bottom N tile scores.

[0081] Process 800 extracts the comparison target region at box 816. In one embodiment, process 800 uses selected(one or more) tiles from box 814 above to extract comparison target regions associated with selected(one or more) tiles having an average score and visually close to at least one of the selected tiles according to a distance metric. For example, and in one embodiment, the distance metric used to evaluate the visual proximity between two tiles, process 800, may be an L2 norm calculated on the extracted features of the two tiles and / or an L2 norm calculated on both tiles.

[0082] As described above, models for classification and tile selection do not require or use local annotations of the image produced by an expert (e.g., a pathologist). In another embodiment, if local annotations are present in the image, these local annotations can be used to improve the predictive use of one or more models. Thus, in one embodiment, when local annotations are available (such as the presence of a tumor in a slice region), these annotations can be taken into account using a fusion technique. To this end, the device can train the machine learning model for two concurrent tasks: (1) local prediction of the presence of macroscopic attributes (e.g., the presence of a tumor or other types of macroscopic attributes) on each tile and prediction of the global set of labels. The device (or multiple devices) can use a complex architecture that, on the one hand, involves the above Figure 3 The classification system described herein processes a set of 128 features. On the other hand, the device applies a convolutional neural network to transform the features of N patches into N*128 feature vectors. Based on this vector, the device trains the convolutional neural network to predict local annotation information for each patch, such as the presence of a tumor. The device can obtain both the predicted output and the N*128 feature vectors, and apply a weighted pooling operation to the concatenation of these two vectors to obtain 128 feature vectors for the input image. The device concatenates the output of the classification model with the obtained 128 features and attempts to predict a global set of labels for the image based on this vector (e.g., survival rate, tumor size, necrosis, and / or other types of predictions). The model's loss involves both global and local predictions. In this embodiment, the overall model performance can be improved by adding information derived from local annotations to the computational flow.

[0083] Figure 9 This is a block diagram of an embodiment of a system 900 for classifying images using a classification model and local annotations. Figure 9 In this system 900, both tile-level processing 932 and tile-level processing 930 can be used. In one embodiment, tile-level processing 932 uses local annotations to process tiles to improve the global label prediction of the entire system 900. In one embodiment, tile-level processing 932 processes as described above. Figure 3 The tiles and feature vectors described herein. Furthermore, the results of each of slice-level processing 930 and tile-level processing 932 can be combined and further processed to determine various predictions, as described below.

[0084] In one embodiment, system 900 receives an image having a set of N tiles and a corresponding set of feature vectors derived from the input image using tile partitioning and feature vector generation operations. For example, in one embodiment, system 900 receives the above... Figure 3The generated patch image and corresponding feature vector set are described above. While in one embodiment, system 900 receives N patches with a feature vector set having 256 features per feature vector, in an alternative embodiment, the feature vector set may include feature vectors with fewer or more features. System 900 further sends the feature vector set to slice-level 930 and patch-level 932 processing. In this embodiment, each feature vector in patch-level processing 932 is reduced from 256 features to 128 features. In one embodiment, the feature vectors are reduced by using a one-dimensional convolutional neural network 916. In this embodiment, the one-dimensional convolutional neural network 916 is used to extract the features of interest for each patch. The one-dimensional convolutional neural network 916 may be applied once, multiple times, or not at all. While in one embodiment, the number of features per feature vector is reduced by half, in alternative embodiments, the number of feature vectors may remain the same, be reduced by different amounts, and / or increased. In another embodiment, after applying a one-dimensional convolutional neural network 928 to extract the features of interest for each patch, system 900 sends the feature vector set to slice-level processing 930.

[0085] In one embodiment, tile-level processing 932 begins by receiving a set of feature vectors 908 generated by the aforementioned one-dimensional convolutional neural network 916. Tile-level processing 932 further trains a second one-dimensional convolutional neural network to the set of feature vectors 908 using local annotations. In this embodiment, the second one-dimensional convolutional neural network can then be used to predict the presence of macroscopic features 910 in each tile of an input image that does not include any local annotations (e.g., the presence of a tumor, other macroscopic medical features (e.g., inflammation, etc.), and / or the presence of other types of features in each tile). If some or all of the tiles have local annotations, then N predictions can be supplemented with or replaced with predictions derived from the available local annotations. For example, and in one embodiment, if a pathologist determines a partial or complete outline or identifier of a tumor on an image, then N predictions corresponding to N tiles can be derived from the indication or absence of a tumor in each image tile. If there are no local annotations, then this trained one-dimensional convolutional neural network can be used to create N predictions. These N predictions 910 can be combined with a set of convolutional feature vectors 908 to create a new feature vector 912 through weighted pooling of the N predictions and the set of convolutional feature vectors 908. In one embodiment, the new feature vector 912 has 128 features. In this embodiment, weighted pooling is used to weight the set of feature vectors 908 with N predictions 910. In alternative embodiments, the new feature vector 912 may have more or fewer features.

[0086] Meanwhile, slice-level processing 930 receives the original feature vector set 902 and performs the above process using dense layer 922. Figure 3 The feature vector processing described herein is used to generate feature set 906. While in one embodiment, feature vector set 906 has 128 features, in alternative embodiments, feature vector set 906 may include more or fewer features.

[0087] In one embodiment, the system has two distinct feature vectors, utilizing feature vector 912 generated by tile-level processing 930 and feature vector 906 generated by slice-level processing 930. The system can additionally combine these feature vectors by concatenating the two feature vectors into a resulting feature vector 914, which can be used to predict one or more global labels 928. For example, and in one embodiment, one or more global labels, such as survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions, can be predicted for a medical image slice. In one embodiment, by using feature vectors from both slice-level processing 930 and tile-level processing 932, the resulting feature vector 914 can have the influence of local annotations, which can then be used for prediction.

[0088] As mentioned above, if available, System 900 can use local annotations to further improve the predictive power of (one or more) classification models. Figure 10 This is a flowchart of an embodiment of image classification using a classification model and local annotations. Figure 10 In this process, processing 1000 begins by receiving an image, local annotations, and global labels at box 1002. In one embodiment, local annotations can be data and / or metadata capable of describing and / or indicating the presence or absence of macroscopic features in the input image. For example, and in one embodiment, local annotations can indicate that certain regions of the image have tumors or tumor-like tissue. At box 1004, processing 1000 divides the image into patches and generates a set of feature vectors. In one embodiment, processing 1000 generates a feature vector for each patch in the image. In this embodiment, each feature vector can include 256 features. For example, and in one embodiment, processing 1000 divides the image into patches and generates the features described above. Figure 3 The feature vectors described in [the document].

[0089] At this point, process 1000 can take two paths for slice-level and tile-level processing. For slice-level processing, at box 1006, process 1000 applies one or more classification models to generate a feature vector with 128 features. In one embodiment, process 1000 applies one or more classification models to, for example... Figure 3 The set of feature vectors described in box 308. Execution proceeds to the following box 1012.

[0090] Process 1000 begins performing tile-level processing at box 1008. At box 1008, process 1000 predicts a local label for each image tile. In one embodiment, process 1000 applies a one-dimensional convolutional neural network to extract a set of 256 feature vectors of interest. Furthermore, process 1000 can use local annotations to train a second convolutional neural network to generate a prediction set for each tile. At box 1010, process 1000 uses the local predictions as features for the model. At this point, process 1000 has generated feature vectors of 128 features from the tile-level processing in boxes 1008 and 1010. Execution proceeds to box 1012 below.

[0091] At box 1012, process 1000 combines features from boxes 1006 and 1010 to create a resulting feature vector of 256 features. In one embodiment, process 1000 creates the resulting feature vector by concatenating two feature vectors from boxes 1006 and 1010. Alternatively, alternative methods can be used to create the resulting feature vector. Furthermore, at box 1012, process 1000 uses the resulting feature vector to predict a global label. In one embodiment, process 1000 may predict different global labels (e.g., for medical image slices, survival or disease-free survival, clinical data, tumor size, vascular invasion, necrosis, and / or other types of predictions). In one embodiment, process 1000 is as described above. Figure 3 Global labels are predicted by scoring tiles and creating predictions, as described in boxes 310 and 312.

[0092] Figure 11 An example of a data processing system 1100 that can be used with an embodiment of the present invention is shown. For example, system 1100 can be implemented to include the features described above. Figure 1 The pretreatment device 102 and / or sorting device 108 shown are illustrated. Note that, although Figure 11 The illustration depicts various components of a computer system, but it is not intended to represent any particular architecture or way of interconnecting components, as such details are not closely related to the present invention. It should also be recognized that networked computers and other data processing systems or other consumer electronic devices with fewer or possibly more components can also be used with the present invention.

[0093] like Figure 11As shown, a computer system 1100, as a form of data processing system, includes a bus 1103 coupled to one or more microprocessors 1105 and ROM (Read-Only Memory) 1107, volatile RAM 1109, and non-volatile memory 1111. The microprocessor 1105 may include one or more CPUs, one or more GPUs, dedicated processors, and / or combinations thereof. The microprocessor 1105 can retrieve instructions from and execute instructions from the memories 1107, 1109, and 1111 to perform the aforementioned operations. The bus 1103 interconnects these various components and also interconnects these components 1105, 1107, 1109, and 1111 to a display controller and display device 9111, as well as peripheral devices such as input / output (I / O) devices, which may be mice, keyboards, modems, network interfaces, printers, and other devices well known in the art. Typically, input / output device 915 is coupled to the system via input / output controller 1113. Volatile RAM (Random Access Memory) 1109 is typically implemented as dynamic RAM (DRAM), which requires continuous power to refresh or maintain the data in the memory.

[0094] Mass storage devices 1111 are typically magnetic hard disk drives, magneto-optical drives, optical disc drives, DVDs, RAM, flash memory, or other types of memory systems that retain data (e.g., large amounts of data) even after power is lost from the system. Typically, mass storage devices 1111 will also be random access memory, but this is not required. Although Figure 11 The mass storage device 1111 is a local device directly coupled to the rest of the data processing system; however, it should be recognized that the invention can utilize non-volatile memory located off-system, such as network storage devices coupled to the data processing system via a network interface (such as a modem, Ethernet interface, or wireless network). Bus 1103 may include one or more buses interconnected with each other via various bridges, controllers, and / or adapters well known in the art.

[0095] The parts described above can be implemented using logic circuit systems such as dedicated logic circuits, or using microcontrollers or other forms of processing cores that execute program code instructions. Therefore, the processing taught by the above discussion can be executed using program code such as machine-executable instructions, which cause the machine executing those instructions to perform a specific function. In this context, "machine" can be a machine that translates intermediate (or "abstract") instructions into processor-specific instructions (e.g., abstract execution environments such as "virtual machines" (e.g., Java Virtual Machines), interpreters, Common Language Runtimes, high-level language virtual machines, etc.), and / or electronic circuit systems deployed on semiconductor chips designed to execute instructions (e.g., "logic circuit systems" implemented with transistors), such as general-purpose processors and / or dedicated processors. The processing taught by the above discussion can also be executed by electronic circuit systems (as a substitute for or in combination with a machine) designed to perform processing (or a portion thereof) without executing program code.

[0096] The present invention also relates to an apparatus for performing the operations described herein. This apparatus may be specifically constructed for a desired purpose, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memory (ROM), RAM, EPROM, EEPROM, magnetic cards or optical cards, or any type of medium suitable for storing electronic instructions, and each is coupled to a computer system bus.

[0097] Machine-readable media include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form. For example, machine-readable media include read-only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flash memory devices; and so on.

[0098] The article of manufacture can be used to store program code. The article of manufacture storing program code can be implemented as, but is not limited to, one or more memories (e.g., one or more flash memories, random access memory (static, dynamic, or other)), optical discs, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic cards or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code can also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by means of data signals implemented in the medium (e.g., via a communication link (e.g., a network connection)).

[0099] The foregoing detailed description is presented in light of the algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are tools used by those skilled in the art of data processing to most effectively communicate the essence of their work to others skilled in the art. Algorithms here and generally are considered to be self-consistent sequences of operations that lead to desired results. These operations are those that require physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. Sometimes, primarily for reasons of general use, it has proven convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0100] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise clearly stated from the above discussion, it should be understood that throughout the description, discussions using terms such as “segmentation,” “chunking,” “receiving,” “computing,” “extracting,” “processing,” “applying,” “enhancing,” “normalizing,” “pre-training,” “sorting,” “selecting,” “aggregating,” and “ranking” refer to the actions and processing of computer systems or similar electronic computing devices that manipulate and transform data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission, or display devices.

[0101] The processing and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used with the programs taught herein, or constructing more specialized devices to perform the described operations may prove convenient. The necessary architecture for various systems in these cases will be apparent from the following description. Furthermore, this invention is described without reference to any particular programming language. It should be recognized that the teachings of this invention as described herein can be implemented using a variety of programming languages.

[0102] The foregoing discussion describes only some exemplary embodiments of the invention. Those skilled in the art will readily recognize from this discussion, the accompanying drawings, and the claims that various modifications can be made without departing from the spirit and scope of the invention.

Claims

1. A method for classifying an input image, the method comprising: The input image is segmented into regions of interest and background regions that include information useful for classification by applying a first convolutional neural network. The region of interest is divided into a set of map tiles; For each tile, a feature vector for that tile is extracted by applying a second convolutional neural network, where the features of the feature vector represent local descriptors of the tile, and the score is associated with each tile in the set of tiles; as well as The extracted feature vectors from the tile set are processed to classify the input image based on a first subset of the tile set with the highest score and a second subset of the tile set with the lowest score.

2. The method according to claim 1, wherein the first convolutional network is a semantic segmentation neural network that classifies each pixel of the input image into one of a region of interest and a background region.

3. The method of claim 2, wherein the semantic segmentation neural network is selected from the group consisting of U-NET, fully convolutional networks for semantic segmentation, SegNet, and DeepLab.

4. The method according to claim 1, wherein the segmentation comprises: A fixed tiled grid is applied at least to the region of interest, wherein each tile in the set of tiles has a predetermined size.

5. The method of claim 1, wherein the scaling level is applied to the set of tiles.

6. The method of claim 1, wherein multiple scaling levels are applied to the tile set and tile sets with different scaling levels are combined.

7. The method of claim 1, wherein the segmentation further comprises: The tile set is enhanced by performing actions on the tile set, wherein the actions are at least one of rotating, translating, cropping, adding noise to the input image, modifying the intensity of one or more colors, or changing the contrast of the input image.

8. The method of claim 1, wherein the segmentation further comprises: The number of tiles in the tile set of each input image is normalized by performing random sampling of the tile set and filling at least one of the tile sets with blank tiles, such that the tile set of each input image includes a given number of tiles.

9. The method of claim 1, wherein the second convolutional neural network is a residual neural network selected from the group consisting of: ResNet-type residual neural networks, VGG neural networks, autoencoders for unsupervised feature extraction, and Inception neural networks.

10. The method according to claim 1, further comprising: The weights of a second convolutional neural network are pre-trained on at least one of another dataset and task, wherein at least one output layer is removed from the second convolutional neural network.

11. The method of claim 1, wherein the processing comprises: For each tile The score associated with the patch is calculated based on the extracted feature vector, wherein the score represents the patch's contribution to the classification of the input image; Sort the set of fractions; Based on at least one of the rank and value of the tiles in the sorted score set, select a first subset with the highest score and a second subset with the lowest score, and A classifier is applied to a subset of the scores to classify the image.

12. The method of claim 1, wherein the segmentation further comprises: Aggregate corresponding tile groups from different images in a plurality of images, wherein the plurality of images includes the input image.

13. The method of claim 1, wherein the segmentation further comprises: Clusters of tiles from a set of tiles within a given distance are aggregated based on a distance metric calculated in at least one of the input image or feature map.

14. The method of claim 13, wherein the clustering of aggregated tiles comprises at least one of the following: Tiles in the clustering of cascaded tiles; Using the clustering of tiles as a multidimensional object for feature vector extraction, select at least a single tile from the cluster of tiles based on a given criterion; Calculate the mean of the eigenvectors of the tiles in the tile cluster; and At least calculate the maximum or minimum value of the eigenvectors of the tiles in the tile cluster.

15. The method of claim 1, wherein the features Vector extraction also includes: An autoencoder is applied to the extracted feature vectors to reduce the dimensionality of the features.

16. The method of claim 1, wherein the input image is a histopathological section and the region of interest is a tissue region.

17. The method of claim 1, wherein image classification is used to perform at least one of diagnostic classification, survival prediction, and response to treatment prediction.

18. The method of claim 1, wherein classifying the input image includes predicting at least one type of global label and local label.

19. The method of claim 1, wherein processing of the extracted feature vectors includes using one or more local annotations associated with the input image, the one or more local annotations being incorporated as weights into a weighted combination of feature vectors derived from the extracted feature vectors that provide additional information for the classification of the input image.

20. The method of claim 1, wherein classifying the input image includes predicting multiple global labels in a multi-task environment.

21. A non-transitory machine-readable medium having executable instructions for causing one or more processing units to perform a method for classifying an input image, the method comprising: The input image is segmented into regions of interest and background regions that include information useful for classification by applying a first convolutional neural network. The region of interest is divided into a set of map tiles; For each tile, a feature vector for that tile is extracted by applying a second convolutional neural network, where the features of the feature vector represent local descriptors of the tile, and the score is associated with each tile in the set of tiles; as well as The extracted feature vectors from the tile set are processed to classify the input image based on a first subset of the tile set with the highest score and a second subset of the tile set with the lowest score.

22. The machine-readable medium of claim 21, wherein the segment comprises: A fixed tiled grid is applied at least to the region of interest, wherein each tile in the set of tiles has a predetermined size.

23. The machine-readable medium of claim 21, wherein the segment further comprises: The number of tiles in the tile set of each input image is normalized by performing random sampling of the tile set and filling at least one of the tile sets with blank tiles, such that the tile set of each input image includes a given number of tiles.

24. The machine-readable medium of claim 21, wherein the second convolutional neural network is a residual neural network selected from the group consisting of: ResNet-type residual neural networks, VGG neural networks, autoencoders for unsupervised feature extraction, and Inception neural networks.

25. The machine-readable medium of claim 21, further comprising: The weights of a second convolutional neural network are pre-trained on at least one of another dataset and task, wherein at least one output layer is removed from the second convolutional neural network.

26. The machine-readable medium of claim 21, wherein the process comprises: For each tile The score associated with the patch is calculated based on the extracted feature vector, where the score represents the patch's contribution to the classification of the input image; Sort the set of fractions; Based on at least one of the rank and value of the tiles in the sorted score set, select the first subset with the highest score and the second subset with the lowest score; as well as A classifier is applied to a subset of the scores to classify the input image.

27. The machine-readable medium of claim 21, wherein the segment further comprises: Aggregate corresponding tile groups from different images in a plurality of images, wherein the plurality of images includes the input image.

28. The machine-readable medium of claim 21, wherein the segment further comprises: Clusters of tiles from a set of tiles within a given distance are aggregated based on a distance metric calculated in at least one of the input image or feature map.

29. The machine-readable medium of claim 21, wherein the features Vector extraction also includes: An autoencoder is applied to the extracted feature vectors to reduce the dimensionality of the features.

30. The machine-readable medium of claim 21, wherein the input image is a histopathological section and the region of interest is a tissue region.

31. The machine-readable medium of claim 21, wherein image classification is used to perform at least one of diagnostic classification, survival prediction, and response to treatment prediction.

32. The machine-readable medium of claim 21, wherein the classification of the input image includes predicting at least one global label and a local label.

33. The machine-readable medium of claim 21, wherein processing of the extracted feature vector includes using one or more local annotations associated with the input image, the one or more local annotations being incorporated as weights into a weighted combination of feature vectors derived from the extracted feature vectors that provide additional information for the classification of the input image.

34. The machine-readable medium of claim 21, wherein the classification of the input image includes predicting multiple global labels in a multi-task environment.

35. A method for generating a classification model, the method comprising: Receive a set of training images, where each training image in the set has an associated known classification; For each training image in the training image set Extracting multiple feature vectors from a training image by applying a first convolutional neural network includes dividing the region of interest of the training image into a set of patches, wherein each of the multiple feature vectors corresponds to a patch from the set of patches, and a score is associated with each patch; as well as The classification model is trained using at least a first subset of the tile set with the highest scores and a second subset of the tile set with the lowest scores, along with associated known classifications.

36. The method of claim 35, further comprising: At least a set of validation images should be used to validate the classification model.

37. The method of claim 36, wherein the verification comprises: Receive a set of verification images, wherein each verification image in the set has an associated known classification; For each verification image in the verification image set Multiple feature vectors are extracted from the verification image by applying a first convolutional neural network, wherein each feature of the multiple feature vectors represents a local descriptor of the verification image. At least the plurality of feature vectors are used to generate a set of verification scores for the verification image, and At least a set of validation scores and a classification model should be used to generate a classification for the validation image; and The generated classifications are compared with their associated known classifications.

38. The method of claim 37, wherein at least one of the following is used to perform the comparison of the plurality of generated classifications: area under receiver operating characteristic curve (ROC-AUC) comparison, area under precision-recall curve (PR-AUC) comparison, or consistency index (c-index) comparison.

39. The method of claim 35, wherein the classification model is a multilayer perceptron with two fully connected layers.

40. The method of claim 35, wherein the images in the training set of the validation set are one of digitized whole slice images (WSI).

41. The method of claim 35, wherein the first convolutional neural network is a ResNet50 neural network.

42. The method of claim 35, wherein the segmentation comprises: A fixed tiled grid is applied at least to the region of interest, wherein each tile in the set of tiles has a predetermined size.

43. The method of claim 42, wherein training the classification model comprises: The score for a tile is calculated using at least a 1D convolutional layer and the corresponding feature vector of each tile in the tile set.

44. The method of claim 35, wherein for each training image in the training set of training images, the method further comprises: By applying a second convolutional neural network, the training image is segmented into regions of interest that include information useful for classification, as well as background regions.

45. The method of claim 44, wherein the second convolutional neural network is a U-NET neural network.

46. ​​The method of claim 35, wherein the classification model comprises one or more separate models.

47. The method of claim 35, wherein the classification model comprises at least one of a multilayer perceptron model and a one-dimensional convolutional neural network model.

48. A machine-readable medium having executable instructions that cause one or more processing units to execute a method for generating a classification model, the method comprising: Receive a set of training images, where each training image in the set has an associated known classification; For each training image in the training image set Extracting multiple feature vectors from a training image by applying a first convolutional neural network includes dividing the region of interest of the training image into a set of patches, wherein each of the multiple feature vectors corresponds to a patch from the set of patches, and a score is associated with each patch; as well as The classification model is trained using at least a first subset of the tile set with the highest scores and a second subset of the tile set with the lowest scores, along with associated known classifications.

49. The machine-readable medium of claim 48, further comprising: At least a set of validation images should be used to validate the classification model.

50. The machine-readable medium of claim 49, wherein the verification comprises: Receive a set of verification images, where each verification image in the set has an associated known classification; For each verification image in the verification image set Multiple feature vectors are extracted from the verification image by applying a first convolutional neural network, wherein each feature of the multiple feature vectors represents a local descriptor of the verification image. At least the plurality of feature vectors are used to generate a set of verification scores for the verification image, and At least a set of validation scores and a classification model should be used to generate a classification for the validation image; and The generated classifications are compared with their associated known classifications.

51. The machine-readable medium of claim 50, wherein at least one of the following is used to perform the comparison of the plurality of generated classifications: area under receiver operating characteristic curve (ROC-AUC) comparison, area under precision-recall curve (PR-AUC) comparison, or consistency index (c-index) comparison.

52. The machine-readable medium of claim 48, wherein the classification model is a multilayer perceptron with two connection layers.

53. The machine-readable medium of claim 48, wherein the training images in the validation set or training set are one of digitized whole-slice images (WSI).

54. The machine-readable medium of claim 48, wherein the first convolutional neural network is a ResNet50 neural network.

55. The machine-readable medium of claim 48, wherein the segment comprises: A fixed tiled grid is applied at least to the region of interest, wherein each tile in the set of tiles has a predetermined size.

56. The machine-readable medium of claim 48, wherein training the classification model comprises: The score for a tile is calculated using at least a 1D convolutional layer and the corresponding feature vector of each tile in the tile set.

57. The machine-readable medium of claim 48, wherein for each training image in the training set of training images, the method further comprises: By applying a second convolutional neural network, the training image is segmented into regions of interest that include information useful for classification, as well as background regions.

58. The machine-readable medium of claim 57, wherein the second convolutional neural network is a U-NET neural network.

59. The machine-readable medium of claim 48, wherein the classification model comprises one or more individual models.

60. The machine-readable medium of claim 48, wherein the classification model comprises at least one of a multilayer perceptron model and a one-dimensional convolutional neural network model.

Images