Cell classification using central enhancement of feature map

Abstract

Techniques described herein include, for example, generating a feature map for an input image, generating a plurality of concentric cuts of the feature map, and using the plurality of concentric cuts to generate an output vector representing features of a structure depicted in a central region of the input image. Generating the output vector may include, for example, aggregating output feature sets generated from the plurality of concentric cuts, and several aggregation methods are described. Applications of classifying structures depicted in the central region of the input image are also described.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 333,924, filed on April 22, 2022, the contents of which are hereby incorporated by reference in their entirety for all purposes. Background Art

[0003] Digital pathology may involve the interpretation of digitized images in order to correctly diagnose a subject's disease and guide treatment decision making. In a digital pathology solution, an image analysis workflow may be established to automatically detect or classify a biological object of interest (e.g., tumor cells that are positive or negative for a specific biomarker or other indicator, etc.). An exemplary digital pathology solution workflow includes: obtaining a slide of a tissue sample, scanning a pre-selected area of ​​a tissue slide or the entire tissue slide with a digital image scanner (e.g., a whole slide image (WSI) scanner) to obtain a digital image, and using one or more image analysis algorithms to perform image analysis on the digital image (e.g., to detect the object of interest). Such a workflow may also include quantifying the object of interest based on image analysis (e.g., counting or identifying an object-specific or cumulative area of ​​an object), and may further include quantitatively or semi-quantitatively scoring the sample based on the quantified results (e.g., positive, negative, moderate, weak, etc.). Summary of the invention

[0004] In various embodiments, a computer-implemented method for classifying an input image is provided, the method comprising: generating a feature map for the input image using a trained neural network comprising at least one convolutional layer; generating multiple concentric crops of the feature map; using information from each of the multiple concentric crops to generate an output vector representing features of a structure depicted in a central region of the input image; and determining a classification result by processing the output vector.

[0005] In some embodiments, generating an output vector using the plurality of concentric crops includes: for each of the plurality of concentric crops, generating a corresponding one of a plurality of feature vectors using at least one pooling operation; and generating the output vector using the plurality of feature vectors. In such methods, the plurality of feature vectors may be sorted according to radial dimensions of the corresponding concentric crops, and generating the output vector may include: independently convolving the trained filters on the sorted plurality of feature vectors of adjacent pairs.

[0006] In some embodiments, generating an output vector using the plurality of concentric crops includes: for each of the plurality of concentric crops, generating a corresponding one of a plurality of feature vectors using at least one pooling operation; and generating the output vector using the plurality of feature vectors. In such methods, the plurality of feature vectors may be sorted according to radial dimensions of the corresponding concentric crops, and generating the output vector may include: convolving the trained filter on the sorted plurality of feature vectors of a first adjacent pair to produce a first combined feature vector; and convolving the trained filter on the sorted plurality of feature vectors of a second adjacent pair to produce a second combined feature vector. In such methods, generating the output vector using the plurality of feature vectors may include: generating the output vector using the first combined feature vector and the second combined feature vector.

[0007] In various embodiments, a computer-implemented method for classifying an input image is provided, the method comprising: generating a feature map for the input image; generating a plurality of feature vectors using information from the feature map; generating a second plurality of feature vectors using a trained shared model, the model being independently applied to each of the plurality of feature vectors; generating an output vector using information from each of the second plurality of feature vectors, the output vector representing features of a structure depicted in the input image; and determining a classification result by processing the output vector.

[0008] In various embodiments, a computer-implemented method for training a classification model is provided, the classification model comprising a first neural network and a second neural network, the method comprising: generating a plurality of feature maps using the first neural network and information from images of a first data set; and training the second neural network using the plurality of feature maps. In the method, each image of the first data set depicts at least one biological cell, and the first neural network is pre-trained on a plurality of images of a second data set, the second data set comprising images that do not depict biological cells.

[0009] In various embodiments, a computer-implemented method for classifying an input image is provided, the method comprising: using a first trained neural network of a classification model to generate a feature map for the input image; using a second trained network of the classification model and information from the feature map to generate an output vector, the output vector representing features of a structure depicted in a central portion of the input image; and determining a classification result by processing the output vector. In the method, the input image depicts at least one biological cell, the first trained neural network is pre-trained on a first plurality of images, the first plurality of images including images that do not depict biological cells, and the second trained neural network is trained by providing the classification model with a second plurality of images depicting biological cells.

[0010] In some embodiments, a system is provided, comprising: one or more data processors and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium containing instructions, which when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0011] In some embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0012] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0013] The terms and expressions that have been adopted are used as terms of description and not of limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed through embodiments and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with one or more color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0015] Aspects and features of various embodiments will become more apparent by describing examples with reference to the accompanying drawings, in which:

[0016] Figure 1A An example of an image processing pipeline 100 is shown in accordance with some embodiments.

[0017] Figure 1BAn example of an input image is shown in which a candidate (a structure predicted to belong to a specific class) is in the center.

[0018] Figure 2A Another example of an input image is shown in which the candidate (indicated by the red circle) is in the center.

[0019] Figure 2B An example of an input image is shown in which the candidate (indicated by the red circle) is not in the center of the image.

[0020] Figure 2C An example of an input image is shown in which a candidate (indicated by a red circle) is surrounded by several structures of another class.

[0021] Figure 3 Six different examples of importance masks are shown.

[0022] Figure 4A A flow chart is presented for an exemplary process for classifying an input image in accordance with some embodiments.

[0023] Figure 4B A block diagram illustrating exemplary architectural components for classifying an input image in accordance with some embodiments is shown.

[0024] Figure 5A An example of actions according to some embodiments is shown, in which a feature map is generated for an input image.

[0025] Figure 5B An example of actions in accordance with some embodiments is shown, in which multiple concentric crops of a feature map are generated.

[0026] Figure 6 Another example of actions in accordance with some embodiments is shown, in which a feature map and multiple concentric crops of the feature map are generated for an input image.

[0027] Figure 7 An example of an action according to some embodiments is shown, in which a corresponding downsampling operation is performed on each of a plurality of concentric crops to produce a corresponding downsampled feature map, a combination operation is performed on the downsampled feature maps to produce an output vector, and the output vector is processed to produce a classification result.

[0028] Figure 8 An example of a structure of four categories is shown.

[0029] Fig. 9 An example of a feature map, a corresponding plurality of concentric crops, and a corresponding plurality of feature vectors is shown in accordance with some embodiments.

[0030] Fig.10 shows that according to some embodiments further includes an output vector Fig. 9 .

[0031] Fig.11A A flow chart is presented for another exemplary process for classifying an input image in accordance with some embodiments.

[0032] Fig. 11B A block diagram illustrating another exemplary architectural components for classifying an input image in accordance with some embodiments.

[0033] Fig.12 shows a method further comprising sharing a model and an output vector according to some embodiments Fig. 9 .

[0034] Fig.13 shows further including convolution on the radius and outputting the vector according to some embodiments Fig. 9 .

[0035] Fig.14A An example of actions according to some embodiments is shown, in which multiple concentric crops of a feature map are used to generate an output vector.

[0036] Fig. 14B A block diagram illustrating a specific implementation of an output vector generation module according to some embodiments.

[0037] Fig.15A A block diagram illustrating a specific implementation of a merge module according to some embodiments.

[0038] Fig. 15B A block diagram illustrating a specific implementation of an additional feature vector generation module according to some embodiments.

[0039] Fig.16A A block diagram illustrating a specific implementation of architectural components according to some embodiments.

[0040] Fig. 16B A block diagram illustrating a specific implementation of a feature vector generation module according to some embodiments.

[0041] Fig.17A A flowchart is presented for an exemplary process for training a classification model including a first neural network and a second neural network, according to some embodiments.

[0042] Fig. 17B A flow chart is presented for an exemplary process for classifying an input image in accordance with some embodiments.

[0043] Fig.18AA flow chart is presented for an exemplary process for classifying an input image in accordance with some embodiments.

[0044] Fig.18B A flow chart is presented for an exemplary process for classifying an input image in accordance with some embodiments.

[0045] Fig.19A A schematic diagram of an encoder-decoder network according to some embodiments is shown.

[0046] Fig.19B An example of actions according to some embodiments is shown, in which a trained encoder is used to generate a feature map for an input image.

[0047] Fig. 20 Examples of input images and corresponding reconstructed images as produced by an encoder-decoder network (also referred to as an “autoencoder” network) at different training epochs are shown in accordance with some embodiments.

[0048] Fig.21 Examples of input images and corresponding reconstructed images as produced by various encoder-decoder networks are shown in accordance with some embodiments.

[0049] Fig.22A An example of actions in accordance with some embodiments is shown, in which a feature map for an input image is generated using a trained encoder and a trained neural network.

[0050] Fig. 22B An example of actions according to some embodiments is shown, in which a feature map for an input image is generated using multiple feature maps and a combining operation.

[0051] Fig.23 An example of actions according to some embodiments is shown, in which multiple concentric crops of a feature map are used to generate an output vector.

[0052] Fig.24 An example of actions according to some embodiments is shown, in which a feature map is used to generate an output vector.

[0053] Fig.25A An example of actions according to some embodiments is shown, in which a feature map for an input image is generated using a trained neural network.

[0054] Fig.25B An example of actions according to some embodiments is shown, in which a feature map for an input image is generated using multiple feature maps and a combining operation.

[0055] Fig.26An example of actions according to some embodiments is shown, in which multiple concentric crops of a feature map are used to generate an output vector.

[0056] Fig. 27 An example of actions according to some embodiments is shown, in which a feature map is generated for an input image.

[0057] Fig.28 An example of actions according to some embodiments is shown, in which multiple concentric crops of a feature map are used to generate an output vector.

[0058] Fig.29 An example of actions according to some embodiments is shown, in which a feature map is generated for an input image.

[0059] Fig.30 An example of actions according to some embodiments is shown, in which multiple concentric crops of a feature map are used to generate an output vector.

[0060] Fig.31 An example of a computing system is shown that may be configured to perform the methods described herein, according to some embodiments.

[0061] Fig.32 Shown are examples of precision-recall curves for a model as described herein, according to some embodiments.

[0062] In the drawings, similar components and / or features may have the same reference numeral. In addition, various components of the same type may be distinguished by following the reference numeral with a dash and a second numeral that distinguishes the similar components. If only the first reference numeral is used in the specification, the description applies to any similar part having the same first reference numeral, regardless of the second reference numeral. DETAILED DESCRIPTION

[0063] Techniques described herein include, for example, generating a feature map for an input image, generating multiple concentric crops of the feature map, and using the multiple concentric crops to generate an output vector representing features of a structure depicted in a central region of the input image. Generating the output vector may include, for example, aggregating a set of output features generated from the multiple concentric crops, and several aggregation methods are described. For example, the set of output features generated from the multiple concentric crops may be aggregated using a "radius convolution," which is defined as convolving a feature map or vector derived from concentric crops of the feature map at multiple radii, and / or convolving each of a plurality of concentric crops of the feature map at different radii along a corresponding feature map or vector derived from the crop. Applications for classifying structures depicted in a central region of an input image are also described.

[0064] One or more such techniques may be applied to, for example, convolutional neural networks capable of image classification applications. Technical advantages of such techniques may include, for example, improved image processing (e.g., higher true positive / negative / detection, lower false positive / negative), improved prognostic assessment, improved ease of diagnosis, and / or improved treatment recommendations. Exemplary use cases presented herein include mitosis detection and classification from images of stained tissue samples (e.g., tissue samples stained with hematoxylin and eosin (H&E)). 1. Background Technology

[0066] A fixative (e.g., a liquid fixative, such as a formaldehyde solution) and / or an embedding material (e.g., a histological wax such as paraffin and / or one or more resins such as styrene or polyethylene) can be used to fix and / or embed tissue samples (e.g., tumor samples). Fixed tissue samples can also be dehydrated before embedding (e.g., via exposure to an ethanol solution and / or a clarified intermediate agent). When the embedding material is in a liquid state (e.g., when heated), the embedding material can infiltrate the tissue sample. The fixed, dehydrated and / or embedded tissue sample can be sliced ​​to obtain a series of slices, wherein each slice has a thickness of, for example, 4 microns to 5 microns. Such slices can be performed by first cooling the sample and slicing the sample in a warm water bath. Before the slices are stained and mounted on a glass slide (e.g., as described below), the slices can be deparaffinized (e.g., using xylene) and / or rehydrated (e.g., using ethanol and water).

[0067] Because tissue sections and cells therein are actually transparent, the preparation of sections generally includes staining the tissue sections (e.g., automatically) to make the relevant structures more visible. For example, different sections of the tissue can be stained with one or more different stains to express different characteristics of the tissue. For example, each section can be exposed to a predefined volume of stain for a predefined period of time. When the sections are exposed to multiple stains, they can be exposed to multiple stains simultaneously.

[0068] Each section can be mounted on a slide and then scanned to create a digital image that can then be reviewed by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). The pathologist can review and manually annotate the digital image of the slide (e.g., tumor area, necrosis, etc.) so that meaningful quantitative measurements can be extracted using image analysis algorithms (e.g., to detect and classify biological objects of interest). Traditionally, a pathologist can manually annotate each successive image of multiple tissue sections from a tissue sample to identify the same aspects on each successive tissue section.

[0069] One type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical staining can be used to indicate general aspects of tissue morphology and / or cell histology (e.g., to distinguish between nuclei and cytoplasm, indicate lipid droplets, etc.). An example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome staining), periodic acid-Schiff (PAS), silver stains, and iron stains. Another type of tissue staining is immunohistochemistry (IHC, also referred to as "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also referred to as a biomarker). IHC can be direct or indirect. In direct IHC, a primary antibody is directly conjugated to a marker (e.g., a chromophore or a fluorophore). In indirect IHC, a primary antibody first binds to a target antigen, and then a secondary antibody conjugated to a marker (e.g., a chromophore or a fluorophore) binds to the primary antibody.

[0070] Mitosis is a stage in the life cycle of biological cells, in which replicated chromosomes separate to form two new nuclei, and "mitotic figures" are cells undergoing mitosis. The prevalence of mitotic figures in tissue samples can be used as an indicator of disease prognosis, particularly in oncology. For example, tumors that exhibit a high mitotic rate tend to have a lower prognosis than tumors that exhibit a low mitotic rate, wherein the mitotic rate can be defined as the number of mitotic figures per a fixed number (e.g., one hundred) of tumor cells. Such indicators can be applied to, for example, various solid and hematological malignancies. It may be highly labor-intensive for pathologists to examine slide images (e.g., of H&E-stained sections) to determine the mitotic rate.

[0071] 2. Introduction

[0072] An image classification task may be configured as a process of classifying structures depicted in a central region of an image. Such a task may be an element of a larger process, such as a process of quantifying the number of structures of a particular class depicted in a larger image (e.g., a WSI). Figure 1AAn example of an image processing pipeline 100 is shown in which a large image (in this example, a WSI or a block of a WSI) that is expected to contain multiple depictions of a structure of a particular class is provided as an input to a detector 110 (e.g., a trained neural network, such as a CNN). The detector 110 processes the large image to find locations of structures (which are referred to as "candidates") that are similar to the structures of the desired class. The predicted likelihood that the candidate actually belongs to the desired class can be indicated by a detection score. Next, candidates with detection scores above a predefined threshold are processed by a classifier 120 (e.g., including an architectural component as described herein) to further predict whether each candidate actually belongs to the desired class, or instead belongs to one of one or more other classes (e.g., even if it may have a high degree of similarity to the structure of the desired class). In such applications, the input to the classifier 120 can be image crops, each centered on a different one of the candidates identified by the detector 110.

[0073] The deep learning architecture components (and their variants) as described herein can be used at the end of a classification model: for example, as the final stage of the classifier 120 of the pipeline 100. For example, such components can be applied in a classifier above a feature extraction backend, although their potential and disclosed uses are not limited to only such applications. Applying such components can improve the quality of, for example, cell classification (e.g., compared to a standard, flat, fully connected end of a transfer learning-based neural network).

[0074] The architecture components as disclosed herein may be particularly advantageous for applications in which the center of the object to be classified (e.g., an individual cell) is exactly or approximately within the input image provided for classification (e.g., Figure 1A Such objects may vary in some size range that can be roughly characterized by various radii.

[0075] The architecture components as disclosed herein may also be implemented to aggregate the output feature sets generated from various radius clippings. Such aggregation may cause the neural network of the architecture component to weight information from the center of the input image more and gradually reduce the influence of the surrounding regions as the distance of the surrounding regions from the center of the image increases. During training, the neural network may learn to aggregate the output features according to the most beneficial distribution of importance influences.

[0076] Techniques described herein may include, for example, applying the same set of convolutional neural network operations on a series of co-centered (e.g., concentric) crops of a 2D feature map having different radii from the center of the feature map. By extension, techniques described herein also include techniques for applying the same set of convolutional neural network operations on a series of co-centered (e.g., concentric) crops of a 3D feature map (e.g., as generated from a volumetric image, such as may be produced by a PET or MRI scan) having different radii from the center of the feature map.

[0077] For example, the architectural components described herein may be applied as a mitotic image detection and classification pipeline (e.g., which may be implemented as Figure 1A 100 ). The detector may be configured to detect the locations of cells predicted to be mitotic figures ("candidates"). Next, candidates having detection scores above a predefined threshold may be processed by a classifier (e.g., including architectural components as described herein) to further predict whether a given cell is indeed a mitotic figure, or instead belongs to one of one or more other cell categories (e.g., including one or more categories that may share a high degree of similarity with mitotic figures). In this application, the input to the classifier may be an image crop centered on a candidate identified by the detector. Figure 1B An example of such an input image is shown, in which the candidate (in this case, a mitotic figure) is in the center and indicated by a bounding box, and similar structures of different classes are also indicated by bounding boxes. Figure 2A Another example of an input image is shown (eg, as produced by a mitotic image detector as described above) in which the candidate (in this case, the mitotic image indicated by the red circle) is in the center.

[0078] It may be desirable to exploit this property of the input image, ie, the region of interest (ROI) is centered within the image, by training the neural network with greater emphasis on analyzing cells located in the center of the image crop (as opposed to analyzing neighboring cells).

[0079] Information such as the size of the central cell and / or the relationship of the central cell to its neighborhood may strongly influence the classification. Solutions in which the input image is obtained by simply extracting the bounding boxes of the detected cells and rescaling them to a constant size may not include such information and such solutions may not be sufficient to produce optimal results.

[0080] Rather than focusing on detected cells by using a bounding box with a size that may vary between cells depending on the size of the cell, it may be desirable to acquire crops around the detected cells with the same resolution and size for each detected cell. Such image crops contain the neighborhood of the cell, which may contain context information useful for cell assessment. Furthermore, the constant resolution of these crops between cells (e.g., without the need for rescaling, which may be desirable for bounding boxes of different sizes) enables the size information of the cells to be inferred.

[0081] Analysis of misclassified test samples can help identify situations that may have caused the error. For example, it was found that test samples for which the cell ground truth annotation was slightly off-center were more likely to be misclassified, especially in situations where another cell was relatively closer to the center of the input image when compared to the mitosis ground truth. Figure 2B An example of an input image is shown in which the mitotic figure (indicated by the red circle) is not in the center of the image.

[0082] This problem may occur in real scenarios where, for example, the detected candidates are slightly off-center. One solution to this problem is to apply a certain degree of proportional random position shift during training, which can help expand the model's attention to include cells in some nearby areas of the center, rather than being limited to cells exactly at the center.

[0083] However, such solutions may be difficult to calibrate because random position shifts may cause new types of misclassifications. One such result may be that the model does not always focus on the center-most cell. For example, when a mitotic figure is surrounded by several tumor cells, it is often misclassified as a tumor cell. Such misclassifications may occur because the characteristics of neighboring cells have a greater impact on the final output despite the fact that the target cell to be classified is actually closer to the center. Figure 2C An example of an input image is shown in which a mitotic figure (indicated by a red circle) is surrounded by several tumor cells.

[0084] Therefore, a more reliable way may be desired: configure the neural network to assign greater weights to cells closest to the center of the input image. One solution is to prepare a hard-coded mask of importance (with values ​​in the range of, for example, 0 to 1 as weights), and multiply the relevant information (the input image, or some in the feature map) by such a mask (e.g., pixel by pixel). The relevant information may be the input image or one or more intermediate feature maps generated from the input image by the neural network.

[0085] Figure 3Six different examples of importance masks are shown. As a result of multiplication with such masks, non-homogeneous convolution outputs are introduced, causing the neural network to take into account the distance from the center and make inferences from it.

[0086] However, using an importance mask may not be the best way to cause a neural network to assign greater importance weights to the center of an input image. The input image may depict cells of various sizes, and the quality of making center detections may vary between different detector networks and may therefore depend on the specific detector network used. A hard-coded mask selection may not be universally applicable in all cases, and a self-calibration mechanism may be desired instead.

[0087] Another problem with using importance masks is that directly multiplying the input image by the mask may cause loss of information about the neighborhood and / or loss of homogeneity on the input to very early convolutional layers. Early convolutional layers tend to extract very basic features of similar image elements in the same way, regardless of their location within the image. Early multiplication by the mask may reduce the similarity between such image elements, which may be detrimental to pattern recognition of neighborhood topology.

[0088] A baseline approach for transfer learning is to add a classic global average pooling after a feature extraction layer from a model that has been pre-trained on a large dataset (e.g., ImageNet), followed by one or two fully connected layers and softmax activations. However, global average pooling can have the problem that it treats information from every region of the image as equally important, and therefore does not exploit prior knowledge that may be available in the context (e.g., target objects are mostly near the center of the image). The method described herein includes modifying the classic transfer learning method with a customization that exploits the centralized location of the most important objects.

[0089] Techniques as described herein (e.g., techniques using radius convolutions, etc.) can be used to implement an architecture component in which the component's neural network self-calibrates the importance distribution between regions at various distances from the center of the image. Compared to methods that multiply the input image by a fixed importance mask, the techniques as described herein can avoid the loss of important information about cell neighborhoods. In addition, the techniques as described herein can allow low-level convolutions to work on different (possibly all) regions of the input image in the same way: for example, by applying heterogeneity based on radius only in the later stages of the neural network. Experimental results show that applying techniques as described herein (e.g., techniques including performing radius convolutions on feature maps extracted through the EfficientNet backbone) can help the neural network achieve higher accuracy than the baseline.

[0090] 3. Use the concentric cutting technique

[0091] As described above, the process of cropping the input image down to the target object, and then scaling the crop up to the input size of the classifier may result in loss of information about the size and neighborhood of the target. In methods described in more detail below, constant resolution crops may be generated such that the target object is near the center of the image (e.g., at least partially represented in each crop) and also includes the surrounding neighborhood of the object. Such methods may be implemented to utilize both information about the size of the target object and information about the relationship of the target object to its neighborhood.

[0092] Figure 4A A flowchart for an exemplary process 400 for classifying an input image is shown. Process 400 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Figure 4A , at box 404, a feature map for an input image is generated using a trained neural network including at least one convolutional layer. At box 408, multiple concentric crops of the feature map are generated. For each of the multiple concentric crops, the center of the feature map may coincide with the center of the crop. At box 412, information from each of the multiple concentric crops is used to generate an output vector, which represents the characteristics of the structure depicted in the central region of the input image. The structure depicted in the central region of the input image can be, for example, a structure to be classified. The structure depicted in the central region of the input image can be, for example, a biological cell. At box 416, a classification result is determined by processing the output vector. For example, the classification result can predict that the input image depicts a mitotic figure.

[0093] Figure 4B A block diagram of an exemplary architecture component 405 for classifying an input image is shown. Component 405 may be implemented using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.28 Reference Figure 4B , a trained neural network 420 including at least one convolutional layer is used to receive an input image and generate a feature map for the input image. A cropping module 430 is used to receive the feature map and generate multiple concentric crops of the feature map. For each of the multiple concentric crops, the center of the feature map may coincide with the center of the crop. An output vector generation module 440 is used to receive the multiple concentric crops and use the multiple concentric crops to generate an output vector, which represents the characteristics of the structure depicted in the central area of ​​the input image. The structure depicted in the central area of ​​the input image can be, for example, a structure to be classified. The structure depicted in the central area of ​​the input image can be, for example, a biological cell. A classification module 450 is used to determine a classification result by processing the output vector. For example, the classification result can predict that the input image depicts a mitotic figure.

[0094] Figure 5AAn example of the actions of box 404 is shown, in which a feature map 210 (e.g., a 2D feature map) for an input image 200 is generated (e.g., extracted from the input image 200) using a trained neural network 420 including at least one convolutional layer. The trained neural network 420 can be, for example, a pre-trained backbone of a deep convolutional neural network (CNN). Specific examples of such backbones that can be used include, for example, a residual network (ResNet), a specific implementation of a MobileNet model, or a specific implementation of an EfficientNet model (e.g., any model in the range of models from EfficientNet-B0 to EfficientNet-B7), but any other neural network backbone configured to extract image feature maps (whether based on, for example, another known model or even a custom model) can be used. The backbone (or backend) of a CNN is defined as the feature extraction portion of the network. The backbone can include all of the CNN except the final layers (e.g., fully connected layers and activation layers), or the backbone can also not include other layers of one or more final stages of the network (e.g., global average pooling). In one example, feature map 210 is the last 2D feature map generated by network 420 before global average pooling.

[0095] The trained neural network 420 can be trained on a large image dataset (e.g., more than ten thousand, more than one hundred thousand, more than one million, or more than ten million). The large image dataset may include images depicting non-biological structures (e.g., cars, buildings, manufactured objects). In addition or alternatively, the large image dataset may include images that do not depict biological cells (e.g., images depicting animals). The ImageNet project (https: / / www.image-net.org) is an example of a large image dataset that includes images depicting non-biological structures and images that do not depict biological cells. (Image datasets that include images that do not depict biological cells are also referred to herein as "generic datasets.") Semi-supervised learning techniques such as Noisy Student training can be used to increase the size of the dataset used to train and / or validate the network 420 by learning labels for previously unlabeled images and adding them to the dataset. Optional further training (e.g., fine-tuning) of the network 420 within component 405 is also contemplated.

[0096] Training of the detector and / or classifier model (e.g., training of network 420) may include augmenting the training image set. Such augmentation may include random changes in color (e.g., rotation of hue angle), size, shape, contrast, brightness, and / or spatial translation (e.g., to increase the robustness of the trained model to off-center) and / or rotation. In one example of such random augmentation, the images of the training set for the classifier network may be slightly proportionally shifted in position along the x and / or y axis (e.g., up to ten percent of the image size along the translation axis). Such augmentation may provide better adaptation to slight imperfections of the detector network, as these imperfections may cause detected cell candidates to be slightly off-center in the input image provided to the classifier. Training of the classifier model may include pre-training the backbone neural network of the model (e.g., on a dataset of general images (such as a dataset including such images that depict objects that will not be seen during inference)), then freezing the initial layers of the model while continuing to train the final layers (e.g., on a dataset of specialized images (such as a dataset of such images that depict objects of one or more categories to be classified during inference)).

[0097] The input image 200 may be generated by a detector stage (e.g., another trained neural network) and may be, for example, a selected portion of a WSI or a selected portion of a block of a WSI. The input image 200 may be configured such that the center of the input image is within the region of interest. For example, the input image 200 may depict a biological cell of interest and may be configured such that the depiction of the cell of interest is centered within the input image.

[0098] exist Figure 5A In the example of , the input image 200 has a size of (ixj) pixels (arranged in i rows and j columns) by k channels, and the feature map 210 has a size of (mxn) spatial elements (arranged in m rows and n columns) by p channels (e.g., features). Exemplary values ​​for i and j may include, for example, 128 or 256. For example, the input image 200 may be a 128x 128 or 256x256 pixel block of a larger image (e.g., as produced by a detector network as described above). Exemplary values ​​for k may include three (e.g., for images in a three-channel color space such as RGB, YCbCr, etc.). Exemplary values ​​for m and n may include, for example, any integer from 7 to 33, and the value of m may be, but is not necessarily, equal to the value of n. Exemplary values ​​for p may include, for example, an integer in the range of 10 to 200 (e.g., in the range of 30 to 100).

[0099] Figure 5BAn example of the actions of block 408 is shown in which a cropping operation (in this example, as performed by cropping module 430) is used to generate the plurality of concentric crops 220 of feature map 210. It may be desirable to configure the cropping operation so that each of the plurality of concentric crops 220 is concentric in a spatial dimension (e.g., Figure 5B The center of the feature map 210 in the spatial dimension (e.g., Figure 5A and Figure 5B In the example of , the centers of the dimensions m and n coincide. For example, for an N x N feature map of square shape cropped at the center, any one or more of the following square sizes of cropping may be obtained: N x N, (N-2)x(N-2), (N-4)x(N-4).... or even 2x 2 or 1x 1 (depending on whether N is even or odd). Figure 5B In the specific example shown, m is equal to n and is an even number (e.g., divisible by two), the number of the plurality of concentric cuts 220 is two, and the dimensions of the concentric cuts are (2x2)xp and (4x4)xp. In other examples, m and / or n can be odd numbers, and / or the number of the plurality of concentric cuts 220 can be three, four, five, or more.

[0100] Figure 6 An example of the actions of block 404 as applied to a configuration in which the trained neural network 420 of component 405 is implemented as an instance 422 of an EfficientNet-B2 backbone; the input image has dimensions of (260 x 260) pixels (perhaps rescaled from another dimension such as 128 x 128 or 256 x 256); and the feature map 210 has dimensions of (9 x 9) spatial elements xp features. Feature levels P1 to P5 and the corresponding feature resolution at each level are also shown. Figure 6 Also shown is an example of the actions of block 408, in which an implementation 432 of the cropping module 430 of component 405 performs a 2D discarding operation on the feature map 210 before cropping the resulting feature map to produce the plurality of concentric crops 220 (in this example, crops having spatial dimensions (5 x 5) and (3 x 3)). It should be understood that an equivalent result may be obtained by performing such a 2D discarding operation within the network 422 instead (e.g., performing a 2D discarding operation on the output of level P5 of the network 422 to produce the feature map 210).

[0101] Figure 7An example of the actions of block 412 as applied to such a configuration is shown in which a specific implementation 442 of the output vector generation module 440 of component 405 includes downsampling modules 4420a, b and a merging module 4424. Each downsampling module 4420a, b is applied to the plurality of concentric crops 220 (in this example, Figure 6 4420a, b) to produce a downsampled feature map, and a merging module 4424 merges the downsampled feature maps to produce an output vector 240. In this particular example, the downsampled feature maps are feature vectors 230a, b, and it may be desirable to configure the downsampling modules 4420a, b so that the dimensions of each feature vector 230a, b are (1x 1) xp. Each of the downsampling modules 4420a, b can downsample the corresponding concentric crop by performing, for example, a global pooling operation (e.g., a global average pooling operation or a global maximum pooling operation). Examples of merging modules 4424 are discussed in further detail below.

[0102] Any set of operations may follow the generation of the output vector 240: for example, any number of layers or even just one fully-connected layer, to derive the final prediction score. Figure 7 Also shown is an example of the actions of block 416, in which the classification module 450 of component 405 processes the output vector 240 to determine the classification result 250. The classification module 450 may process the output vector 240 using, for example, a sigmoid or softmax activation function. In a specific application for mitosis classification, the following four categories may be used: granulocytes, mitotic figures, similar cells (non-mitotic cells that resemble mitotic figures), and tumor cells (e.g., Figure 8 shown).

[0103] Fig. 9 A specific example of application of process 400 is shown, in which the size of feature map 210 is (9x 9)x3; the number of the plurality of concentric crops 220 is five; the sizes of the plurality of concentric crops 220 are (9x 9)xp, (7x 7)xp, (5x 5)xp, (3x 3)xp, and (1x 1)xp; and each of the feature vectors 230 is generated by performing global average pooling on a corresponding one of the plurality of concentric crops 220 (e.g., each of the downsampling modules 4420 performs global average pooling). Alternatively, one or more (possibly all) of the feature vectors 230 may be generated by performing global max pooling on a corresponding one of the plurality of concentric crops 220 (e.g., one or more (possibly all) of the downsampling modules 4420 perform global max pooling). In a further alternative, both types of pooling may be performed, in which case the resulting pairs of feature vectors may be concatenated.

[0104] Merging module 4424 can be implemented as aggregating feature vector instance sets generated from concentric cropping (e.g., by pooling or other downsampling operations). An example of aggregation is a specific implementation of weighted sum or weighted average. Such aggregation can be achieved by multiplying each output feature vector by its individual weight, and then adding the weighted feature vectors together. The result is a weighted sum. Further dividing it by the sum of the weights yields a weighted average, but the division step is optional, because appropriate weights can be learned by the training process anyway, so that similar practical functions can be achieved.

[0105] Such an aggregate solution can be achieved, for example, by assigning weight vectors that participate in training. The trained weights can learn the optimal distribution of the importance of feature vectors that characterize regions of different radii. Fig.10 In the example of , the vector with values ​​annotated as A, B, C, D, E is a vector of importance weights.

[0106] Fig.10 Shows Fig. 9 , in which the output vector 240 is calculated (e.g., in block 412 or by a merging module 4424) as a weighted sum (or weighted average) of the feature vectors 230. In this example, each of the five feature vectors 230 is weighted with a corresponding one of the weights A, B, C, D, and E, which may be trained parameters.

[0107] Aggregation of feature vectors generated from feature maps (e.g., feature vectors generated from multiple concentric crops as described above) may include: applying a trained model (also referred to as a "shared model", "shared weight model", or "shared visual model") independently to each of the feature vectors to be aggregated. For example, for each of multiple concentric crops with feature maps at different radii, a trained visual model may be applied along the corresponding feature map or vector derived from the crop as described above, which may be described as an example of a "radius convolution". Here, a "shared model" may be implemented as a solution in which the same set of neural network layers with exactly the same weights are applied to different inputs (e.g., as in a "shared visual model" for processing different input images in Siamese and triplet neural networks). For example, a trained shared weight model may apply the same set of equations on the corresponding feature map or vector derived from the crop for each of the multiple concentric crops. The layers in a shared visual model may differ from each other in their number, shape, and / or other details.

[0108] Fig.11AA flowchart for another exemplary process 1100 for classifying an input image is shown. Process 1100 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Fig.11A , process 1100 includes an instance of block 404 as described herein. At block 1108, a plurality of feature vectors are generated using information from the feature map. For example, a plurality of concentric cuts of the feature map may be generated at block 1108 (e.g., as described above with reference to block 408), and for each of the plurality of concentric cuts generated at block 1108, the center of the feature map may coincide with the center of the cut. At block 1110, a second plurality of feature vectors are generated using a trained model that is independently applied to each of the plurality of feature vectors generated at block 1108. At block 1112, an output vector is generated using information from each of the second plurality of feature vectors that represents a feature of a structure depicted in an input image (e.g., in a central region of the input image). The structure depicted in the input image may be, for example, a structure to be classified. The structure depicted in the input image may be, for example, a biological cell. At block 1116, a classification result is determined by processing the output vector (e.g., as described above with reference to block 416). For example, the classification result may predict that the input image depicts a mitotic figure.

[0109] Fig. 11B A block diagram of another exemplary architecture component 1105 for classifying an input image is shown. Component 1105 may be implemented using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Component 1105 may be implemented as a neural network including two sub-networks (a first neural network that is an instance of trained neural network 420, and a second neural network that is a trained shared model as described herein), wherein the second neural network is configured to: process input based on the output latent space vector (feature map) generated by the first neural network. Fig. 11B, component 405 includes an instance of a trained neural network 420 as described herein (e.g., backbone 422) for generating a feature map. A feature vector generation module 1130 is used to generate a plurality of feature vectors using the feature map. For example, the feature vector generation module 1130 can be used to generate a plurality of concentric cuts of the feature map (e.g., as an instance of the cut module 430 as described above), and for each of the plurality of concentric cuts generated by the feature vector generation module 1130, the center of the feature map can coincide with the center of the cut. A second feature vector generation module 1135 is used to generate a second plurality of feature vectors using a trained model (e.g., a second trained neural network), which is independently applied to each of the plurality of feature vectors generated by module 1130. An output vector generation module 1140 is used to generate an output vector using the second plurality of feature vectors, which output vector represents a feature of a structure depicted in an input image (e.g., in a central region of the input image). The structure depicted in the input image can be, for example, a structure to be classified. The structure depicted in the input image can be, for example, a biological cell. The classification module 1150 is used to determine a classification result by processing the output vector. For example, the classification result may predict that the input image depicts a mitotic figure.

[0110] Fig.12 Shows Fig. 9 , in which the shared model is applied individually to each of the plurality of feature vectors 230 (e.g., at block 1112 or by second feature vector generation module 1135) to obtain second plurality of feature vectors 235. This example also illustrates the use of self-adjusting aggregation (in this example, by using the method described above with reference to Fig.10 The output vector 240 is calculated from the second plurality of feature vectors 235 returned by the shared model (eg, at block 1112 or by output vector generation module 1140) by using the weighted sum described above.

[0111] Another example of aggregation may include concatenating the output feature vectors into a feature table and performing a set of 1D convolutions that exchange information between cropped feature vectors from adjacent radii. Such convolutions may be performed in several layers: for example, until a planar vector with information exchanged between all radii has been reached. In this way, the training process may learn the relationship between adjacent radii. Such a technique may be described as another example of "radius convolutions", in which groups of operations are convolved over a varying spectrum (e.g., a spectrum at various radii from the center of the feature map).

[0112] Fig.13 Shows Fig. 9 In this example, one or more convolutional layers are applied to the feature vectors 230 of adjacent pairs (e.g., at block 1112 or by output vector generation module 1140). Fig.13 In the further example shown, the one or more convolutional layers may be applied to adjacent pairs of feature vectors that are the result of applying the shared model to each of the feature vectors 230 individually.

[0113] Fig.14A An implementation 434 of the cropping module 430 is shown for generating the plurality of concentric crops 220 into five concentric crops having spatial dimensions (1 x 1), (3 x 3), (5 x 5), (7 x 7), and (9 x 9). In this example, a feature map 210 having a size of (9 x 9) spatial elements xp channels (e.g., features) is included as one of the plurality of concentric crops 220.

[0114] Fig. 14B A block diagram of a specific implementation 446 of the output vector generation module 440 is shown, which includes a pooling module 4460, a shared visual model 4462, and a merging module 4464. The pooling module 4460 is used to generate a feature vector 230 by performing a pooling operation (e.g., global average pooling) on ​​each of the plurality of concentric crops 220. The shared visual model 4462 (e.g., a second trained neural network of the architecture component 405) is used to generate the plurality of second feature vectors 235 by applying the same trained model to each of the plurality of feature vectors 230 individually. In one example, the trained model is implemented as three fully connected layers, each with sixteen neurons.

[0115] The merging module 4464 is used to generate the output vector 240 using information from each of the plurality of second feature vectors 235. For example, the merging module 4464 may be implemented to calculate a weighted average (or weighted sum) of the plurality of second feature vectors 235. The merging module 4464 may also be implemented to merge (e.g., concatenate and / or add) the weighted average (or weighted sum, or a feature vector based on information from such a weighted average or weighted sum) with one or more additional feature vectors, and / or perform additional operations such as discarding and / or applying a dense (e.g., fully connected) layer.

[0116] Fig.15A A block diagram of such a specific implementation 4465 of the merging module 4464 is shown. The module 4465 is used to calculate a weighted average of the plurality of second feature vectors 235, to combine (e.g., concatenate) the weighted average with one or more additional feature vectors 250 (e.g., as generated by the optional additional feature vector generation module 1160 of the architecture component 405), and to perform a discard operation on the combined vectors, followed by a dense layer, to produce an output vector 240.

[0117] Fig. 15BA block diagram of a specific implementation 1162 of the additional feature vector generation module 1160 is shown. The module 1162 is used to generate the additional feature vector 250 by applying three layers of 3x3 convolutions (each layer having sixteen neurons) to the feature map 210 and flattening the resulting map.

[0118] Fig.16A A block diagram of an implementation 1106 of an architecture component 1105 is shown, the implementation comprising an implementation 1132 of a feature vector generation module 1130, an implementation 1137 of a second feature vector generation module 1135 (e.g., as an instance of a shared visual model 4462 as described herein), and an implementation 1142 of an output vector generation module 1140. The output vector generation module 1140 may be implemented, for example, as an instance of a merging module 1165 as described herein. In such cases, a combining module 1165 may be used to combine a weighted average (or a weighted sum, or a feature vector based on information from such a weighted average or weighted sum) with one or more additional feature vectors (e.g., as generated by an optional instance of the additional feature vector generation module 1132 of the architecture component 1105). Fig. 16B A block diagram of a feature vector generation module 1132 is shown, which includes instances of the cropping module 434 and the pooling module 447 as described above.

[0119] 4. Use mixed training techniques

[0120] Fig.17A A flowchart for an exemplary process 1700 for training a classification model including a first neural network and a second neural network is shown. Process 1700 can be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Fig.17A , at block 1704, a first neural network of the classification model and information from images of the first data set are used to generate a plurality of feature maps. Each image of the first data set may depict at least one biological cell. The first neural network may be pre-trained on a plurality of images of a second data set, the second data set comprising images that do not depict biological cells (e.g., ImageNet or another general data set). For example, the first neural network may be a specific implementation (e.g., backbone 422) of network 420 as described herein, which may be used to generate each of the plurality of feature maps (as an instance of feature map 210) from a corresponding image of the first data set (as input image 200). At block 1708, a second neural network of the classification model is trained using information from each of the plurality of feature maps. For example, the second neural network may be a specific implementation (e.g., shared vision model 4462) of a shared vision model as described herein.

[0121] Fig. 17B A flowchart for another exemplary process 1710 for classifying an input image is shown. Process 1710 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Fig. 17B , at block 1712, a feature map is generated using a first trained neural network of a classification model. The input image may depict at least one biological cell, and the first trained neural network may be pre-trained on a first plurality of images (e.g., ImageNet or another general dataset) that includes images that do not depict biological cells. For example, the first neural network may be a specific implementation (e.g., backbone 422) of network 420 as described herein, and the feature map may be an instance of feature map 210 as described herein.

[0122] At box 1716, a second trained neural network of the classification model and information from the feature map are used to generate an output vector that represents the features of the structure depicted in the central region of the input image. For example, the second trained neural network can be a specific implementation of a shared visual model as described herein (e.g., shared visual model 4462). The second trained neural network can be trained by providing the classification model with a second plurality of images depicting biological cells. For example, box 1716 can be performed by a module of the architecture component 405 or 1105 as described herein. The structure depicted in the central region of the input image can be, for example, a structure to be classified. The structure depicted in the central region of the input image can be, for example, a biological cell. At box 1720, a classification result is determined by processing the output vector. For example, the classification result can predict that the input image depicts a mitotic figure.

[0123] 5. Backend changes: encoder

[0124] FIG. 18A to FIG. 30 Further examples of methods and architectural components of the examples described above that extend the use of center emphasis (e.g., radius convolutions) of feature maps are shown. The end elements of these methods and components are similar to those described above, but the feature maps processed by the end elements have more features generated by additional elements. The additional elements may include, for example, a concatenation of feature maps from more than one model (e.g., feature maps from EfficientNet-B2 and from a region variational autoencoder), and / or a concatenation of feature maps from or with feature maps from one or more other CNN backends (e.g., a region autoencoder; U-Net; one or more other networks pre-trained on a dataset of images from the ImageNet project or the NoisyStudent dataset, which may be further fine-tuned on a specialized dataset (e.g., a dataset of images depicting biological cells).

[0125] FIG. 18A to FIG. 24 Concerning a specific implementation of the technology as described above, the specific implementation includes a trained encoder configured to produce a feature map. Such an encoder (e.g., an encoder of an encoder-decoder or "autoencoder" network, such as, for example, a variational autoencoder) can be trained to extract features independent of the type of coloring, and the best strategy for transfer learning with a specialized training image dataset improves the quality of data from labs, scanners, and colorings never seen during training.

[0126] Fig.18A A flowchart for an exemplary process 1800 for classifying an input image is shown. Process 1800 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Fig.18A , at box 1804, a trained encoder including at least one convolutional layer is used to generate a feature map for an input image, wherein the trained encoder is used to generate a potential embedding of at least a portion of the input image. At box 1808, multiple concentric crops of the feature map are generated (e.g., as described above with reference to box 408). For each of the multiple concentric crops, the center of the feature map may coincide with the center of the crop. At box 1812, an output vector is generated using the multiple concentric crops, which represents the features of the structure depicted in the central region of the input image (e.g., as described above with reference to box 412). The structure depicted in the central region of the input image can be, for example, a structure to be classified. The structure depicted in the central region of the input image can be, for example, a biological cell. At box 1816, a classification result is determined by processing the output vector (e.g., as described above with reference to box 416). For example, the classification result can predict that the input image depicts a mitotic figure.

[0127] Fig.18B A flowchart for an exemplary process 1400a for classifying an input image is shown. Process 400 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to Fig.31 Reference Fig.18B At block 1804a, a feature map for the input image is generated using a trained neural network including at least one convolutional layer and a trained encoder including at least one convolutional layer, wherein the trained encoder is used to produce a potential embedding of at least a portion of the input image. For example, the trained neural network can be a specific implementation (e.g., 422) of network 420 as described herein, and the feature map can be based on an example of feature map 210 as described herein. Blocks 1808, 1812, and 1816 are described above with reference to Fig.18AThe specific implementation of processes 1800 and 1800a is described in further detail below.

[0128] Fig.19A A schematic diagram of an encoder-decoder (or "autoencoder") network is shown, which is configured to receive an input image of size (ixj) spatial elements xk channels to encode the image into a feature map in a latent embedding space of size qxrxs, and to decode the feature map to produce a reconstructed image of size (ixj) spatial elements xk channels. Fig.19B An example of block 1804 is shown in which a feature map 212 is produced by a trained encoder 150 of such an encoder-decoder network.

[0129] Fig. 20 Shown are examples of input images and the corresponding reconstructed images as produced by an encoder-decoder network, featuring a latent embedding space of size 4 x 4 x 128 at various epochs during training. Fig.21 Examples of input images (labeled “reference”) and the corresponding reconstructed images as produced by the encoder-decoder network for latent embedding spaces of different sizes (4 x 4 x 128, 1 x 1 x 1024, and 1 x 1 x 128) are shown.

[0130] Fig.22A and Fig. 22B An example of block 1804a is shown in which a feature map 212 is generated from an input image 200 using a trained encoder 150, and a feature map 210 is generated from an input image 200 using a trained neural network 420 ( Fig.22A ), and using feature maps 210, 212 and merge operation 160 to generate feature map 214 ( Fig. 22B ). In this particular example, the feature map 212 has a size of (q x r) spatial elements xs channels, and the feature map 210 has a size of (m x n) spatial elements xp channels. The merging operation 160 may include resizing one or more of the feature maps 210, 212 to a common size along the spatial dimension, and then concatenating the common size feature maps to generate the feature map 214.

[0131] Modules 430 , 440 , and 450 of architecture component 405 as described herein may be used to perform blocks 1808 , 1812 , and 1816 . Fig.23A block diagram of an example is shown in which a feature map 214 of size (9 x 9) spatial elements x (p + s) channels (e.g., features) is processed using instances of a cropping module 434, a pooling module 4460, a shared visual model 4462, a merging module 4465, and an additional feature vector generation module 1162 as described herein to produce an output vector 240. Likewise, modules 1130, 1135, 1140, and 1150 of the architecture component 405 as described herein may be used to perform blocks 1808, 1812, and 1816. Fig.24 A block diagram of such an example is shown in which a feature map 214 of size (9 x 9) x (p+s) is processed using instances of a feature vector generation module 1132, a second feature vector generation module 1137, an output vector generation module 1142, and an additional feature vector generation module 1162 as described herein to produce an output vector 240.

[0132] 6. Backend changes: multiple feature maps (e.g., “feature pyramids”)

[0133] It may be desirable to implement an architecture as described herein to merge multiple feature maps. For example, it may be desirable to merge feature maps (e.g., feature map pyramids) representing input images at different scales. Fig.25A and Fig.25B An example of an action is shown in which a plurality of feature maps 210 ( Fig.25A ), and uses feature map 210 and merge operation 164 to generate feature map 216 ( Fig.25B In this particular example, the number of feature maps 210 is two, and the dimensions of the feature maps 210 are (m x n) spatial elements x p channels and (t xu) spatial elements x v channels. The merging operation 164 may include resizing (e.g., rescaling) one or more of the feature maps 210 to a common size along the spatial dimension (e.g., Fig.25B mxn in the example of , and then concatenate the common size feature maps to generate feature map 216. Fig.26 A block diagram of such an example is shown in which a feature map 216 of size (9 x 9) spatial elements x (p+v) channels (e.g., features) is processed using instances of a cropping module 434, a pooling module 4460, a shared visual model 4462, a merging module 4465, and an additional feature vector generation module 1162 as described herein to produce an output vector 240.

[0134] Each of the plurality of feature maps 210 may be an output of a different corresponding layer of the backbone. Fig. 27A block diagram of a specific example is shown in which each of the plurality of feature maps 216 is obtained as an output of a corresponding final layer of an EfficientNet-B2 implementation 426 of a trained neural network 420. In this example, two of the feature maps 210 are rescaled to a common size of (9 x 9) spatial elements, the common size feature maps are combined (concatenated), and a 2D dropout operation is applied to the concatenated map to produce the feature map 216. Fig.28 A block diagram of such an example is shown in which a feature map 216 of size (9 x 9) spatial elements x (L+M+N) channels (e.g., features) is processed using instances of a cropping module 434, a pooling module 4460, a shared visual model 4462, a merging module 4465, and an additional feature vector generation module 1162 as described herein to produce an output vector 240.

[0135] Fig.29 and Fig.30 An example of merging a feature map from a pyramid as described above with a feature map from a trained encoder also as described above is shown. Fig.29 A block diagram of a specific example is shown in which a feature map of dimension (8 x 8) x H as generated using the trained encoder 152 is rescaled and combined (concatenated) with multiple common size feature maps, as shown in FIG. Fig. 27 As shown, a 2D discarding operation is applied to the concatenated graph to produce a feature map 218. Fig.30 A block diagram of such an example is shown in which a feature map 218 of size (9 x 9) spatial elements x (H+L+M+N) channels (e.g., features) is processed using instances of a cropping module 434, a pooling module 4460, a shared visual model 4462, a merging module 4465, and an additional feature vector generation module 1162 as described herein to produce an output vector 240.

[0136] Fig.31 An example of a computing system 3100 is shown that may be configured to perform methods as described herein.

[0137] 7. Experiment

[0138] Four experiments as described below were carried out, and the results were evaluated using the parameters VACC (verification accuracy) and VMAUC (verification mitosis area under the curve) for the same validation set. The parameter VACC was calculated by measuring the accuracy on all four categories and averaging the results. In this case, accuracy was measured with reference to the top 1 classification (e.g., for each prediction, only the category with the highest classification score was "yes", and the other three categories were all "no"). The parameter VMAUC was calculated as the area under the precision-recall (PR) curve. The PR curve is a graph of the precision (on the Y axis) and recall (on the X axis) at various binary thresholds for a single classifier (in this case, only the mitosis category). Each point on the curve corresponds to a different binary threshold, and indicates the precision and recall obtained when the threshold is used. If the score for the mitosis category is equal to or higher than the threshold, the answer is "yes", and if the score is lower than the threshold, the answer is "no". Higher values ​​of VMAUC (e.g., larger area under the PR curve) indicate that the overall model performs better at many different binary thresholds, allowing selection from among these thresholds. Among these checkpoints, one checkpoint with the best achieved VMAUC on the four categories and one checkpoint with the best VACC (achieving two different optimization criteria at different training moments for the same validation set) are presented in Table 1 below. Fig.32 As shown in this article Fig.29 and Fig.30 Examples of PR curves for the described models.

[0139] Experiment 1: Training with "flat tail" (global average pooling and a fully connected layer) applied to the last layer of EfficientNet-B2. Checkpoint with best area under the precision-recall curve for mitosis class: VACC=0.69833, VMAUC=0.74141; checkpoint with best validation accuracy (for all 4 classes): VACC=0.72168, VMAUC=0.72249.

[0140] Experiment 2: Using exactly the same image augmentation parameterization and the same dataset but ending up with “light radius convolution” (the name indicates that the paper references e.g. Fig.12 and Fig.13The improved architecture (with the "radius convolution" model ending) was trained with one of the specific implementations of the "radius convolution" model ending described above: checkpoint with the best precision-recall area under the curve for the mitosis class: VACC=0.70843, VMAUC=0.74549; checkpoint with the best validation accuracy (for all 4 classes): VACC=0.73178, VMAUC=0.73948. In each optimization metric, this improved architecture (with the "radius convolution" ending) is able to achieve higher (better) scores on the same validation set and makes both metrics better than the architecture with the "flat ending".

[0141] Experiment 3: The same architecture (with “Efficient Net-B2” backend and “radius convolution” tail) can be used to achieve better model performance with the following improvements: Checkpoint with best area under the precision-recall curve for mitosis class: VACC=0.81161, VMAUC=0.83776; Checkpoint with best validation accuracy (for all 4 classes): VACC=0.82203, VMAUC=0.83076. Improvements include:

[0142] Improvements in training techniques (a series of trainings with selective freezing / unfreezing of some of the neural network layers in a specific order, manipulating the learning rate in successive training runs). The best working strategy is to select the set of weights of the model with the best accuracy achieved so far and do the following:

[0143] 1) First, use ADAM gradient descent with a high starting learning rate to train only the frozen model end with the remaining weights;

[0144] 2) Second, unfreeze the selected number of last layers of the core part of the model backend, and continue training from the best checkpoint with ADAM at a smaller starting learning rate by at least ten times;

[0145] 3) Third, freeze the entire backend again and train only the end of the neural network (similar to the first step, but with an even lower initial learning rate).

[0146] Improvements in data augmentation (internal augmentation library mixed with built-in augmentations from Keras and FastAI); these augmentations and manipulation of their magnitude and probability of occurrence had a significant impact on improving results, making the model more robust to scanner, tissue, and / or staining variations. Extending the training set with data scanned by the same scanner as the validation set further improved robustness on the validation data.

[0147] The strategy also includes transfer learning from the best model trained on a first dataset (which was acquired from slides of tissue samples by at least one scanner different from the validation dataset), further training only on a second dataset (which was acquired from slides of tissue samples by the same scanner as the validation dataset) until the model stops improving, and then adjusting it again on the mixed training set. The slides used to obtain the second dataset may be from a different laboratory, a different organism (e.g., dog tissue vs. human tissue), a different kind of tissue (e.g., skin tissue vs. various different breast tissues), and / or a different kind of tumor (e.g., lymphoma vs. breast cancer) than the slides used to obtain the first dataset. In addition or alternatively, the slides used to obtain the second dataset may be stained with chemical coloring substances and / or at a different ratio than the slides used to obtain the first dataset. The scanner used to obtain the first dataset may have different color, brightness, and / or contrast characteristics than the scanner used to obtain the second dataset. Several strategies for mixing different proportions of the first and second datasets were tried, and the best one was selected.

[0148] Experiment 4: In further experiments, the strategy remained the same, but an additional feature extraction backend model was added to provide a mix of feature maps with pre-trained knowledge. A "feature pyramid" strategy was used (the earlier layers of the selected backend model were scaled down to match the size of the last layer, and they were concatenated to create a feature map with more features corresponding to each local region in the input image, and these features represent various levels of abstraction). In this strategy, more terminal layers of the model backbone are unfrozen during model fine-tuning. In some cases, the entire model backbone is unfrozen for fine-tuning, and in other cases, only the last pyramid level and the end of the model are unfrozen. In one specific example, the EfficientNet-B0 model backbone was used, and the last twenty layers were unfrozen for fine-tuning (e.g., eighteen layers of the last pyramid level of the backbone, and two layers at the end of the model on top of the last pyramid level). In another specific example, the EfficientNet-B2 model backbone was used, and the last thirty-five layers were unfrozen for fine-tuning (e.g., the thirty-three layers of the last pyramid level of the backbone, and the two layers at the end of the model on top of the last pyramid level). (Note that the end of the model to which the radius convolution as described herein is applied may have more than two layers.) In other such examples, the entire model backbone was unfrozen for fine-tuning. In addition, the feature map from the bottleneck layer of one of the variational autoencoders was added to even further expand the feature map. The autoencoder was trained on various cells from H&E tissue images, where each cell was again assumed to be at the center of the image crop.

[0149] Such experiments led to slight further improvements: area under the precision-recall curve for the best implementation for the mitosis class: VMAUC = 0.83966; validation accuracy of the best implementation (for all 4 classes): VACC = 0.82550. In this method, the end of the "radius convolution" model was changed just to accommodate the increase in the number of input features per feature map region.

[0150] Table 1:

[0151]

[0152] 8. Exemplary system for center emphasis

[0153] Fig.31 3100, 1700, 1710, 1800, and / or 1800a). The computing device 3105 in the computing environment 3100 may include one or more processing units, cores, or processors 3110, memory 3115 (e.g., RAM, ROM, etc.), internal memory 3120 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interfaces 3125, any of which may be coupled to a communication mechanism or bus 3130 for communicating information, or embedded in the computing device 3105.

[0154] The computing device 3105 may be communicatively coupled to an input / user interface 3135 and an output device / interface 3140. One or both of the input / user interface 3135 and the output device / interface 3140 may be a wired or wireless interface and may be detachable. The input / user interface 3135 may include any physical or virtual device, component, sensor, or interface (e.g., button, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, etc.) that may be used to provide input. The output device / interface 3140 may include a display, television, monitor, printer, speaker, Braille, etc. In some exemplary embodiments, the input / user interface 3135 and the output device / interface 3140 may be embedded or physically coupled to the computing device 3105. In other exemplary embodiments, other computing devices may serve as or provide the functions of the input / user interface 3135 and the output device / interface 3140 for the computing device 3105.

[0155] The computing device 3105 can be communicatively coupled (e.g., via I / O interface 3125) to external storage device 3145 and network 3150 for communicating with any number of networked components, devices, and systems (including one or more computing devices of the same or different configurations). The computing device 3105 or any connected computing device can act as, provide services of, or be referred to as a server, client, thin server, general purpose machine, special purpose machine, or another label.

[0156] I / O interface 3125 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, universal system bus, WiMax, modems, cellular network protocols, etc.) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 3100. Network 3150 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0157] The computing device 3105 may use and / or communicate using computer-usable or computer-readable media (including transitory and non-transitory media). Transitory media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0158] The computing device 3105 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. Computer-executable instructions can be retrieved from a temporary medium and stored on and retrieved from a non-temporary medium. Executable instructions can be derived from one or more of any programming, scripting, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0159] The processor 3110 can execute under any operating system (OS) (not shown) in a local or virtual environment. One or more applications can be deployed, which include a logic unit 3160, an application programming interface (API) unit 3165, an input unit 3170, an output unit 3175, a boundary mapping unit 3180, a control point determination unit 3185, a transformation calculation and application unit 3190, and an inter-unit communication mechanism 3195 for different units to communicate with each other, with the OS, and with other applications (not shown). For example, the trained neural network 420, the clipping module 430, the output vector generation module 440, and the classification module 450 can implement Figure 4A , Fig.11A , Fig.17A , Fig. 17B , Fig.18A and / or Fig.18B One or more processes described and / or shown. The units and elements described may vary in design, function, configuration or implementation and are not limited to the description provided.

[0160] In some exemplary implementations, when information is received or instructions are executed by the API unit 3165, it may be transmitted to one or more other units (e.g., the logic unit 3160, the input unit 3170, the output unit 3175, the trained neural network 420, the cropping module 430, the output vector generation module 440, and / or the classification module 450). For example, after the input unit 3170 has detected the user input, it may use the API unit 3165 to transmit the user input to the implementation 112 of the detector 100 to generate an input image (e.g., from a WSI or a block of the WSI). The trained neural network 420 may interact with the detector 112 via the API unit 3165 to receive the input image and generate a feature map. Using the API unit 3165, the cropping module 430 may interact with the trained neural network 420 to receive the feature map and generate a plurality of concentric crops. Using the API unit 3165, the output vector generation module 440 can interact with the cropping module 430 to receive the concentric crops and use information from each of the multiple concentric crops to generate an output vector that represents the characteristics of the structure depicted in the central region of the input image. Using the API unit 3165, the classification module 450 can interact with the output vector generation module 440 to receive the output vector and determine the classification result by processing the output vector. Further exemplary implementations of the application that can be deployed may include the second feature vector generation module 1135 as described herein (e.g., reference Fig. 11B ).

[0161] In some cases, the logic unit 3160 may be configured to: control the flow of information between the units and direct the services provided by the API unit 3165, the input unit 3170, the output unit 3175, the trained neural network 420, the cropping module 430, the output vector generation module 440, and the classification module 450 in some of the above-described exemplary implementations. For example, the flow of one or more processes or implementations may be controlled by the logic unit 3160 alone or in combination with the API unit 3165.

[0162] 9. Other matters needing attention

[0163] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions, which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes disclosed herein.

[0164] The terms and expressions that have been adopted are used as terms of description rather than limitation, and when such terms and expressions are used, there is no intention to exclude any equivalents of the features shown and described or parts thereof, but it should be recognized that various modifications are possible within the scope of the invention claimed. Therefore, it should be understood that although the claimed invention is specifically disclosed through embodiments and optional features, those skilled in the art may make modifications and changes to the concepts disclosed herein, and such modifications and changes are considered to be within the scope of the invention defined by the appended claims.

[0165] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the description of the preferred exemplary embodiments will provide a feasible description for implementing various embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0166] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as parts in block diagram form to avoid obscuring the embodiments in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. A computer-implemented method for classifying an input image, the method comprising: include: generating a feature map for the input image using a trained neural network comprising at least one convolutional layer; generating a plurality of concentric crops of the feature map; generating an output vector using information from each of the plurality of concentric crops, the output vector representing features of a structure depicted in a central region of the input image; as well as A classification result is determined by processing the output vector. 2 . The computer-implemented method of claim 1 , wherein the structure depicted in the central region of the input image is the structure to be classified.

3. The computer-implemented method of any one of claims 1 and 2, wherein the structure depicted in the central region of the input image is a biological cell.

4. A computer-implemented method according to any one of claims 1 to 3, in, For each of the plurality of concentric crops, a center of the feature map coincides with a center of the crop.

5. The computer-implemented method of any one of claims 1 to 4, wherein the classification result predicts that the input image depicts a mitotic figure.

6. The computer-implemented method according to any one of claims 1 to 5, further comprising: include: A latent embedding of at least a portion of the input image is generated using a trained encoder comprising at least one convolutional layer, wherein generating the feature map also uses the latent embedding.

7. The computer-implemented method according to any one of claims 1 to 6, further comprising: include: Each of a plurality of feature maps is generated using a corresponding one of a plurality of final layers of the trained neural network, wherein generating the feature map uses a concatenation of the plurality of feature maps.

8. The computer-implemented method of any one of claims 1 to 7, wherein the output vector is generated using information from each of the plurality of concentric crops. include: For each of the plurality of concentric crops, generating a corresponding one of a plurality of feature vectors using at least one pooling operation; as well as The output vector is generated using information from each of the plurality of feature vectors.

9. The computer-implemented method of claim 8, wherein the output vector is generated using information from each of the plurality of feature vectors. include: The output vector is generated using a weighted sum of the plurality of feature vectors.

10. The computer-implemented method of claim 8, further comprising: include: The plurality of feature vectors are sorted according to radial dimensions of corresponding concentric crops, and wherein generating the output vector comprises independently convolving the trained filters on adjacent pairs of the sorted plurality of feature vectors.

11. The computer-implemented method of claim 8, wherein the plurality of feature vectors are sorted according to radial dimensions of corresponding concentric crops, and The output vector is generated include: convolving the trained filter on the ordered plurality of feature vectors of the first adjacent pairs to produce a first combined feature vector; as well as convolving the trained filter on the ordered plurality of feature vectors of a second adjacent pair to produce a second combined feature vector, and The using the multiple feature vectors to generate the output vector comprises: using the first combined feature vector and the second combined feature vector to generate the output vector.

12. The computer-implemented method of claim 8, wherein the output vector is generated using the plurality of feature vectors include: generating a second plurality of feature vectors using the trained model, comprising independently applying the trained model to each of the plurality of feature vectors; as well as The output vector is generated using information from each of the second plurality of feature vectors.

13. The computer-implemented method of claim 12, wherein the output vector is generated using information from each of the second plurality of feature vectors. include: The output vector is generated using a weighted sum of the second plurality of feature vectors.

14. The computer-implemented method according to any one of claims 1 to 13, wherein the method further include: The input image is selected as a tile of a larger image using a second trained neural network, wherein a central region of the input image depicts a biological cell.

15. The computer-implemented method according to any one of claims 1 to 14, wherein processing the output vector include: Applies a sigmoid function to the output vector.

16. A computer-implemented method for classifying an input image, the method include: generating a feature map for the input image; Using the feature map to generate a plurality of feature vectors; generating a second plurality of feature vectors using the trained model, comprising independently applying the trained model to each of the plurality of feature vectors; generating an output vector using information from each of the second plurality of feature vectors, the output vector representing features of a structure depicted in the input image; as well as A classification result is determined by processing the output vector.

17. The computer-implemented method of claim 16, wherein the feature map is used to generate the plurality of feature vectors include: Use at least one pooling operation.

18. The computer-implemented method of any one of claims 16 and 17, wherein the structure is depicted in a central portion of the input image.

19. The computer-implemented method of any one of claims 16 to 18, wherein the output vector is generated using the second plurality of feature vectors. include: The output vector is generated using a weighted sum of the second plurality of feature vectors.

20. A computer-implemented method for training a classification model, the classification model comprising a first neural network and a second neural network, the method include: generating a plurality of feature maps using the first neural network and information from images of a first dataset; as well as training the second neural network using information from each of the plurality of feature maps, wherein each image of the first data set depicts at least one biological cell, and The first neural network is pre-trained on a plurality of images of a second dataset, wherein the second dataset includes images that do not depict biological cells.

21. The computer-implemented method of claim 20, wherein the plurality of images of the second data set include images depicting non-biological structures.

22. A computer-implemented method according to any one of claims 20 and 21, in, For each image of the first data set, a central region of the image depicts at least one biological cell.

23. A computer-implemented method for classifying an input image, the method include: generating a feature map for the input image using a first trained neural network of a classification model; generating an output vector using a second trained neural network of the classification model and information from the feature map, the output vector representing features of a structure depicted in a central portion of the input image; and Determine a classification result by processing the output vector, wherein the input image depicts at least one biological cell, and wherein the first trained neural network is pre-trained on a first plurality of images, the first plurality of images comprising images not depicting biological cells, and Wherein the second trained neural network is trained by providing the classification model with a second plurality of images depicting biological cells.

24. The computer-implemented method of claim 23, wherein the first plurality of images includes images depicting non-biological structures.

25. A computer-implemented method according to any one of claims 23 and 24, in, For each image of the second plurality of images, a central region of the image depicts at least one biological cell.

26. A system, wherein the system include: one or more data processors; and A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

27. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform a portion or all of one or more methods disclosed herein.

Images