A multi-label pipeline for multiplex cell / phenotype detection and classification

Abstract

The present disclosure relates to a multi-label classification for multiplex phenotype detection and classification by leveraging a multi-label classification pipeline. The classification pipeline may process a multiplex image labeled with multiple markers to generate a set of annotated images. Each marker may correspond to a label type with multiple associated labels representing various phenotype characteristics (e.g., cellular-membrane protein or a cell nucleus protein). The pipeline may be trained using a set of semantic masks derived from a set of ground truth annotations. Each image in the set of annotated images may correspond to a label type of the multiple labels including distinct markings (e.g., colored dots, circles) on portions or cell locations for multiple labels corresponding to the label type on the multiplex image. The set of annotated images may help clinicians identify one or more biological conditions (e.g., cancer, osteoporosis, fibroids) and develop personalized treatment plans.

Description

A MULTI-LABEL PIPELINE FOR MULTIPLEX CELL / PHENOTYPE DETECTION AND CLASSIFICATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Application 63 / 637,264, filed on April 22, 2024, which is hereby incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Digital pathology enhances diagnosis and treatment by improving tissue analysis, detecting markers, supporting personalized treatments, and boosting pathologist efficiency through Al-based computational analyses. By digitizing glass slides to generate digital pathology images (e.g., singleplex image or multiplex images), digital pathology may enable remote slide review (telepathology). Multiplex images capture a range of signals, representing multiple biomarkers in a single scan, to assess disease progression, identify targets, predict responses, and support medical research and personalized treatments by revealing a patient's cancer profile. Multiplex images may be used to train machine-learning models to automatically identify patterns or predict outcomes from histological data, enhancing diagnostic efficiency and reducing human error. As the number of markers in digital pathology images increases, training a single model for accurate predictions becomes more complex. More markers may require a larger model with additional parameters and a bigger dataset. Additionally, data imbalance, where some biomarkers are more frequently observed than others, may create further challenges in obtaining a dataset that enables the model to effectively detect and differentiate all labels, including the less frequent ones.

[0003] One approach may comprise training separate models for each marker. However, merging predictions from multiple models may lead to spatial inconsistencies (e.g., incorrect cell or nuclei locations, due to the lack of a consistent integration method) or mismatched results, increase computational complexity. Additionally, this approach may complicate the learning of label dependencies and worsen class imbalance, especially when some labels rarely or never cooccur in the same region. In such instances, there are more 0s (pixels without biomarkers) thanIs (pixels with biomarkers). The class imbalance may result in a biased model, thereby the model may predict 0s (i .e., the absence of biomarkers) relatively more often. Another approach may involve training a multi-classification model, either by using separate models (e.g., one for multiple markers and another for a single marker) or a single model for all biomarkers. However, this approach may introduce spatial inconsistencies, and the limited data for each specific category may negatively impact the accuracy of the model's predictions. The inconsistencies make it challenging to merge outputs from multiple models into a single, reliable multi-label prediction output. While training such models may be feasible when the number of classes is small, the challenge may significantly increase as the number of classes grows (e.g., if it's 10, 20, or even 40) making it difficult to find enough data points for each label, posing a significant challenge to training a robust multi-classification model. Both approaches may encounter scaling issues, as combining results from multiple models and markers across various labels may be challenging for accurate classification, particularly when data samples are limited, and spatial inconsistencies exist in structures like nuclei. In conclusion, there is still a requirement for a more efficient and advanced model that may effectively address the challenges of data scarcity and classification complexity in multi-label tasks involving multiple markers.SUMMARY

[0004] Some aspects of the present disclosure relate to techniques for implementing a multilabel classification pipeline for multiplex phenotype detection and classification. The multiplex phenotypes may correspond to a multiplex image of a biological specimen, generated by an image generation system and labelled with multiple markers. Each marker of the multiple markers may be a biological indicator or a biomarker corresponding to cellular-membrane protein (e.g., HER2) or to a particular cell nucleus protein (e.g., ER or PR) identified by a label type from a set of label types. For each label type, multiple labels (e.g., ER-positive, ERnegative, or non-tumor for ER label type) may be identified representing expressions or predictions for each cell location or portion (such as negative, intermediate, positive, or nontumor cells).

[0005] The multiplex image may be processed by leveraging the multi-label classification pipeline to generate a set of annotated images. The process may begin with the multiplex imagebeing analyzed by a pre-processor to generate a pre-processed image, enhancing the quality of the image for further analysis. A set of interim signal maps may be generated, each corresponding to a label (e.g., ER-positive, PR-negative, HER2+or non-tumor), by processing the pre-processed image using a machine-learning model or a feature generator. The interim signal maps collectively form a multi-dimensional representation of the multiplex image. Each interim signal map may predict the distribution of a specific label across the multiplex image, indicating label consistency at each cell location or portion. The interim signal maps may assist in understanding the spatial patterns and predictions of the multiple labels in the multiplex image. In some aspects, the feature generator may generate a set of probability matrices for each interim signal map corresponding to a specific label of the multiple labels. A probability matrix of the set of probability matrices, corresponding to a specific label, may determine a likelihood of each portion or cell location belonging to the specific label. The set of interim signal maps combined with the set of probability matrices may provide detailed classification and localization, improving accuracy in identifying and segmenting labels across the pipeline.

[0006] The interim signal map may go through various post-processing steps to generate a set of cell-specific point locations. The post-processing steps may include generating a set of interim nuclei-signal maps for each label (e.g., ER-positive, ER-negative, or non-tumor for ER label type) by processing the set of interim signal maps. The set of interim nuclei-signal maps may identify predicted locations of cell nuclei corresponding to the multiple labels. Position of the cell nuclei may be detected through methods like contour detection or connected component labeling and may be represented by centroids or enclosed in bounding boxes or ellipses. The predicted locations may be overlaid onto the interim signal maps or binary images highlighting their detection. For visualization, a binary mask or RGB image may be generated, marking detected nuclei with distinct symbols or colors, providing a clear visualization of nuclei locations based on the interim signal maps.

[0007] The set of cell-specific point locations may be generated by processing at least two interim signal maps or interim nuclei-signal maps using one or more mapping functions. The processing may identify precise, spatially resolved locations within the multiplex image that may correspond to an individual cell location or a portion for the multiple labels. By correctly identifying the locations, the multi-label classification pipeline may provide meaningful insights into distribution and concentration of multiple labels, aiding development of more accuratediagnostic tools and personalized treatment strategies. The mapping functions may apply mathematical operations on data structures, such as matrices or arrays, to adjust data based on factors like signal intensity or distribution, enhancing sensitivity and resolution of the pipeline for diagnostic and therapeutic applications. In some aspects, each cell-specific location corresponding to a specific label may be assigned a distinct color, creating an RGB image where nuclei positions for each label are rendered in the same color, providing clear visualizations of nuclei locations. At least two interim signal maps or nuclei-signal maps may be used to generate cell-specific point locations that may correspond to multiple labels within the same label type. For example, labels ER+ (estrogen receptor-positive), ER- (estrogen receptor-negative), and nontumor labels may be combined to generate a signal cell-specific point map for ER label type.

[0008] A set of annotated images may be generated by overlaying the set of cell-specific locations onto the multiplex image. Each label from the multiple labels may be represented by a distinct color, with each annotated image corresponding to a specific label type. Each label, linked to a particular label type, may be assigned to its corresponding cell location or portion on the multiplex image, allowing for a better visual differentiation. The set of annotated images (i.e., output) may be used to identify one or more biological conditions associated with the multiple different markers or label types. The feature generator may utilize various machinelearning models, including convolutional neural networks (CNN), deep neural networks (DNN), or specialized architectures (e.g., UNet or VNet), trained on labeled data to automatically predict cell boundaries corresponding to multiple labels of interest, and segment the multiplex image, generating the set of interim signal maps. The machine-learning models may enable accurate spatial predictions in image segmentation, trained by using a binary cross-entropy loss function, allowing the multi-label classification pipeline to precisely label regions in the multiplex image, improving the accuracy of diagnostics and marker detection.

[0009] The feature generator may be trained using pretrained detection models for each label type, which generate ground truth annotations for label presence, location, and boundaries based on singleplex images with specific markers. The annotations may be used for training, validation, and evaluation to improve label recognition in multiplex images, with results displayed on a user-friendly GUI. Semantic masks may be generated to highlight labels and differentiate them from the background, with adjustable weights addressing class imbalance by increasing sensitivity to less common labels and improving accuracy. Binary cross-entropy may be used asa loss function, enabling faster convergence and weight adjustment through backpropagation. Interim signal maps and overlaid images may help calculate the loss and address class imbalance, enhancing the model's performance over time.

[0010] The present disclosure may enable clinicians or healthcare providers to make relatively accurate and comprehensive diagnoses, especially for complex diseases like cancer. By using a multi-label classification pipeline, it may eliminate the requirement to merge predictions for multiple biomarkers, allowing simultaneous detection and classification in a single step. The pipeline may enhance efficiency and accuracy by streamlining model management and tackling challenges like class imbalance and integration of spatially distinct phenotypes.

[0011] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0012] In some embodiments, a computer-program product tangibly embodied in a non- transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods or processes disclosed herein.

[0013] In some embodiments, a system is provided that includes one or more means to perform part or all of one or more methods or processes disclosed herein.

[0014] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. The present disclosure is described in conjunction with the appended figures:

[0016] FIG. 1 illustrates an exemplary block diagram for multi-label detection and classification of multiplex phenotypes in accordance with some aspects of the present disclosure.

[0017] FIG. 2 illustrates an exemplary block diagram for generating a digital pathology image in accordance with some aspects of the present disclosure.

[0018] FIG. 3 is a block diagram illustrating an exemplary architecture for generating ground truth annotations by a set of detection models in accordance with some aspects of the present disclosure.

[0019] FIG. 4 is a block diagram illustrating an exemplary architecture for generating a set of semantic masks for each label from the ground truth annotations in accordance with some aspects of the present disclosure.

[0020] FIG. 5 illustrates an exemplary workflow of a multi-label classification pipeline for detection and classification of multiplex phenotypes in accordance with some aspects of the present disclosure.

[0021] FIG. 6 illustrates an exemplary workflow of a back-propagator of the multi-label classification pipeline in accordance with some aspects of the present disclosure.

[0022] FIG. 7 illustrates a set of singleplex images (corresponding to ER, PR and HER2) along with a set of semantic masks in accordance with some aspects of the present disclosure.

[0023] FIG. 8 illustrates an input triplex image with a set of interim nuclei-signal maps and a set of interim signal maps for three label types (ER, PR and HERZ) generated by the multi-label classification pipeline in accordance with some aspects of the present disclosure.

[0024] FIG. 9 illustrates an exemplary output of the input triplex image having markers for the three label types (ER, PR and HER2) by the multi-label classification pipeline in accordance with some aspects of the present disclosure.

[0025] FIG. 10 illustrates exemplary plots displaying one or more performance metrics graphs on training and validation sets in accordance with some aspects of the present disclosure.

[0026] FIG. 11A shows confusion matrices illustrating performance of a set of detection models at training for ER, PR, and HER2 label types in accordance with some aspects of the present disclosure.

[0027] FIG. 11B shows confusion matrices illustrating performance of a set of detection models at validation set for ER, PR, and HER2 label types in accordance with some aspects of the present disclosure.

[0028] FIG. 12 shows a confusion matrix illustrating performance of the multi-label classification pipeline from the input triplex image in accordance with some aspects of the present disclosure.

[0029] FIG. 13 illustrates an exemplary flowchart for detection and classification of the multiplex phenotype in accordance with some aspects of the present disclosure.

[0030] FIG. 14A illustrates a side-by-side comparison of overlay estimations of PR and ER on a first triplex image and synthetic PR and ER singleplex images in accordance with some aspects of the present disclosure.

[0031] FIG. 14B illustrates a side-by-side comparison of overlay estimations of PR and ER on a second triplex image and augmented images in accordance with some aspects of the present disclosure.

[0032] FIG. 15 illustrates a side-by-side comparison of overlay estimations of HER2 on a third triplex image and an augmented image in accordance with some aspects of the present disclosure.DETAILED DESCRIPTION

[0033] Some aspects of the present disclosure relate to techniques for implementing a multilabel classification pipeline methodology for detecting and classifying multiplex phenotypes or markers by leveraging various machine-learning techniques. Traditional approaches often require merging locations and phenotype predictions but in the absence of evidence-based rules, this stillremains a challenging task. Additionally, issues such as class imbalance in training data, limited training data for some labels, and an ever-increasing number of classes may hinder the scalability and effectiveness of the traditional approaches. Thus, obtaining a training set capable of teaching a model to detect and differentiate all labels (including less frequent ones) may become a significant challenge. The methodology in the present disclosure may eliminate the requirement for merging multiple marker predictions, streamlining prediction processes, and significantly improving efficiency and accuracy of predictions.

[0034] In some aspects, a multi-label classification pipeline may process a multiplex image labeled with multiple markers depicting a biological specimen within a tissue sample. The multiple markers may include various biomarkers or cellular components. By multiplexing multiple markers, the pipeline may enable a detailed analysis of spatial relationships for diagnosing one or more biological conditions. A set of label types may be identified corresponding to the multiplex image of the biological specimen within the tissue sample. Each label type of the set of label types may correspond to a marker of the multiple markers. The label types may function as identifiers, enabling the classification and distinction of biological components, such as biomarkers or cellular structures, based on their distinct characteristics. The biomarkers may be employed for categorizing biological features or molecular characteristics, including the presence, expression level, or localization of target molecules. The label types may include proteins such as cellular membrane proteins (e.g., HERZ) or nuclear proteins (e.g., ER or PR), offering meaningful insights into the cellular composition and functionality of the tissue sample.

[0035] For each label type, multiple labels may be identified, each representing various categories or expression intensities (e.g., negative, intermediate, positive, or non-tumor cells) of a label type associated with a specific marker. For example, for HER2 protein biomarker, an HERZ label type may be identified having five distinct labels corresponding to a different level of HERZ protein expression with their spatial distribution around the cellular-membrane typically used for determining tumor characteristics. The five distinct labels may comprise HERZ 2+, HER2 3+, HERZ 1+, non-tumor, and HER2-. HERZ- may indicate no overexpression of the HERZ protein or gene, while HERZ 3+ may signify an elevated level of overexpression. HER2 1+ and HER2 2+ may represent intermediate levels of expression. Additionally, HER2 non- tumor regions may be identified, representing areas where HERZ is not overexpressed.

[0036] The multiple labels may be used to annotate and classify the multiple labels associated with each label type on the multiplex image, resulting in a set of annotated images as an output. Each annotated image of the set of annotated images may be a multi-labeled image corresponding to a label type of the set of label types highlighting distinct biological features or different biomarkers. The set of annotated images may provide a comprehensive view, enabling complex analysis, such as the identification of co-expressions, differentiation of tumor versus non-tumor cells, or identification of various subtypes of cells based on their molecular profiles (e.g., gene expression, protein expression, cell surface markers, metabolites or signaling pathways etc.). The analysis may enable researchers and clinicians to visually assess the spatial relationships and expression patterns of various markers via the multiple labels within the multiplex image of the biological specimen. For example, an annotated image may highlight regions where specific markers have elevated expression with corresponding labels indicating the areas of elevated expression, or presence of distinct cellular components.

[0037] The multi-label classification pipeline may process the multiplex image by analyzing its spatial distribution and intensity of the multiple markers. The multi-label pipeline may include several pre-processing steps to generate a pre-processed image, enhancing the quality of the multiplex image for further analysis. The pre-processing steps may include resizing, normalization, contrast adjustment, background subtraction, noise reduction or thresholding to improve the accuracy and reliability of the multi-label classification pipeline. Normalization may adjust the intensity to account for variations in lighting or exposure, ensuring that features are comparable across images. In addition, contrast adjustment techniques (e.g., histogram equalization or adaptive contrast enhancement) may further highlight key features including cell boundaries that make them distinct. Furthermore, removing unwanted background noise or irrelevant information from the image may enhance the visibility of various key features that are of interest. In addition, applying methods including gaussian or median filtering may smooth the image and eliminate noise that may interfere with classification during later stages in the multilabel classification pipeline. Moreover, converting the image into a binary format by leveraging techniques like Otsu's method or adaptive thresholding may be utilized. Additionally, advanced pre-processing techniques may be applied based on the complexity of the multiplex image, and after examining the multiple markers and understanding various requirements of the labeling process.

[0038] The pre-processed image may be segmented by a machine-learning model or a feature generator, generating a set of interim signal maps corresponding to the multiple labels. Each interim signal map may represent predictions for each portion or cell location within the multiplex image, indicating whether the portions align with the corresponding labels of the multiple labels (e.g., ER-positive, ER-negative, or non-tumor for ER label type). The interim signal maps may collectively form a multi-dimensional representation of the multiplex image, where each map may represent a predicted distribution of a specific label across the image. The interim signal maps may be useful in understanding the spatial patterns and predictions of various biomarkers in the tissue sample. In some aspects, the feature generator may also generate a set of probability matrices for each interim signal map associated with each label. A probability matrix of the set of probability matrices, associated with a particular label may reflect the likelihood of each portion or cell location being assigned to the particular label. When combined, the set of interim signal maps and the set of probability matrices may enable detailed classification and localization, enhancing the accuracy of label identification and segmentation throughout the pipeline.

[0039] The interim signal map may go through various post-processing steps to generate a set of cell-specific point locations. The post-processing steps may include generating a set of interim nuclei-signal maps for each label of the multiple labels (e.g., ER-positive, ER-negative, or non-tumor for ER label type) by processing the set of interim signal maps. The set of interim nuclei-signal maps may identify the representative location of a cell that may be predicted to correspond to each label. Nuclei may appear as distinct regions within cells in the set of nuclei signal maps, and their locations may be detected using methods like contour detection or connected component labeling. For visualization, a binary mask image or interim nuclei signals maps may be generated whereby detected nuclei are marked in white (or another color) against a black or transparent background. The detected nuclei locations represented by centroids or enclosed within bounding boxes or ellipses may be overlaid onto the interim signal maps or on newly generated binary images highlighting their detection. Alternatively, an RGB image may be generated where the nuclei positions corresponding to a specific label of the multiple labels may be highlighted using distinct markers, such as dots, crosses, or other symbols, all rendered in the same color. A clear and precise visualization of nuclei locations based on the segmented cell images may be provided that may be further refined based on images’ quality.

[0040] Further, the set of cell-specific point locations may be generated by processing at least two of the interim signal maps or at least two of the interim nuclei-signal maps based on one or more mapping functions. The processing may involve identifying precise, spatially resolved locations within the multiplex image corresponding to individual cells or specific cellular regions corresponding to the multiple labels. By pinpointing such cell-specific point locations, the multi-label classification pipeline may deliver accurate insights into the distribution and concentration of biomarkers across the tissue facilitating the development of more precise diagnostic tools and personalized treatment strategies allowing clinicians to better assess how biomarkers may be expressed at the cellular level within the tissue environment. The mapping functions may execute mathematical operations on data structures (e.g., matrices or arrays) with coefficients, enabling a function to adjust to data based on criteria like signal intensity or distribution in interim signal or nuclei maps. The mathematical operations may support the identification of accurate locations for individual cells, helping to improve the overall sensitivity and resolution of the model for diagnostic and therapeutic applications.

[0041] In some aspects, each cell-specific location corresponding to a specific label may be assigned to a distinct color, creating an RGB image where nuclei positions for each label may be rendered in the same color. In some aspects of the present disclosure, at least two of the interim signal maps or at least two of the interim nuclei-signal maps, that are used to generate the set of cell-specific point locations, may be associated with multiple labels corresponding to the same label type of the set of label types. For example, combining signal maps for multiple labels such as ER+ (estrogen receptor-positive), ER- (estrogen receptor-negative), and non-tumor for ER label type.

[0042] During processing of the signal maps, the label with the highest probability may be selected for each cell location or portion by comparing probability matrices. This may improve the ability of the classification pipeline to differentiate between labels related to cellular or tissue characteristics, enhancing diagnostic, prognostic, and treatment precision while enabling relatively accurate label mapping for personalized medical insights. The post-processing steps may also include thresholding to classify each interim signal map, predicting presence of a specific label. For example, a threshold of 0.7 may be set, where probability values (from a probability matrix) above this threshold may be classified as labels, and those below may be considered as background. Thresholding ensures that high-confidence predictions may beretained, reducing noise and improving the pipeline's accuracy by focusing on the most reliable predictions. Thresholding may also help with non-maximum suppression (NMS), which may remove redundant detections, resulting in a cleaner and relatively more accurate output. After thresholding classifies labels and background, NMS may eliminate overlapping or low- confidence detections by retaining reliable predictions, based on a predefined intersection over union (loU) threshold, ensuring precise label classification by suppressing false positives. Centroids of positive responses may identify predicted nuclei locations, while the multiple labels associated with each nucleus or cell location or portion may provide deeper insights, enhancing the information extracted from the predictions.

[0043] Furthermore, the set of annotated images may be generated by overlaying the cellspecific locations onto the triplex image using an annotator. In some aspects, each label of the multiple labels may be represented by distinct colors on the multiplex image or each portion of a digital pathology image (e.g., cell, pixel, or segmented region). Finally, a label (associated with the respective biomarker) may be selected and assigned to a portion. The annotated images may provide meaningful insights into a biomarker status at a granular level and may be used to identify biological conditions associated with various markers, including cancer, osteoporosis, fibroids, autoimmune diseases, hormone-related conditions, cardiovascular diseases, PCOS, menstrual disorders, neurological diseases, and some thyroid disorders.

[0044] Training the feature generator may utilize a set of pretrained detection models adapted to each label type of the set of label types. The set of detection models may generate a set of ground truth annotations providing accurate references for the presence, location, and boundaries of multiple labels corresponding to each label type. The set of detection models may utilize a set of singleplex images that are generated by an image generation system. Each image in the set of singleplex images may comprise markers for a specific label type that may be used to generate ground truth annotations for the specific label type. The set of ground truth annotations may be useful for training, validation, and evaluation, enabling the feature generator to accurately recognize and predict multiple labels in multiplex images ensuring enhanced detection accuracy. Each detection model of the set of detection models may include traditional machine-learning models (e.g., SVM, KNN, random forests), CNNs (e.g., LeNet, AlexNet, DenseNet), transfer learning models (e.g., VGG16, ResNet50), or specialized models (e.g., U- Net, DeepLab).

[0045] A set of semantic masks may be generated based on the ground-truth annotations to highlight the presence of multiple labels differentiating them from background. A set of semantic masks may include a semantic mask for each label of the multiple labels and one additional mask for the background representing areas without any biomarkers. To generate semantic masks, a set of empty masks may be initialized, each populated with zeros to signify no detected labels. Each empty mask of the set of empty masks may correspond either to one label of the multiple labels or the background. Iterative biomarker processing may refine mask generation by detecting marker regions or boundaries and defining a circular region around the predicted label position based on ground truth annotations by a boundary detector. The radius of the region may be determined by a prior knowledge or ground truth annotations, localizing the label within the boundaries. The processing may further include generating binary masks for each label and one for the background, pixels inside the circular region may be set to 1 (indicating presence) and those outside are set to 0 (indicating absence). A background mask may be generated by identifying biomarker regions (regions with labels) and assigning a value of 1 to all non-marker pixels. The processing may also include overlaying a set of binary masks with the background mask, producing the set of semantic masks distinguishing biomarkers from non-biomarker regions. Additionally, the generated masks may be used to train the feature generator or fed back into boundary detector to improve the quality of the semantic masks in subsequent iterations.

[0046] In some aspects of the present disclosure, some labels from the multiple labels may correspond to more prevalent phenotypes or markers, either generally or within a specific training set, leading to a class imbalance. If not properly addressed, this imbalance may cause the model to favor more prevalent labels, resulting in both false negatives and false positives, underpredicting fewer common ones and reducing performance, in detecting rare labels or abnormal features for accurate diagnosis. To address class imbalance, the feature generator may be trained with adjustable weights to influence label assignments, including the application of weighted masks or weight bias in case of conflicts. The weight bias may facilitate by assigning higher weights to less prevalent labels, balancing the training process and improving learning from underrepresented data.

[0047] The weight bias may ensure the model receives more exposure to positive examples, helping it learn distinguishing features. However, even with introducing bias weights, overall representation of true positives may not surpass false negatives or false positives due to naturaldata distribution or resource constraints. As a result, the model may be trained to handle both positive and negative detections effectively, ensuring robust prediction accuracy for both outcomes. The degree of weight bias may vary based on factors such as label frequency, pixel distribution, or external data on label prevalence, ensuring the feature generator is more sensitive to rare labels and improving its relative accuracy for detecting some labels (e.g., non-tumor vs. ER+). The weight bias may resolve conflicts by prioritizing labels based on predefined criteria, such as false positives, false negatives, and label prevalence across interim signal maps or ground truth annotations. Applying the weights may allow the feature generator to resolve label conflicts and assign the correct label to each portion, enhancing diagnostic prediction accuracy.

[0048] In some aspects, a set of weights may be incorporated into the set of semantic masks during the training process to adjust the threshold that is applied during the process of defining whether a nucleus should be recognized within a given region of a particular semantic mask. By modifying the threshold, the set of weights may influence the sensitivity of the model to detect the presence of a nucleus in that region, ensuring relatively more accurate delineation based on predefined criteria. The modification may allow the model to prioritize specific portions (e.g., regions or features), enhancing the precision of nucleus identification for each label.

[0049] In some aspects, another set of weights may be applied to the nuclei in one interim nuclei-signal map compared to another, allowing for effective handling of varying scenarios or labels based on changes in the dataset. A baseline nuclei map may be generated by analyzing the location of all nuclei by comparing multiple interim nuclei-signal maps to determine whether detected nuclei belong to the same portion or a different portion within the multiplex image. If the analysis determines that multiple detected nuclei correspond to the same cell, a single, consolidated location may be identified. However, when multiple inconsistent labels are assigned to a given cell (for a specific label type), the weights or probabilities of the conflicting labels may be compared. The comparison may help resolve the inconsistency and select the appropriate label for each cell or portion, ensuring that the final label assignment is accurate and consistent with the underlying data.

[0050] Binary cross-entropy may be used as a loss function to train the feature generator by calculating the difference between predicted and true labels, particularly for binary classification tasks, such as determining whether a pixel or region belongs to a specific label (e.g., tumor ornon-tumor). The cross-entropy may measure discrepancy between predicted and actual labels, aiming to minimize the gap by penalizing predictions based on confidence and accuracy. Relatively smaller losses may occur for closer matches, while relatively larger losses may occur for significant deviations. The cross-entropy may provide a steep gradient for large discrepancies, helping the model converge faster and be more sensitive to errors. Compared to other loss functions like mean squared error (MSE), cross-entropy may be effective for binary classification as it produces a relatively more responsive gradient. During backpropagation, the loss may be utilized to generate one or more weights (or adjusted wights) based on the set of interim signal maps and overlaid images, which are created by combining the triplex image with the semantic masks. The one or more weights may be based on a weight adjustment unit and used to update the feature generator, improving its performance. In some aspects, the weight bias to address class imbalance, along with one or more weights, may be combined into a single back- propagator to improve the model’s training.

[0051] In some aspects, a labeled portion of the multiplex image or a digital pathology image may be smaller than a physiological cell or cellular component. Additionally, in some aspects, spatial clustering of a given marked element (e.g., a biomarker or cell type) may be expected based on physiological characteristics. For example, biomarkers or cellular features may naturally cluster in tissue samples, reflecting biological structures or patterns. In such instances, where portions are relatively small or clustering is physiologically expected, one or more masks may be applied to an interim signal map. The masks may help transform specific regions of the map, allowing the model to accurately identify predicted nuclei locations (or other cell-specific points). The process helps in refining the segmentation of individual cells, even when they are small or tightly clustered.

[0052] To illustrate, for a marker that is localized to specific nuclei (e.g., ER or PR), a relatively small semantic mask (e.g., a 5x5x5 mask) may be applied to identify the regions of interest. The mask may be configured in such a way that the signal centroid (i.e., central point of the detected label) is characterized as a nucleus. However, the characterization may be based on setting a threshold value on a colocalized signal, meaning that the signal meets a threshold in terms of intensity and spatial overlap that may be classified as a valid nucleus location. While the thresholding step may not require to be explicitly performed during image processing, the thresholding step remains integral in the outcome of the mask application. Alternatively, for amarker that is associated with a cellular-membranes protein (e.g., HER2), a larger mask (e g., a 20x20x20 mask) may be used to cater for its relatively larger size and distinct spatial arrangement of a cellular-membrane. In some aspects, the threshold for detecting cellular- membrane using this mask may be different from the one used for nuclei, as the signal intensity and spatial distribution of cellular-membrane protein markers may vary.

[0053] In some aspects, a graphical user interface (GUI) may be provided to display at least two interim signal maps (e.g., corresponding to different labels or label types) or interim nuclei- signal maps. The GUI may enable users to interact with the maps and provide input, such as reject, approve, or modify results of the image analysis. Users may be able to adjust labels (e.g., modify predicted labels for specific regions or pixels), nuclei (e.g., update or refine detected nuclei), or nuclei locations (e.g., change predicted positions of nuclei within a tissue). The GUI may automatically update the displayed maps in real time to reflect user changes, enabling efficient feedback and refinement for greater accuracy. Additionally, another graphical user interface may display ground truth annotations alongside semantic masks for each label, enabling user interaction and feedback in case of discrepancies. The other interface may also allow users to modify, approve, or reject aspects of the image analysis results.

[0054] The feature generator may utilize various machine-learning models, including convolutional neural networks (CNN), deep neural networks (DNN), or specialized architectures like UNet or VNet, trained on labeled data (e.g., the set of ground truth annotation and / or the set of semantic masks) to automatically predict cell boundaries of interest and segment images, generating the set of interim signal maps. The models may enable accurate spatial predictions in image segmentation, allowing the multi-label classification pipeline to precisely label regions in the multiplex image, improving diagnostics and marker detection.

[0055] Consider an example of a triplex image having stains from three markers (e.g., ER, PR, and HER2) that may help predict various biological conditions. Each marker of the three markers may be a biological indicator or a biomarker corresponding to cellular-membrane protein (e.g., HER2) or to a particular cell nucleus protein (e.g., ER or PR) that is identified by a label type from a set of label types. ER may detect hormone-related conditions like breast cancer, osteoporosis, and thyroid disorders, with outputs as ER+, ER-, and non-tumor labels for ER label type. PR may aid in identifying breast cancer, fibroids, and reproductive health issues, withoutputs as PR+, PR-, and non-tumor labels for PR label type. HER2 may be helpful in diagnosing breast and other cancers, with outputs as HER2+, HER3+, HER1+, non-tumor, and HERZ- labels for HER2 label type.

[0056] A pre-processed image may be generated from the triplex image and fed to a feature generator, producing twelve interim signal maps (eleven for labels and one for the background). The interim signal maps may be used to generate twelve interim nuclei-signal maps, which may be further refined to generate twelve cell-specific point locations. Finally, three annotated images may be generated, each corresponding to a label type. Each annotated image may have markings for multiple labels corresponding to that label type. For training the feature generator, three pretrained detection models may generate twelve ground truth annotations (eleven for labels and one for the background) from a corresponding set of singleplex images for each marker or label type. Based on the annotations, twelve semantic masks, representing multiple labels and the background, may be generated and initialized with pixel values of 0 (no marker detected). For example, an ER+ mask may label ER+ regions in white and others in black. The feature generator may process the masks to create markers based on intensity levels, enabling the detection of ER, PR, or HERZ, enhancing classification precision. Some labels may be mutually independent, such as classifying ER-positive and ER-negative cells within the same sample or identifying a cell pathology that shows both a tumor and non-tumor simultaneously.

[0057] FIG. 1 illustrates an exemplary block diagram 100 for multi-label detection and classification of multiplex phenotypes in accordance with some aspects of the present disclosure. The multiplex phenotypes may be defined based on multiple markers or biological indicators that may be detected from a multiplex image, depicting a biological specimen within a tissue sample. The multiplex phenotypes may be characterized by the spatial distribution and intensity of multiple markers, such as proteins, genes, or cellular components, that are highlighted in the multiplex image. By analyzing the patterns of the markers, multiplex phenotypes may provide meaningful insights into complex biological processes, cellular interactions, or disease states. The provided insights may be useful in understanding mechanisms of various biological conditions, improving diagnostic accuracy, and developing targeted therapies, especially in fields like cancer research, immunology, and pathology.

[0058] The multiple markers may correspond to a set of label types, with each label type having multiple associated labels. The exemplary block diagram 100 may outline the processthrough which multiple markers may be identified, analyzed, and classified simultaneously. The exemplary block diagram 100 may include an image generation system 102, a network 106, a multi-label classification pipeline 108, a database 110 and a computer system 114. The image generation system 102 may generate a multiplex image 104 marked with the set of label types. The multiplex image may capture complex biological data, related to the biological specimen, using advanced imaging techniques. The image generation system 102 may be connected to a network 106, enabling the transfer of the multiplex image 104 to other components (e.g., the computer system 114 and the multi-label classification pipeline 108) in the exemplary block diagram 100. The network 106 may function as a bridge, ensuring seamless communication and integration across the exemplary block diagram 100.

[0059] The multi-label classification pipeline 108 may analyze the multiplex image 104, assigning multiple labels associated with each label type to generate a set of annotated images 112, showing the detection of various phenotypes or markers. Each annotated image of the set of annotated images 112 may correspond to a label type of the set of label types. Each annotated image associated with a particular label type may indicate, for each portion (one or more pixels or cell location) of the image, and the corresponding label of the multiple label types associated with the particular label type. The pipeline 108 may utilize machine-learning models, image processing techniques, and advanced algorithms to accurately identify and categorize different biological indicators within the multiplex image. The computer system 114 may be connected to a database 110 that may store images generated by the image generation system 102. The database 110 may allow for easy retrieval and management of multiplex images, enabling access for further processing.

[0060] For example, a triplex image marked with three label types (ER, PR, and HER2) associated with one or more biological conditions (e.g., cancer) may be generated by the image generation system 102. The triplex image may be fed to the multi-label classification pipeline 108 through the network 106. The pipeline 108 may generate three images, with each image corresponding to one label type, as shown in the displayed output (i.e., the annotated images 112). To illustrate, regarding HER2 protein, a first annotated image of the set of annotated images 112 may indicate, for each portion or cell location, one of five labels (HER2 2+, HER2 3+, HER2 1+, Non tumor and HER2-). Each of the five labels may correspond to a different expression of the ELER2 protein, for example, HER2- may correspond to a prediction of nooverexpression of the HER2 protein or gene, while HER2 3+ may indicate overexpression. HERZ 1+ and HER2 2+ may represent varying levels of expression, with their spatial distribution around the cellular-membrane may be utilized for determining tumor characteristics. Additionally, HERZ non-tumor regions may be identified, reflecting areas where the HERZ protein is not overexpressed. As another illustration, regarding the ER protein, a second annotated image of the set of annotated images 112 may indicate, for each portion or cell location, one of three classes (ER positive, ER negative, or ER non-tumor). Similarly, regarding the PR protein, a third annotated image of the set of annotated images 112 may indicate, for each portion or cell, one of three classes (PR positive, PR negative, or PR non-tumor). Each portion or cell location may correspond to a pixel or one or more pixel. The set of annotated images 112 may enable a comprehensive analysis of the tissue, capturing diverse expressions and relationships of the set of label types. The analysis may enhance a clinician’s understanding of underlying biological conditions and their progression, supporting more effective intervention strategies.

[0061] FIG. 2 illustrates an exemplary block diagram for generating a digital pathology image in accordance with some aspects of the present disclosure. An image generation system 102 may be utilized to generate digital pathology images 212 including a singleplex image or the multiplex image. The image generation system 102 may include a fixation / embedding system 204, a tissue slicer 206, a staining system 208, and an imaging system 210. The fixation / embedding system 204 fixes and / or embeds a tissue sample (e.g., a liquid fixing agent such as formaldehyde solution) and / or an embedding substance (e.g., a historical wax such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by exposing the slice to a fixating agent for a predefined period of time (e.g., at least 3 hours) and by then dehydrating the slice (e.g., via exposing it to an ethanol solution and / or a clearing intermediate agent). The embedding substance may infiltrate the slice when it is in a liquid state (e.g., when heated).

[0062] The tissue slicer 206 then slices a fixed and / or embedded tissue sample (e g., a sample of a tumor) to obtain a series of sections, with each section having a thickness of (for example) 4 to 5 microns. Such a slicing of a tissue into sections may be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue may be sliced using (for example) a vibratome or compress tome. Because the tissue sections and the cellswithin them are transparent, preparation of the slides typically includes staining (e.g., automatically staining) the tissue sections to render relevant structures more visible. In some aspects, the staining may be performed manually. In some other aspects, the staining may be performed semi-automatically or automatically using the staining system 208.

[0063] The staining may include exposing an individual section of the tissue to one or more different stains (e.g., consecutively, or concurrently) to express distinct characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for a predefined period of time. The staining agent may include (for example) an RNA probe, a protein probe (e.g., nuclear-protein probe or cytoplasm-protein probe), an immunohistochemistry stain, or a probe for a secreted substance etc. In some aspects, the staining agent is the one that stains for KAPPA mRNA or LAMBDA mRNA.

[0064] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of a tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's Trichrome), Periodic Acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e.g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian Blue, phosphomolybdic acid (PMA)) may have molecular weights of up to two or three thousand kD. One case of a high-molecular-weight histochemical staining reagent is alpha-amylase (about 55 kD), which may be used to indicate glycogen.

[0065] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that binds specifically to the target antigen of interest (biomarker). IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody is first bound to the target antigen, and then a secondary antibody that is conjugated with a label (e.g., a chromophore or fluorophore) bound to the primary antibody. The molecular weights of IHC reagents are much higher than those of histochemical staining reagents, as the antibodies have molecular weights of about 150 kD or more.

[0066] The sections may then be individually mounted on corresponding slides. The imaging system 210 may then scan the slides to generate digital pathology images 212. Each section may be mounted on a slide, which is then scanned to create a digital image that may be subsequently examined by a digital pathology image analysis and / or interpreted by a human pathologist (e.g., using an image viewer software). The imaging system 210 may digitize pathology slides (complete slide or a section) using bright-field imaging, dark-field imaging, or fluorescence imaging. The imaging system 210 may include, but is not limited to, microscope with digital camera, robotic microscopes, or WSI scanners such as Ventana iScan HT, Ventana DP 200, or Ventana DP 600. In some aspects, a pathologist may review and manually annotate the digital image of the slides (e.g., tumor area, necrosis, etc.). Annotation of regions of interest may be performed automatically using a computer-vision technique. Digital pathology images 212 may be converted into other domains for further processing.

[0067] A digital histopathology image (e.g., 212) typically includes an array, usually a rectangular matrix, of pixels. Each “pixel” is one picture element and is a digital quantity that represents some property of the image at a location in the array corresponding to a particular location in the image. Typically, in continuous tone black and white images the pixel values represent a gray scale value. Pixel values for a digital image typically conform to a specified range. For example, each array element may be one byte (i.e., eight bits) representing pixel values in the range of 0 to 255. In a gray scale image, a value of “255” may represent white color and a zero (‘0’) value may represent black. Color images may comprise of, corresponding to red, green, and blue (RGB). For a particular pixel, there is one value for each of the three-color planes, (i.e., a first value representing the red component, a second value representing the green component, and a third value representing the blue component). By varying the intensity of the three components, all colors in the color spectrum may be created. A specimen stained by multiplex IHC may be illuminated sequentially with multiple light channels that are matched to the absorbance bands of the chromogens to capture brightfield images. In the case of multiplex immunofluorescence, fluorescence microscopy with different filters may be used to capture fluorescence or emitted light from fluorophores associated with each biomarker.

[0068] FIG. 3 is a block diagram illustrating an exemplary architecture 300 for generating a set of ground truth annotations 306 by a set of detection models 304 in accordance with some aspects of the present disclosure. The set of detection models 304 may be trained to detectmultiple labels corresponding to a label type. A set of singleplex images 302 generated by the image generator system 102 may be fed to the set of detection models. Each singleplex image of the set of singleplex images 302 may correspond to a particular label type of the set of label types. A detection model of the set of detection models 304 may analyze a singleplex image of the set of singleplex images 302 to identify and classify multiple labels corresponding to a particular label type by extracting relevant features from corresponding singleplex image.

[0069] The results of the ground truth annotations 306 may be presented on a graphical user interface (GUI) 308 on the computer system 114. The GUI 308 may provide an intuitive and user-friendly interface for visualizing results that may aid in further analysis and decisionmaking. The GUI 308 may include multiple images, where each image may correspond to a particular label type that is marked with multiple labels corresponding to the particular label type. The pipeline 108 may then be able to perform advanced feature generation and classification tasks, allowing for simultaneous detection and categorization of multiple labels from the multiplex image 104. For example, the GUI 308 may display results for label types such as ER, PR, and HER2, each marked with their corresponding labels as shown. Each detection model of the set of detection models may include traditional machine-learning models (e.g., SVM, KNN, random forests, or logistic regression), convolutional neural networks (e.g., LeNet, AlexNet, DenseNet, MobileNet, or EfficientNet), transfer learning models (e.g., VGG16, ResNet50, InceptionV3, or Xception), or other specialized models (e.g., U-Net or DeepLab).

[0070] FIG. 4 is a block diagram illustrating an exemplary architecture 400 for generating a set of semantic masks 410 for each label of multiple labels from the ground truth annotations 306 in accordance with some aspects of the present disclosure. The set of semantic masks 410 may be generated based on the ground truth annotations 306 highlighting the presence of multiple labels that differentiate them from background. The semantic masks 410 may include a semantic mask for each label of the multiple labels and one additional mask for the background representing areas without any biomarkers or a label type.

[0071] The exemplary architecture 400 may start with initialized masks 402, where each mask in a set of empty masks (one for each label of the multiple labels and an additional one for the background) may be initialized with zeros to signify no detected labels, and each mask corresponds to either one of the multiple labels or the background. Initialized masks 402 mayrefer to creating initial binary masks in image processing or computer vision, highlighting areas of interest in an original image. In medical imaging, the initialized masks 402 may be set up empty and later refined to highlight specific features like biomarkers, tumors, or cells.

[0072] Iterative biomarker processing may then refine initialized masks for detecting and classifying multiple labels. A boundary detector 404 may iteratively detect boundaries or define a circular region around predicted biomarker position on the set of ground truth annotations 306 within a specified radius. The radius of the circular region may be determined by a prior knowledge or the set of ground truth annotations 306, localizing the multiple labels within the biological specimen. A mask creator 406 may iteratively generate a set of binary masks for multiple labels, assigning a value of 1 to pixels inside the circular region (indicating label presence) and 0 outside the region (indicating absence), thereby localizing the label's position and accurately representing biomarker boundaries. The mask creator 406 may create a background mask by utilizing the set of binary masks to identify biomarker regions. The mask creator 406 may generate a background mask by using the set of binary masks to identify label regions, setting the pixels within the label regions to zero and assigning a value of one to all other pixels, thereby creating a binary mask for the background.

[0073] A mask overlaying unit 408 may iteratively merge the set of binary masks for the multiple labels with the background mask, distinguishing biomarker regions from non-biomarker regions, and producing the set of semantic masks 410. The semantic masks 410 may play a vital role for downstream analysis, such as evaluating label distribution and intensity, and enabling precise localization and detection of cellular features for diagnostic applications like cancer detection and treatment planning. The generated set of semantic masks 410 may be used to train the feature generator or fed into the boundary detector 404 as part of the iterative processing, enabling further refinement and enhancement of the semantic masks 410 after each iteration.

[0074] FIG. 5 illustrates an exemplary workflow 500 of the multi-label classification pipeline 108 for detection and classification of multiplex phenotypes in accordance with some aspects of the present disclosure. The multi-label classification pipeline 108 may include a pre-processor 502, a feature generator 504, a post-processor 508 and an annotator 510. The multiplex image 104 may undergo a series of pre-processing steps through the pre-processor 502 (may vary depending on the image type and model used), generating a pre-processed image for furtherdownstream analysis. The pre-processing steps may include resizing, normalization, contrast adjustment, background subtraction, noise reduction, and thresholding to improve an image’s quality and enhance the accuracy and reliability of the classification pipeline 108. Normalization may adjust the intensity to account for lighting or exposure variations, enabling them to have consistent features are across images. In addition, contrast adjustment techniques (e.g., histogram equalization or adaptive contrast enhancement) may help highlight key features (e.g., cell boundaries) by making them more distinct. Moreover, background subtraction may remove unwanted noise or irrelevant information, improving the visibility of vital features. Additionally, noise reduction methods (e.g., Gaussian, or median filtering) may smoothen the image and eliminate interference to enable a better classification. Binary conversion techniques like Otsu's method or adaptive thresholding may also be applied. Advanced pre-processing techniques may also be employed based on the complexity of the multiplex image, the number of markers examined, and the specific requirements of the labeling process.

[0075] Once pre-processing is complete, the pre-processed image may undergo segmentation by utilizing the feature generator 504, generating a set of interim signal maps 506. Each interim signal map of the set of interim signal maps 506 may include multiple portions or cell locations. Each portion may represent predictions for the portion indicating whether the multiple portions may align with the corresponding labels. Collectively, the interim signal maps 506 form a multidimensional representation of the multiplex image 104, such that each map displaying predicted distribution of a specific label of the multiple labels across the image patterns. The interim signal maps 506 may help in understanding the spatial relationships within each portion or cell location corresponding to various regions of interest, typically aligned with multiple labels, where each interim signal map image may represent a specific label. They may isolate and identify areas where label types may be present, enabling precise analysis. In some aspects, the feature generator 504 may also generate a probability for each portion or cell location corresponding to a label across the set of interim signal maps 506. The feature generator 504 may be trained using ground truths annotations 306 and the set of semantic masks 410 providing the correct labels for each pixel or segment, allowing the model to improve its ability to distinguish between different biomarkers. The feature generator 504 may be trained using a back-propagator 514, where one or more weights are assigned to each label. The weights for each label may then be adjusted and sent back to the feature generator 504 through the adjusted weights 516.

[0076] In some aspects, class imbalance, where some labels are more frequent than others, may cause the pipeline 108 to better classify common labels while underperforming on rare ones. To address this issue, the feature generator 504 may be trained with adjustable weights, assigning a weight bias to less frequent labels to improve the detection accuracy of rare labels. The weight bias may adjust semantic mask thresholds for better nucleus detection and resolve label conflicts by selecting the accurate label for each region. Assigning a weight bias may involve giving higher weights to samples that are rare in the training set, which improves the model's ability to detect underrepresented labels.

[0077] The interim signal maps 506 may undergo multiple post-processing steps in the postprocessor 508 to generate a set of cell-specific point locations. The post-processing steps may include creating a set of interim nuclei-signal maps for each label (e.g., ER-positive, ERnegative, or non-tumor label for the ER label type) by processing the interim signal maps 506. The interim nuclei-signal maps may identify the predicted locations of cells corresponding to each label, with nuclei appearing as distinct regions within the cells. For example, nuclei may appear as white dots (indicating the presence of a label or 1), while the background may appear as black dots (indicating the absence of a label or 0), resulting in a binary image. Nuclei locations may be detected using techniques like contour detection or connected component labeling (e.g., using ski mage, measure, lab el function in Python).

[0078] The detected nuclei locations, represented by centroids or enclosed within bounding boxes or ellipses, may be overlaid onto the interim signal maps 506. For visualization, a set of binary masks or the interim nuclei-signal maps may highlight the detected nuclei in white (or another color) against a black or transparent background. Alternatively, an RGB image may be generated, whereby nuclei positions corresponding to specific labels are marked using distinct markers, such as dots, crosses, or other symbols, all rendered in the same color. The highlighting may provide a clear and precise visualization of nuclei locations based on segmented cell images with an option to further refine images based on images’ quality.

[0079] The post-processing steps may also include generating the set of cell-specific point locations by processing at least two of the interim signal maps and / or interim nuclei-signal maps by utilizing one or more mapping functions. The processing may include identifying precise, spatially resolved locations within the multiplex image 104 that may correspond to portions thatare linked to multiple labels. By pinpointing the cells’ locations, the multi-label classification pipeline 108 may deliver relatively accurate insights into the distribution and concentration of biomarkers across the tissue, aiding in the development of relatively precise diagnostic tools and personalized treatment strategies, enabling clinicians to better assess how biomarkers are expressed at the cellular level within a tissue. The mapping functions used to generate cellspecific points may involve mathematical operations on data structures (e.g., matrices or arrays) with coefficients that may be adjusted based on criteria like signal intensity or distribution in the set of interim signal maps 506 or the set of nuclei-signal maps. The mathematical operations may help identify accurate locations for individual cells, improving the sensitivity and resolution of the model for diagnostic and therapeutic applications.

[0080] In some aspects, each cell location corresponding to a particular label may be assigned to a distinct color, creating an RGB image where nuclei positions for each label are rendered in one color providing a clear visualization of nuclei locations for the particular label. In some aspects, at least two of the interim signal maps and / or interim nuclei-signal maps may be used to generate the cell-specific point locations that may correspond to multiple labels corresponding to the same label type. For instance, combining signal maps for labels such as ER+ (estrogen receptor-positive), ER- (estrogen receptor-negative), or non-tumor for the ER label type. A label, which has the highest probability of being associated with a given cell location or portion, may be selected from the multiple labels within the same label type. By selecting the label with the highest probability, pipeline 108 may improve its ability to distinguish between various labels, enhancing precision of diagnostic, prognostic, and treatment planning methods. The selection may also support a relatively accurate mapping of the multiple labels, providing advanced diagnostic and personalized medical insights.

[0081] The post-processor 508 may apply thresholding to classify each interim signal or nuclei-signal map, predicting the presence of specific labels. Thresholding may be applied to probability values for each cell location or portion, distinguishing the multiple labels from the background. For example, a threshold of 0.7 may classify values that are above this threshold value as labels, and the values that are below this threshold value as the background, considering high-confidence predictions and consequently reducing the impact of noise. Thresholding may improve the accuracy of the pipeline 108 by focusing on reliable predictions.

[0082] Thresholding may also be vital for non-maximum suppression (NMS) by removing redundant detections and ensuring a cleaner output. After thresholding, NMS may eliminate overlapping or lower-confidence detections. Based on a predefined intersection over union (loU) threshold, NMS may retain confident predictions, reducing false positives, enhancing label clarity, and helping to identify the centroids of predicted nuclei. Additionally, NMS may enable the prediction of multiple labels, and label types associated with each nucleus or cell, offering deeper insights and improving the accuracy of the analysis.

[0083] Furthermore, the set of annotated images 112 may be generated by overlaying the cell-specific locations onto the multiplex image 104 by leveraging the annotator 510. Each annotated image in the set of annotated images 112 may correspond to a label type, such that multiple labels are marked according to a particular type. The annotator 510 may take the multiplex image 104 and overlay the identified features or the set of cell-specific locations (e.g., cell locations, biomarker presence) onto the original triplex image (or other base images).

[0084] The overlaying may highlight regions for multiple labels, often using distinct colorcoding or markers for each label. For example, multiple labels may be represented by red, green, or blue, ensuring visual distinction. For example, in an annotated image for an ER label type, red may represent “ER+,” yellow may represent “ER-,” and black marking may represent “nontumor” cell locations or portions. A similar classification may be applied for PR in another annotated image. For a third annotated image, the HER2 label type may be shown with various color-coded intensity levels: red for “HER22+”, maroon for “HER2 3+”, green for “HER2 1+”, yellow for “non-tumor” cell locations, and orange for “HER2-”. The color coding may provide better and clear visualization of the annotated images 112, allowing clinicians and researchers to easily distinguish between the multiple biomarkers that may be present in the biological specimen.

[0085] In some aspects, an annotation tool may be used to gather input from a user or an expert to manually or automatically assign labels to specific portions or cell locations in the multiplex image 104. The annotation tool may help highlighting areas with biomarkers, marking regions of interest, or adding textual descriptions. The tool may enable the user to associate multiple labels (e.g., ER-positive, ER-negative, or non-tumor for ER label type) with specific portions or cell locations. The tool may also be used to draw boundaries, add text labels, ormodify colors to emphasize some features. The resulting annotated images 112 may offer a comprehensive view of the spatial distribution of multiple labels, enabling clinicians and researchers to precisely localize multiple biomarkers within a tissue for analysis, diagnosis, or treatment planning. The annotated images 112 may also support research into the relationships between biomarkers, cellular structures, and tissue characteristics.

[0086] FIG. 6 illustrates an exemplary workflow 600 of the back-propagator 514 of the multi-label classification pipeline 108 in accordance with some aspects of the present disclosure. In some aspects, the feature generator 504 may be trained using a cross-entropy module 602 to optimize the model’s performance by minimizing an error between predicted labels and actual labels based on the ground truth annotations 306. The cross-entropy module 602 may be effective for classification tasks that may involve binary outcomes, such as determining whether a portion or cell location belongs to a specific label (e.g., tumor or non-tumor). The crossentropy module 602 may measure the difference between actual and predicted labels or probabilities, aiming to minimize this difference for more accurate predictions. The crossentropy module 602 may serve as a directional mechanism during training, enabling the feature generator 504 to learn effectively by adjusting its parameters or weights through back-propagator 514. By iteratively refining the network’s parameters based on the error, the model may enhance the accuracy of its predictions, resulting in a better alignment with the actual labels to improve the overall model performance.

[0087] The cross-entropy module 602 may provide a steep gradient when there is a relatively large error observed between predicted and actual labels, making it more sensitive to errors and helping the model converge faster during training. Compared to other loss functions, like mean squared error (MSE), binary cross-entropy may perform better for binary classification tasks, as MSE offers a gradual gradient, which may potentially slow down the learning process. The cross-entropy module 602 may use the interim signal maps 506 and a set of overlayed images to generate an output loss. The set of overlayed images may be generated by combining the multiplex image 104 with the set of semantic masks 410 using the overlay module 604. Adjusted weights 516 may then be generated by utilizing weight adjustment unit 606 based on the loss generated by the cross-entropy module 602. The adjusted weights 516 may be used to update the feature generator 504 by leveraging the back-propagator 514, refining the feature generator 504,thereby enhancing the performance and accuracy of the multi -label classification pipeline 108 over a period of time.

[0088] FIG. 7 illustrates a set of singleplex images (corresponding to ER, PR and HER2) along with a set of semantic masks in accordance with some aspects of the present disclosure. The set of label types (i.e., ER, PR and HERZ) may play a significant role in detecting various hormone related conditions, including breast cancer, endometriosis, osteoporosis, fibroids, cardiovascular diseases, polycystic ovary syndrome (PCOS), menstrual disorders, neurological diseases, and some thyroid disorders. ER and PR labels may produce three possible labels: ER+ (estrogen receptor positive), ER- (estrogen receptor negative), and non-tumor, with a similar classification for PR. On the other hand, HER2 may generate five possible labels based on the intensity of the marker: HER2+, HER3+, HER1+, non-tumor, and HER2-. The multiple labels may provide meaningful insights into the molecular characteristics of the tissues, enabling more accurate diagnosis and treatment planning for the mentioned conditions.

[0089] Each semantic mask of the set of semantic masks 410 may visualize the presence of multiple labels on the singleplex images, which may be generated by the exemplary architecture 400. For example, block 702 displays corresponding singleplex image for ER label type and its corresponding semantic masks (ER+, ER-, and non-tumor) are displayed. Similarly, for the PR label type, block 704 displays corresponding singleplex image for PR label type and semantic masks corresponding to the label type (i.e., PR+, PR-, and non-tumor) are shown. Block 706 displays for HER2, various masks for HER2 1+, HER2 2+, HER2 3+, HER2-, and non-tumor. Generating the set of semantic masks 410 for each label type may allow for detailed and precise analysis of the multiple labels, improving the understanding of their roles in various biological conditions.

[0090] FIG. 8 illustrates an input triplex image with the set of semantic masks 410 and the set of interim signal maps for three label types (ER, PR and HER2) generated by the multi-label classification pipeline 108. The set of semantic masks 410 may be utilized to train the feature generator 504, and the set of interim signal maps 506 may be generated by the feature generator 504, making the set of interim signal maps 506 an output derived from the semantic masks 410. Each label type may correspond to multiple labels, which may be combined to aid in detecting one or more biological conditions. For example, the multiple labels may include three for ER(ER+, ER-, and non-tumor), three for PR (PR+, PR-, and non-tumor), and five for HER2 (HER2 1+, HER2 2+, HER2 3+, HER2-, and non-tumor). Twelve semantic masks may be generated for the multiple labels, along with an additional mask for the background.

[0091] Block 802 displays the triplex (i.e., the multiplex image 104 fed to the pipeline 108 as input) that may be used to generate the set of semantic masks 410 and the set of interim signal maps 506. Block 804 displays the set of semantic masks 410 corresponding to the PR (progesterone receptor) label type on the right side. On the left side, the corresponding interim signal maps for the PR label type are displayed. Like block 804, block 806 displays the set of semantic masks 410 for the ER (estrogen receptor) label type on the right side. The interim signal maps 506 related to the ER label type are displayed on the left side. Additionally, in block 808, both the semantic masks 410 and the interim signal maps 506 corresponding to the HER2 label type may be displayed. Each mask and map may be labeled with a text to identify specific features or various characteristics that are detected by the feature generator 504, thereby offering a comprehensive overview of the HER2 -related features within the multiplex image 104.

[0092] In some aspects, the set of semantic masks 410 may be referred to as a reference set of interim nuclei-signal maps, as the semantic masks 410 may include the corresponding set of nuclei’s that are generated by the feature generator 504. The reference set of nuclei-signal maps may be compared with the set of nuclei-signal maps that are derived from the interim signal maps 506 to assess the model’s performance or loss function to compute the accuracy of the model. The comparison may demonstrate how the application of different masks and thresholds may yield distinct signal representations for each label, enabling a relatively precise identification of specific cellular structures and biomarkers.

[0093] FIG. 9 illustrates an exemplary output 900 of an input triplex image having markers for three label types (ER, PR and HER2) by the multi-label classification pipeline 108 in accordance with some aspects of the present disclosure. The triplex image 902 may be generated using the image generation system 102, as shown in the exemplary block diagram 200. Each annotated image may represent output for a specific label type, such that distinct markers (circles, dots in multiple colors) for each of the multiple labels correspond to a particular label type. For example, in an annotated image 904, each cell may be labeled according to its ER status, with red representing “ER+”, yellow representing “ER-”, and black marking “non-tumor”cells or portions. A similar classification may be applied for PR in another annotated image 906. For a third annotated image 908, the HER2 biomarker output may be shown with various color- coded intensity levels: red for “HER2 2+”, maroon for “HER2 3+”, green for “HER2 1+”, yellow for “non-tumor” cells, and orange for “HER2-”.

[0094] The distinct markers may be generated based on various intensity levels that may be derived from processing the set of semantic masks 410 (corresponding to each label type) by the feature generator 504. Depending on the intensity levels of the markers, more than three classification types for ER or PR or more than five classification types for HERZ may be detected in some aspects of the present disclosure. The increased sensitivity may be identified either by an expert in the field or through advanced imaging technologies that may be capable of distinguishing finer intensity variations, thereby enhancing the accuracy of the classification process.

[0095] FIG. 10 illustrates exemplary plots 1000 displaying one or more performance metrics graphs on training and validation sets in accordance with some aspects of the present disclosure. The exemplary plots 1000 may display multiple graphs that measure performance for the multilabel classification pipeline 108 applied to the triplex image 902. The triplex image 902 may comprise markers from ER, PR and HER2 label types. Graph 1002 displays the validation accuracy in relation to multiple steps. The validation accuracy measures how well the feature generator 504 may perform on a separate subset of data known as a validation set. Unlike training sets, the validation set may not be used to train the feature generator 504 and may be utilized to predict the model's performance on unseen data. The validation accuracy may be calculated by determining the percentage of correct labels on the validation set. The validation accuracy may identify overfitting (superior performance on a training dataset but an inferior performance on the validation dataset) or underfitting (inferior performance on both training and validation datasets). Monitoring validation accuracy might be useful for adjusting generalized performance of the feature generator 504 to new unseen datasets in the real world. In Graph 1002, the validation accuracy may reach approximately 0.85, indicating an acceptable level of performance for real world applications. Notably, a significant improvement in accuracy from 0.6 to 0.8 highlights the model’s ability to generalize the learning process.

[0096] Moreover, a graph 1004 displays a training loss in relation to the multiple steps. The training loss measures how well the model may be performing on the training dataset. The training loss may be calculated by determining the difference between predictions labels and true labels based on a chosen loss function (e.g., cross-entropy for classification). A lower training loss indicates that predictions based on the feature generator 504 are closer to the actual values in the training dataset. Monitoring training loss during the training process may help assess how effectively the model is learning from the training data. Training loss may decrease over time as the ability of the feature generator 504 to generalize improves. However, if the training loss decreases significantly while the validation loss remains high, it may signal an overfitting, such that the model performs well on the training dataset but may fail to generalize to unseen validation datasets. In graph 1004, the training loss reaches approximately 0.4, indicating an acceptable level of performance for real world applications. Notably, a significant decrease in the training loss from around 1.2 to 0.4 over the training steps reflects a significant improvement in the model’s performance.

[0097] Additionally, a graph 1006 displays a validation loss in relation to the multiple steps. The validation loss may be used to evaluate the performance of the feature generator 504 on the validation set. Unlike accuracy (measuring the proportion of correct predictions), validation loss measures how well a predicted label may align with the actual outcomes based on the chosen loss function (e.g., cross-entropy). A lower validation loss may indicate better performance, as it means the model's predictions are closer to the true values in the validation data. Monitoring validation loss during training might be relevant for identifying issues such as overfitting, such as the feature generator 504 may perform better on the training data but poorly on the unseen validation data. By minimizing validation loss, the feature generator 504 may be trained to generalize, leading to improved performance on real world datasets. In Graph 1004, the validation loss reaches approximately 0.3, indicating an acceptable level of performance for real world applications. Notably, a significant decrease in validation loss from 0.5 to 0.3 over the training steps reflects a significant improvement in the performance of the model.

[0098] The validation accuracy may serve as a better metric to evaluate the overall performance of the model in detecting ER / PR / HER2 biomarkers. To calculate the accuracy, ER, PR, and HER2 may be evaluated separately, with weights assigned as 0.3 for ER, 0.3 for PR, and0.4 for HER2. The optimal accuracies for ER, PR, and HER2 are 0.89, 0.93, and 0.80, respectively. As a result, the optimal comprehensive accuracy is 0.86.

[0099] FIG. HA shows confusion matrices illustrating performance of a set of detection models 304 at training for ER, PR, and EIER2 label types in accordance with some aspects of the present disclosure. The set of singleplex images 302 corresponds to three label types (ER, PR and HER2) that may be produced by the image generation system 102 and are fed to the detection models 304. Each image in the set of singleplex images 302 may be characterized with a label of a specific label type (e.g., ER, PR, or HER2). A confusion matrix is a table that may be used to evaluate the performance of a classification model. The confusion matrix may display true positives, true negatives, false positives, and false negatives, that may assess the alignment of the model's predictions with true results. The rows in the confusion matrix may represent true labels, and the columns may represent predicted labels, providing insight into accuracy, precision, recall, and other key metrics, while highlighting potential areas of error or misclassification.

[0100] The multiple confusion matrices 1102 may include three tables for training with each set of a label type of the three label types. To illustrate, consider the results generated by a detection model 304 corresponding to ER (estrogen receptor) label type, represented by 1102-A. According to the confusion matrix 1102-A, 20,974 cells were correctly marked as "ER+", while 3,651 cells were incorrectly labeled as "ER-" despite being part of the "ER+" group. Additionally, 2,758 cells were incorrectly marked as "non-tumor" instead of "ER+". Similarly, results for PR (progesterone receptor) label type may be represented by a confusion matrix 1102- B and results for HER2 (Human epidermal growth factor receptor 2) may be represented by a confusion matrix 1102-C showing similar patterns of correct and incorrect classifications for each label of the multiple labels of a label type. As a result, the overall accuracy is 0.86 for the ER label type, 0.91 for the PR label type, and 0.80 for the HER2 label type, respectively. Moreover, the balanced accuracy is 0.89 for ER, 0.86 for PR, and 0.77 for HER2, respectively. The overall accuracy and the balanced accuracy metrics may reflect the ability of the set of detection models 304 to correctly classify multiple labels categories within the label types during the training phase, helping to assess the performance and robustness of the detection models 304.

[0101] FIG. 1 IB shows confusion matrices illustrating performance of a set of detection models 304 at validation set for ER, PR, and HER2 label types in accordance with some aspects of the present disclosure. The set of singleplex images 302 corresponding to three label types (ER, PR and HER2) that may be produced by the image generation system 102 may be fed to the detection models 304. Each image in the set of singleplex images 302 may be characterized with a label of the multiple labels that may correspond to a specific label type (e.g., ER, PR, or HER2) of the set of label types.

[0102] The multiple confusion matrices 1104 may include three tables for validation phase with each set corresponding to a label type of the three label types. To illustrate, consider the results generated by a detection model of the set of detection models 304 corresponding to ER (estrogen receptor) label type, represented by 1104- A. According to the confusion matrix 1104- A, 5800 cells were correctly marked as "ER+", while 1050 cells were incorrectly labeled as "ER- " despite being part of the "ER+" group. Additionally, 351 cells were incorrectly marked as "non-tumor" instead of "ER+". Similarly, results for PR label type are represented by 1104-B and results for HER2 are represented by 1104-C. As a result, the overall accuracy is calculated to be 0.88 for ER, 0.94 for PR, and 0.80 for HER2, respectively. Additionally, the balanced accuracy is 0.90 for ER, 0.95 for PR, and 0.75 for HER2, respectively. The overall accuracy and balanced accuracy metrics may reflect the ability of the set of detection models 304 to correctly classify multiple labels categories within the label types during the validation phase, helping to assess the performance and robustness of the detection models 304.

[0103] FIG. 12 shows a confusion matrix illustrating performance of the multi-label classification pipeline 108 from the triplex image 902 in accordance with some aspects of the present disclosure. The triplex image 902, produced by the image generation system 102 may have markings from three distinct label types (i.e., ER, PR, and HER2). The triplex image 902 may be processed through the multi-label classification pipeline 108 generating three annotated images, each corresponding to a different label type. The annotated images 112 may provide predictions for each region of the image, corresponding to the multiple labels (for example, labels “ER+”, “ER-” or “non-tumor” for ER label type; “PR+”, “PR-” or “non-tumor” for the label PR+; and “HER2 2+”, “HER2 3+”, “HER2 1+”, “non-tumor” or “HER2 -” for HER2 label type). The confusion matrix 1202 may be generated to represent the classification results for each label type, offering a detailed comparison of predicted versus true labels across multiplelabels. The confusion matrix 1202 may help assess the overall accuracy and effectiveness of the multi-label classification pipeline 108 by measuring the true positive, false positive, true negative, and false negative predictions for each label.

[0104] The confusion matrix 1202 may include a set of classes generated based on multiple labels. The set of classes may include one for the "non-tumor" class, and the remaining twelve may correspond to various combinations of labels, one from each of the label types (such as ER, PR, and HER2). For example, 6403 cells may be correctly marked as “non-tumor”, a class from the set of classes in the matrix may be represented as a combination of <"ER-", "PR+", "HER2 l+>" labels. In this class, 843 cells may be correctly identified based on the combination of specified labels. The confusion matrix 1202 may also highlight misclassified cells, with errors stemming from various label combinations across different label types. For example, 127 labels are incorrectly marked as <"ER-", "PR-", "HER2 l+">, while 119 labels are marked as <"ER-", "PR-", "HER2 l+">. The confusion matrix 1202 may provide detailed insights into the performance of multi-label classification pipeline 108 across multiple label combinations, highlighting areas of misclassification. The insights may be meaningful for refining and optimizing the feature generator 504 for improved accuracy.

[0105] FIG. 13 illustrates an exemplary flowchart 1300 for detection and classification of the multiplex phenotype in accordance with some aspects of the present disclosure. The blocks in flowchart are illustrated in a specific order, while the order may be modified, for example, some blocks may be performed before others, and some blocks may be performed simultaneously. The blocks may be performed in hardware or software or a combination thereof. At block 1302, the process flow may begin by accessing the multiplex image 104, generated by the image generation system 102, depicting the biological specimen that may be labeled with the multiple markers. Multiplexing multiple markers may identify various biomarkers and cellular components, enabling the visualization and comprehensive analysis of spatial relationships, thereby providing information for diagnosing biological conditions and creating personalized treatment plans.

[0106] At block 1304, a set of label types may be identified such that each label type may correspond to a marker of the multiple markers corresponding to the multiplex image 104. The set of label types may be used for categorizing biological features or molecular characteristics,representing specific marker expressions, such as the presence, expression level, or localization of target molecules. The label types may correspond to proteins, such as a cellular-membrane protein (e.g., HER2) or a cell nucleus protein (e.g., ER or PR), providing detailed insights into the cellular composition and functionality within the tissue sample. At block 1306, for each label type of the set of label types, multiple labels may be identified. The multiple labels may represent a prediction of a cell (e.g., negative, intermediate confidence, or positive expression for each label type, or a depiction of a non-tumor biological component). A set of interim signal maps 506 may be generated at block 1308 for each label by processing the multiplex image 104, using the machine-learning model, or the feature generator 504. Each interim signal map may include predictions for each portion or cell location, indicating whether it aligns with the corresponding label. The indication may help evaluate pixel alignment, enabling more accurate mapping of cellular features and biomarker distributions.

[0107] At block 1310, the set of interim nuclei-signal maps may be generated by processing the interim signal maps for each label (e g., ER-positive, ER-negative, or non-tumor). The set of interim nuclei-signal maps may identify portions or cell locations, predicted to correspond to each label for nuclei locations, which may be detected using methods like contour detection or component labeling. Nuclei locations may be marked on the maps as centroids, or bounding boxes, or ellipses, or a visualization in binary or RGB images, such that the detected nuclei may be highlighted for a clear representation. At block 1312, a set of cell-specific point locations may be generated by processing at least two interim signal maps and / or interim nuclei-signal maps using one or more mapping functions. The one or more mapping functions may be based on mathematical operations on data structure like matrices to adjust data based on signal intensity or distribution, enhancing sensitivity and resolution of the feature generator 504. The mapping functions may help identify precise locations within the multiplex image 104 that may correspond to individual cells or specific regions, allowing for relatively accurate insights into biomarker distribution and concentration. Each cell-specific location may be assigned to a distinct color for clear visualization in an RGB image. In some aspects, interim signal maps for multiple labels may correspond to the same label type (e.g., ER+, ER-, non-tumor) and may be processed together, selecting the label with the highest probability for improving diagnostic and treatment planning accuracy.

[0108] At block 1314, an output or the set of annotated images 112 may be generated for each cell-specific point location, indicating at least one predicted label corresponding to that location. The set of annotated images 112 may be created by overlaying the cell-specific locations onto the multiplex image 104 utilizing the annotator 510. Each label may be represented by distinct colors on the multiplex image 104, corresponding to specific biomarkers that may be assigned to each portion (e.g., cell location, pixel, or segmented region). At block 1316, the output may be used to identify one or more biological conditions that may be associated with the different markers, such as cancer, osteoporosis, fibroids, autoimmune diseases, hormone-related conditions, cardiovascular diseases, PCOS, menstrual disorders, neurological diseases, and some thyroid disorders.

[0109] FIG. 14A illustrates a side-by-side comparison of overlay estimations of PR and ER on a first triplex image and synthetic PR and ER singleplex images in accordance with some aspects of the present disclosure. Images on the left side are generated by overlaying the cellspecific point locations generated by the multi-label classification pipeline 108 on a first triplex image for the PR 1402 and ER 1406 label types, respectively. While the images on the right side are generated by overlaying the cell-specific point locations on synthetic images for PR 1404 and ER 1408 images, respectively. The synthetic images may be singleplex images generated by color unmixing technique, that unmix the triplex image into its constituents singleplex images.

[0110] FIG. 14B illustrates a side-by-side comparison of overlay estimations of PR and ER on a second triplex image and augmented images in accordance with some aspects of the present disclosure. Images on the left side are generated by overlaying the cell-specific point locations generated by the multi-label classification pipeline 108 on a second triplex image for the PR 1410 and ER 1414 label types, respectively. While the images on the right side are generated by overlaying the cell-specific point locations on an augmented image for PR 1412 and ER 1416, respectively. The augmented image may be generated by applying various transformations on the triplex image to increase the number training examples.[0U1] FIG. 15 illustrates a side-by-side comparison of overlay estimations of HER2 on a third triplex image and an augmented image in accordance with some aspects of the present disclosure. Image on the left side may be generated by overlaying the cell-specific point locations generated by the multi-label classification pipeline 108 on a third triplex image onHER2 1502 image. While the image on the right side may be generated by overlaying the cellspecific point locations on an augmented image 1504 that may be generated by applying various transformations on the triplex image to increase the number training examples.

[0112] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0113] Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0114] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

[0115] The present description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the present description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0116] Specific details are given in the present description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without specific details. For example, circuits, systems, networks, processes, and othercomponents may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

CLAIMSWhat is claimed:

1. A computer-implemented method comprising: accessing a multiplex image that depicts a biological specimen labeled with multiple different markers, wherein the multiple different markers identify different biomarkers or cellular components relative to each other; identifying a set of label types, wherein each label type of the set of label types corresponds to a marker of the multiple different markers; for each label type of the set of label types: identifying multiple labels, wherein each of the multiple labels represents a prediction of: a negative expression for the marker corresponding to the label; a given intermediate confidence of expression for the marker corresponding to the label; a positive expression for the marker corresponding to the label; or a depiction of a biological component that is not a tumor cell; generating, for each label of the multiple labels, a set of interim signal maps by processing the multiplex image using a machine-learning model associated with the label type, wherein each interim signal map of the set of interim signal maps includes, for each portion of one or more portions of the interim signal map, a prediction as to whether the portion is consistent with the label; generating, for each label of the multiple labels, a set of interim nuclei- signal maps that corresponds to the set of interim signal maps, wherein each interim nuclei-signal map of the set of interim nuclei-signal maps identifies a representative location of a cell that is predicted to correspond to the label; generating a set of cell-specific point locations by processing at least two of the set of interim signal maps or at least two of the set of interim nuclei-signal maps using one or more mapping functions; generating an output that includes, for each point location of at least some of the set of cell-specific point locations, an indication of at least one label that is predicted tocorrespond to the point location, wherein the at least one label includes a label of the set of labels corresponding to one or more label types; and availing the output to identify one or more biological conditions associated with the multiple different markers.

2. The computer-implemented method of claim 1, wherein, for a label type of the set of label types: the machine-learning model associated with the label type is trained to assign bias weights to training elements corresponding to a first label of the multiple labels relative to a second label of the multiple labels.

3. The computer-implemented method of claim 1, wherein the machinelearning model associated with at least one label type was trained using a binary cross-entropy loss function.

4. The computer-implemented method of claim 1, wherein the one or more mapping functions used to generate the set of cell-specific point locations includes a matrix or an array that includes a coefficient for each element of the matrix of the array.

5. The computer-implemented method of claim 1, wherein the set of label types includes at least three label types.

6. The computer-implemented method of claim 1, wherein at least one label type of the set of label types includes a label type corresponding to a particular cellular- membrane protein; and wherein at least another label type of the set of label types includes a label type corresponding to a particular cell nucleus protein.

7. The computer-implemented method of claim 1, wherein the machinelearning model includes a UNet model.

8. A system comprising: one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations including: accessing a multiplex image that depicts a biological specimen labeled with multiple different markers, wherein the multiple different markers identify different biomarkers and / or cellular components relative to each other; identifying a set of label types, wherein each label type of the set of label types corresponds to a marker of the multiple different markers; for each label type of the set of label types: identifying multiple labels, wherein each of the multiple labels represents a prediction of: a negative expression for the marker corresponding to the label; a given intermediate confidence of expression for the marker corresponding to the label; a positive expression for the marker corresponding to the label; or a depiction of a biological component that is not a tumor cell; generating, for each label of the multiple labels, a set of interim signal maps by processing the multiplex image using a machine-learning model associated with the label type, wherein each interim signal map of the set of interim signal maps includes, for each portion of one or more portions of the interim signal map, a prediction as to whether the portion is consistent with the label; generating, for each label of the multiple labels, a set of interim nuclei-signal maps that corresponds to the set of interim signal maps, wherein each interim nuclei-signal map of the set of interim nuclei-signal maps identifies a representative location of a cell that is predicted to correspond to the label;generating a set of cell-specific point locations by processing at least two of the set of interim signal maps and / or at least two of the set of interim nuclei-signal maps using one or more mapping functions; generating an output that includes, for each point location of at least some of the set of cell-specific point locations, an indication of at least one label that is predicted to correspond to the point location, wherein the at least one label includes a label of the set of labels corresponding to one or more label types; and availing the output to identify one or more biological conditions associated with the multiple different markers.

9. The system of claim 8, wherein, for a label type of the set of label types: the machine-learning model associated with the label type is trained to assign bias weights to training elements corresponding to a first label of the multiple labels relative to a second label of the multiple labels.

10. The system of claim 8, wherein the machine-learning model associated with at least one label type was trained using a binary cross-entropy loss function.

11. The system of claim 8, wherein the one or more mapping functions used to generate the set of cell-specific point locations includes a matrix or an array that includes a coefficient for each element of the matrix of the array.

12. The system of claim 8, wherein the set of label types includes at least three label types.

13. The system of claim 8, wherein at least one label type of the set of label types includes a label type corresponding to a particular cellular-membrane protein; and wherein at least another label type of the set of label types includes a label type corresponding to a particular cell nucleus protein.

14. The system of claim 8, wherein the machine-learning model includes aUNet model.

15. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of operations comprising: accessing a multiplex image that depicts a biological specimen labeled with multiple different markers, wherein the multiple different markers identify different biomarkers and / or cellular components relative to each other; identifying a set of label types, wherein each label type of the set of label types corresponds to a marker of the multiple different markers; for each label type of the set of label types: identifying multiple labels, wherein each of the multiple labels represents a prediction of: a negative expression for the marker corresponding to the label; a given intermediate confidence of expression for the marker corresponding to the label; a positive expression for the marker corresponding to the label; or a depiction of a biological component that is not a tumor cell; generating, for each label of the multiple labels, a set of interim signal maps by processing the multiplex image using a machine-learning model associated with the label type, wherein each interim signal map of the set of interim signal maps includes, for each portion of one or more portions of the interim signal map, a prediction as to whether the portion is consistent with the label; generating, for each label of the multiple labels, a set of interim nuclei- signal maps that corresponds to the set of interim signal maps, wherein each interim nuclei-signal map of the set of interim nuclei-signal maps identifies a representative location of a cell that is predicted to correspond to the label; generating a set of cell-specific point locations by processing at least two of the set of interim signal maps and / or at least two of the set of interim nuclei-signal maps using one or more mapping functions;generating an output that includes, for each point location of at least some of the set of cell-specific point locations, an indication of at least one label that is predicted to correspond to the point location, wherein the at least one label includes a label of the set of labels corresponding to one or more label types; and availing the output to identify one or more biological conditions associated with the multiple different markers.

16. The computer-program product of claim 15, wherein, for a label type of the set of label types: the machine-learning model associated with the label type is trained to assign bias weights to training elements corresponding to a first label of the multiple labels relative to a second label of the multiple labels.

17. The computer-program product of claim 15, wherein the machine-learning model associated with at least one label type was trained using a binary cross-entropy loss function.

18. The computer-program product of claim 15, wherein the one or more mapping functions used to generate the set of cell-specific point locations includes a matrix or an array that includes a coefficient for each element of the matrix of the array.

19. The computer-program product of claim 15, wherein the set of label types includes at least three label types.

20. The computer-program product of claim 15, wherein at least one label type of the set of label types includes a label type corresponding to a particular cellular- membrane protein; and wherein at least another label type of the set of label types includes a label type corresponding to a particular cell nucleus protein.

Images