Multi-label pipeline for multiplex cell / phenotype detection and classification
By processing multiple images through a multi-label classification pipeline and using a machine learning model to generate mid-term signal maps and cell-specific point locations, the spatial inconsistency and class imbalance problems in multi-marker images are solved, achieving efficient multi-marker detection and classification, and improving the accuracy and efficiency of cancer diagnosis.
Patent Information
- Application Number
- CN202580029219.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-22
- Filing Date
- 2025-04-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies suffer from spatial inconsistency, class imbalance, and computational complexity when processing multi-label images with multiple markers, making it difficult to effectively train robust multi-classification models. This is especially true when data is scarce and spatial inconsistency exists in cell nuclei, leading to a decrease in detection and classification accuracy.
A multi-label classification pipeline is adopted, which uses machine learning models such as CNN and DNN to generate mid-term signal maps and cell-specific point locations through preprocessing, feature generation and post-processing steps. Combined with mapping functions and weight adjustment, it solves class imbalance and improves detection and classification accuracy.
This technology enables the simultaneous detection and classification of multiple biomarkers in a single step, improving diagnostic efficiency and accuracy, simplifying model management, and enhancing the accurate diagnosis of complex diseases such as cancer.
Smart Images

Figure CN123125624A_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims the benefit and priority of U.S. Provisional Patent Application 63 / 637,264, filed April 22, 2024, the entire contents of which are incorporated herein by reference for all purposes. Background Technology
[0003] Digital pathology enhances diagnosis and treatment by improving tissue analysis, detecting biomarkers, supporting personalized treatment, and increasing the efficiency of pathologists through AI-based computational analytics. By digitizing glass slides to generate digital pathology images (e.g., single- or multiple images), digital pathology enables remote slide review (remote pathology). Multiple images capture a range of signals representing multiple biomarkers in a single scan, allowing for assessment of disease progression, target identification, response prediction, and support for medical research and personalized treatment by revealing a patient's cancer profile. Multiple images can be used to train machine learning models to automatically identify patterns or predict outcomes from histological data, thereby enhancing diagnostic efficiency and reducing human error. As the number of biomarkers in digital pathology images increases, training a single model to make accurate predictions becomes increasingly complex. More biomarkers may require larger models with additional parameters and larger datasets. Furthermore, data imbalance (where some biomarkers are observed more frequently than others) can pose further challenges to obtaining datasets that enable models to effectively detect and differentiate all labels, including less common ones.
[0004] One approach could involve training separate models for each biomarker. However, merging predictions from multiple models can lead to spatial inconsistencies (e.g., incorrect cell or nucleus placement due to a lack of consistent integration) or mismatched results, increasing computational complexity. Furthermore, this approach can complicate the learning of label dependencies and exacerbate class imbalance, especially when some labels rarely or never appear simultaneously in the same region. In this case, there are more 0s (pixels without the biomarker) than 1s (pixels with the biomarker). Class imbalance can cause model bias, whereby the model may predict 0 (i.e., no biomarker) relatively more frequently. Another approach could involve training a multi-class model using separate models (e.g., one model for multiple biomarkers and another for a single biomarker) or using a single model for all biomarkers. However, this approach can introduce spatial inconsistencies, and limited data for each specific class can negatively impact the model's predictive accuracy. These inconsistencies make merging the outputs of multiple models into a single, reliable multi-label prediction output challenging. While training such models may be feasible when the number of categories is small, the challenge can increase significantly with the number of categories (e.g., if the number of categories is 10, 20, or even 40), making it difficult to find enough data points for each label. This poses a significant challenge to training robust multi-class classification models. Both approaches can encounter scalability issues, as combining results from multiple models and markers across a wide range of labels for accurate classification can be challenging, especially when data samples are limited and spatial inconsistencies exist in structures such as cell nuclei. In conclusion, a more efficient and advanced model is still needed that can effectively address the challenges of data scarcity and classification complexity in multi-label tasks involving multiple markers. Summary of the Invention
[0005] Some aspects of this disclosure relate to techniques for implementing a multi-label classification pipeline for multiple phenotypic detection and classification. Multiple phenotypes can correspond to multiple images of a biological sample, generated by an image generation system and labeled with multiple markers. Each of the multiple markers can be a biological indicator or biomarker corresponding to a cell membrane protein (e.g., HER2) or a specific nuclear protein (e.g., ER or PR) identified by a tag type from a set of tag types. For each tag type, multiple tags can be identified (e.g., ER positive, ER negative, or non-tumor for the ER tag type), representing expression or prediction for each cell location or region (such as negative, intermediate, positive, or non-tumor cells).
[0006] Multi-image processing can be leveraged using a multi-label classification pipeline to generate an annotated image ensemble. This process can begin by analyzing the multi-images using a preprocessor to generate pre-processed images, enhancing image quality for further analysis. A set of interphase signal maps can be generated by processing the pre-processed images using a machine learning model or feature generator, each signal map corresponding to a label (e.g., ER positive, PR negative, HER2+, or non-tumor). Together, the interphase signal maps constitute a multidimensional representation of the multi-images. Each interphase signal map can predict the distribution of a specific label across the multi-images, indicating label consistency at each cell location or part. Interphase signal maps can aid in understanding spatial patterns and predictions of multiple labels within the multi-images. In some aspects, the feature generator can produce a set of probability matrices for the interphase signal maps, corresponding to a specific label among the multiple labels. The probability matrices in the set corresponding to the specific label can determine the likelihood that each part or cell location belongs to that specific label. The combination of the interphase signal map set and the probability matrix set can provide detailed classification and localization, improving the accuracy of label identification and segmentation across the pipeline.
[0007] Interim signal maps can undergo various post-processing steps to generate a set of cell-specific point locations. Post-processing steps may include generating a set of interim nuclear signal maps for each label (e.g., for ER label types: ER-positive, ER-negative, or non-tumor). The set of interim nuclear signal maps can identify the predicted locations of nuclei corresponding to multiple labels. Nucleus locations can be detected using methods such as contour detection or connected component labeling, and can be represented by centroids or enclosed by bounding boxes or ellipses. Predicted locations can be overlaid on the interim signal map or binary image to highlight their detection. For visualization, a binary mask or RGB image can be generated, labeling detected nuclei with different symbols or colors, thus clearly visualizing nucleus locations based on the interim signal map.
[0008] Cell-specific point locations can be generated by processing at least two interphase signal maps or interphase nuclear signal maps using one or more mapping functions. Processing can identify precise, spatially resolved locations within multiple images, which may correspond to a single cell location or a portion of multiple labels. By correctly identifying these locations, multi-label classification pipelines can provide meaningful insights into the distribution and concentration of multiple labels, thus aiding in the development of more accurate diagnostic tools and personalized treatment strategies. Mapping functions can apply mathematical operations to data structures such as matrices or arrays to adjust the data based on factors such as signal strength or distribution, thereby enhancing the sensitivity and resolution of pipelines used for diagnostic and therapeutic applications. In some aspects, each cell-specific location corresponding to a particular label can be assigned a different color, creating an RGB image where nuclear locations for each label are rendered in the same color, providing a clear visualization of nuclear locations. Cell-specific point locations, which may correspond to multiple labels within the same label type, can be generated using at least two interphase signal maps or nuclear signal maps. For example, tags ER+ (estrogen receptor positive), ER- (estrogen receptor negative), and non-tumor tags can be combined to generate signaling cell-specific dot maps for ER tag types.
[0009] Annotated image sets can be generated by overlaying a set of cell-specific locations onto multiple images. Each label from multiple tags can be represented by a different color, with each annotated image corresponding to a specific tag type. Each tag associated with a specific tag type can be assigned to its corresponding cell location or portion on the multiple images, allowing for better visual differentiation. The annotated image set (i.e., the output) can be used to identify one or more biological conditions associated with multiple different markers or tag types. The feature generator can leverage various machine learning models, including convolutional neural networks (CNNs), deep neural networks (DNNs), or specialized architectures (e.g., UNet or VNet), trained on labeled data to automatically predict cell boundaries corresponding to multiple target labels and segment multiple images, thereby generating a set of mid-term signal maps. Machine learning models enable accurate spatial predictions in image segmentation through training with a binary cross-entropy loss function, allowing multi-label classification pipelines to accurately label regions in multiple images, improving diagnostic and marker detection accuracy.
[0010] The feature generator can be trained using pre-trained detection models for each label type, which generate ground truth annotations for label presence, location, and boundaries based on single-image datasets with specific markers. These annotations can be used for training, validation, and evaluation to improve label identification in multiple images, with results displayed on a user-friendly GUI. Semantic masks can be generated to highlight labels and distinguish them from the background, where adjustable weights address class imbalance by increasing sensitivity to less common labels and improving accuracy. Binary cross-entropy can be used as the loss function, enabling faster convergence and weight adjustment via backpropagation. Mid-signal maps and overlaid images help compute the loss and address class imbalance, thus enhancing model performance over time.
[0011] This disclosure enables clinicians or healthcare providers to make relatively accurate and comprehensive diagnoses, especially for complex diseases such as cancer. By using a multi-label classification pipeline, the requirement to merge predictions for multiple biomarkers can be eliminated, allowing for simultaneous detection and classification in a single step. The pipeline enhances efficiency and accuracy by simplifying model management and addressing challenges such as class imbalance and integration of spatially dissimilar phenotypes.
[0012] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.
[0013] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium includes instructions configured to cause one or more data processors to perform part or all of one or more methods or processes disclosed herein.
[0014] In some embodiments, a system is provided that includes one or more means for performing some or all of the methods or processes disclosed herein.
[0015] The terms and expressions used are descriptive rather than restrictive, and in using such terms and expressions, no equivalent of the features shown and described or any part thereof is intended to be excluded. However, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, those skilled in the art can employ modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. Attached Figure Description
[0016] This patent or application document contains at least one color drawing. Upon request and payment of the necessary fees, the official authority will provide a copy of this patent or application disclosure with (one or more) color drawings. This disclosure is described in conjunction with the drawings:
[0017] Figure 1 illustrates an exemplary block diagram for multi-label detection and classification of multiple phenotypes according to some aspects of this disclosure.
[0018] Figure 2 illustrates an exemplary block diagram for generating digital pathology images according to some aspects of this disclosure.
[0019] Figure 3 is a block diagram illustrating an exemplary architecture for generating ground truth annotations from a set of detection models, according to some aspects of this disclosure.
[0020] Figure 4 is a block diagram illustrating an exemplary architecture for generating a set of semantic masks for each tag from ground truth annotations, according to some aspects of this disclosure.
[0021] Figure 5 illustrates an exemplary workflow for a multi-label classification pipeline for detecting and classifying multiple phenotypes according to some aspects of this disclosure.
[0022] Figure 6 illustrates an exemplary workflow of a backpropagator for a multi-label classification pipeline according to some aspects of this disclosure.
[0023] Figure 7 illustrates a set of single images (corresponding to ER, PR, and HER2) and a set of semantic masks according to some aspects of this disclosure.
[0024] Figure 8 illustrates an input triple image according to some aspects of this disclosure, having a set of interphase nuclear signal maps and a set of interphase signal maps for three label types (ER, PR, and HER2) generated by a multi-label classification pipeline.
[0025] Figure 9 illustrates an exemplary output of an input triple image with markers for three label types (ER, PR, and HER2) generated by a multi-label classification pipeline, according to some aspects of this disclosure.
[0026] Figure 10 illustrates exemplary graphs according to some aspects of this disclosure, showing one or more performance metric graphs on the training and validation sets.
[0027] Figure 11A shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of the detection model set during training for ER, PR, and HER2 label types.
[0028] Figure 11B shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of the detection model set on the validation set for ER, PR, and HER2 tag types.
[0029] Figure 12 shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of a multi-label classification pipeline from input triple images.
[0030] Figure 13 illustrates an exemplary flowchart for detecting and classifying multiple phenotypes according to some aspects of this disclosure.
[0031] Figure 14A illustrates a side-by-side comparison of superimposed estimates of PR and ER on a first triplet image and synthesized single PR and ER images according to some aspects of this disclosure.
[0032] Figure 14B illustrates a side-by-side comparison of superimposed estimates of PR and ER on a second triple image and an enhanced image, according to some aspects of this disclosure.
[0033] Figure 15 illustrates a side-by-side comparison of superimposed HER2 estimates on a third triplet image and an enhanced image, according to some aspects of this disclosure. Detailed Implementation
[0034] Some aspects of this disclosure relate to techniques for implementing a multi-label classification pipeline approach by leveraging various machine learning techniques to detect and classify multiple phenotypic or biomarkers. Traditional methods often require merging location and phenotypic predictions, which remains a challenging task in the absence of evidence-based rules. Furthermore, issues such as class imbalance in the training data, limited training data for some labels, and the ever-increasing number of classes can hinder the scalability and effectiveness of traditional methods. Therefore, obtaining a training set capable of teaching a model to detect and distinguish all labels, including less common ones, can be a significant challenge. The method disclosed in this disclosure eliminates the need to merge predictions of multiple biomarkers, simplifies the prediction process, and significantly improves the efficiency and accuracy of predictions.
[0035] In some respects, a multi-label classification pipeline can process multiple images labeled with multiple biomarkers to depict biological specimens within tissue samples. These multiple biomarkers can include various biomarkers or cellular components. By multiplexing multiple biomarkers, the pipeline enables detailed analysis of spatial relationships for the diagnosis of one or more biological conditions. A set of label types corresponding to multiple images of biological specimens within a tissue sample can be identified. Each label type in the set of label types can correspond to a biomarker among multiple biomarkers. Label types can act as identifiers, enabling classification and differentiation based on the unique characteristics of biological components, such as biomarkers or cellular structures. Biomarkers can be used to classify biological features or molecular properties, including the presence, expression level, or localization of target molecules. Label types can include proteins, such as cell membrane proteins (e.g., HER2) or nuclear proteins (e.g., ER or PR), providing meaningful insights into the cellular composition and function of tissue samples.
[0036] For each tag type, multiple tags can be identified, each representing various categories or expression intensities of the tag type associated with a specific biomarker (e.g., negative, intermediate, positive, or non-tumor cells). For example, for the HER2 protein biomarker, HER2 tag types can be identified, which have five distinct tags corresponding to different levels of HER2 protein expression. Their spatial distribution around the cell membrane is often used to determine tumor characteristics. The five distinct tags can include HER2 2+, HER2 3+, HER2 1+, non-tumor, and HER2-. HER2- can indicate the absence of HER2 protein or gene overexpression, while HER2 3+ can represent elevated overexpression levels. HER2 1+ and HER2 2+ can represent intermediate expression levels. Furthermore, HER2 non-tumor regions can be identified, representing areas where HER2 is not overexpressed.
[0037] Multiple labels can be used to annotate and classify multiple labels associated with each label type on multiple images, resulting in an annotated image set as output. Each annotated image in the annotated image set can be a multi-labeled image corresponding to a label type in the label type set, thus highlighting different biological features or different biomarkers. The annotated image set can provide a comprehensive view, enabling complex analyses such as identifying co-expression, distinguishing tumor cells from non-tumor cells, or identifying various cell subtypes based on their molecular profiles (e.g., gene expression, protein expression, cell surface markers, metabolites, or signaling pathways). This analysis allows researchers and clinicians to visually assess the spatial relationships and expression patterns of various biomarkers through multiple labels within multiple images of biological samples. For example, annotated images can highlight areas of elevated expression of a specific biomarker, with corresponding labels indicating the areas of elevated expression or the presence of different cellular components.
[0038] A multi-label classification pipeline processes multiple images by analyzing the spatial distribution and intensity of various markers within them. The pipeline can include several preprocessing steps to generate preprocessed images, thereby enhancing the quality of the multiple images for further analysis. Preprocessing steps may include resizing, normalization, contrast adjustment, background subtraction, noise reduction, or thresholding to improve the accuracy and reliability of the multi-label classification pipeline. Normalization can adjust intensity to compensate for variations in lighting or exposure, ensuring that features across images are comparable. Furthermore, contrast adjustment techniques (e.g., histogram equalization or adaptive contrast enhancement) can further highlight key features, including cell boundaries that distinguish them from each other. Additionally, removing unwanted background noise or irrelevant information from the image can enhance the visibility of various target key features. Furthermore, applying methods including Gaussian filtering or median filtering can smooth the image and eliminate noise that may interfere with classification during later stages of the multi-label classification pipeline. Finally, techniques such as the Otsu method or adaptive thresholding can be used to convert the image to a binary format. Furthermore, advanced preprocessing techniques can be applied based on the complexity of multiple images and after examining various markers and understanding the requirements of the labeling process.
[0039] Preprocessed images can be segmented using machine learning models or feature generators to generate a set of mid-phase signal maps corresponding to multiple labels. Each mid-phase signal map can represent a prediction for each part or cell location in the multi-image dataset, indicating whether these parts match the corresponding labels for multiple labels (e.g., ER-positive, ER-negative, or non-tumor for ER label types). The mid-phase signal maps can collectively form a multidimensional representation of the multi-image dataset, where each map can represent the predicted distribution of a specific label across the images. Mid-phase signal maps can be used to understand the spatial patterns and predictions of various biomarkers in tissue samples. In some aspects, the feature generator can also generate a set of probability matrices for each mid-phase signal map associated with each label. The probability matrices in the set of probability matrices associated with a specific label can reflect the likelihood that each part or cell location is assigned to a specific label. The combination of the mid-phase signal map set and the probability matrix set enables detailed classification and localization, thereby enhancing the accuracy of label identification and segmentation throughout the pipeline.
[0040] Interim signal maps can undergo various post-processing steps to generate a set of cell-specific point locations. Post-processing steps may include generating a set of interim nuclear signal maps for each of multiple labels (e.g., for ER label types: ER positive, ER negative, or non-tumor). The set of interim nuclear signal maps can identify representative locations of cells, and these locations can be predicted to correspond to each label. In the set of nuclear signal maps, the nucleus can be displayed as different regions within the cell, and its location can be detected using methods such as contour detection or connected component labeling. For visualization, a binary mask image or interim nuclear signal map can be generated, where detected nuclei are marked with white (or other colors) against a black or transparent background. Detected nucleus locations, represented by centroids or enclosed in bounding boxes or ellipses, can be overlaid on the interim signal map or the newly generated binary image to highlight their detection results. Alternatively, an RGB image can be generated, where nucleus locations corresponding to a specific label among multiple labels can be highlighted using different markers (such as dots, crosses, or other symbols), all rendered in the same color. It can provide clear and accurate visualization of cell nucleus locations based on segmented cell images, and can further refine the segmented cell images based on image quality.
[0041] Furthermore, a set of cell-specific point locations can be generated by processing at least two of the interphase signal maps or at least two of the interphase nuclear signal maps based on one or more mapping functions. The processing can involve identifying precise, spatially resolved locations within multiple images corresponding to a single cell or specific cellular regions corresponding to multiple labels. By accurately determining such cell-specific point locations, multi-label classification pipelines can deliver accurate insights into the distribution and concentration of biomarkers across tissues, facilitating the development of more precise diagnostic tools and personalized treatment strategies, thereby allowing clinicians to better assess how biomarkers are expressed at the cellular level within the tissue context. Mapping functions can perform mathematical operations on data structures with coefficients (e.g., matrices or arrays), allowing the function to adjust the data based on criteria such as signal intensity or distribution in interphase or nuclear maps. Mathematical operations can support the accurate identification of individual cell locations, helping to improve the overall sensitivity and resolution of models for diagnostic and therapeutic applications.
[0042] In some aspects, each cell-specific location corresponding to a particular label can be assigned a different color, thereby creating an RGB image in which the nucleus location of each label can be rendered in the same color. In some aspects of this disclosure, at least two of the interphase signal maps used to generate a set of cell-specific point locations, or at least two of the interphase nuclear signal maps, can be associated with multiple labels corresponding to the same label type in a set of label types. For example, signal maps for multiple labels such as ER+ (estrogen receptor positive), ER- (estrogen receptor negative), and non-tumor labels for the ER label type can be combined.
[0043] During signal map processing, the label with the highest probability can be selected for each cell location or region by comparing probability matrices. This improves the classification pipeline's ability to distinguish labels associated with cell or tissue characteristics, thereby enhancing the accuracy of diagnosis, prognosis, and treatment, while enabling relatively accurate label mapping for personalized medical insights. Post-processing steps may also include thresholding to classify each interphase signal map to predict the presence of a specific label. For example, a threshold of 0.7 can be set, where probability values (from the probability matrix) above this threshold are classified as labels, and those below are considered background. Thresholding ensures that high-confidence predictions are preserved, thereby reducing noise and improving pipeline accuracy by focusing on the most reliable predictions. Thresholding also aids in non-maximum suppression (NMS), which removes redundant detections, resulting in a clearer and relatively more accurate output. After thresholding classifies labels and background, NMS can eliminate superimposed or low-confidence detections by preserving reliable predictions based on a predefined intersection-over-union (IoU) threshold, thus ensuring accurate label classification by suppressing false positives. The centroid of a positive reaction can identify the predicted nucleus location, while multiple tags associated with each nucleus or cell location or part can provide deeper insights, thereby enhancing the information extracted from the prediction.
[0044] Furthermore, annotators can be used to generate annotated image sets by overlaying specific cell locations onto triple images. In some aspects, each of the multiple labels can be represented by different colors on the multiple images, or by each part of the digital pathology image (e.g., cells, pixels, or segmented regions). Finally, labels (associated with the corresponding biomarkers) can be selected and assigned to the parts. Annotated images can provide meaningful insights into the state of biomarkers at a granular level and can be used to identify biological conditions associated with a variety of biomarkers, including cancer, osteoporosis, uterine fibroids, autoimmune diseases, hormone-related conditions, cardiovascular diseases, PCOS, menstrual disorders, neurological diseases, and some thyroid conditions.
[0045] The feature generator can be trained using a pre-trained set of detection models applicable to each label type in the set of label types. The set of detection models can generate a set of ground truth annotations, providing accurate references for the presence, location, and boundaries of multiple labels corresponding to each label type. The set of detection models can also utilize a set of single-image datasets generated by an image generation system. Each image in the single-image dataset can contain markers specific to a particular label type, which can be used to generate ground truth annotations for that specific label type. The set of ground truth annotations can be used for training, validation, and evaluation, enabling the feature generator to accurately identify and predict multiple labels in multiple images, thereby ensuring enhanced detection accuracy. Each detection model in the set of detection models can include traditional machine learning models (e.g., SVM, KNN, Random Forest), CNNs (e.g., LeNet, AlexNet, DenseNet), transfer learning models (e.g., VGG16, ResNet50), or specialized models (e.g., U-Net, DeepLab).
[0046] A set of semantic masks can be generated based on ground truth annotations to highlight the presence of multiple labels, thus distinguishing them from the background. The set of semantic masks can include a semantic mask for each of the multiple labels, and an additional mask for the background representing regions without any biomarkers. To generate semantic masks, a set of blank masks can be initialized, each mask padded with zeros to indicate that no label was detected. Each blank mask in the set of blank masks can correspond to either a label or the background among the multiple labels. Iterative biomarker processing refines mask generation by detecting marker regions or boundaries and defining circular regions around the predicted label locations based on ground truth annotations from a boundary detector. The radius of the region can be determined using prior knowledge or ground truth annotations, thus locating the label within the boundary. The processing can further include generating a binary mask for each label and a binary mask for the background, where pixels within the circular region can be set to 1 (indicating presence), and those outside the circular region can be set to 0 (indicating absence). A background mask can be generated by identifying biomarker regions (regions with labels) and assigning a value of 1 to all non-marker pixels. The processing can also include overlaying a set of binary masks onto a background mask to generate a set of semantic masks that distinguish between biomarker regions and non-biomarker regions. Furthermore, the generated masks can be used to train a feature generator or fed back to a boundary detector to improve the quality of the semantic masks in subsequent iterations.
[0047] In some aspects of this disclosure, some labels from multiple labels may correspond to more general phenotypes or markers, whether general or specific to a particular training set, leading to class imbalance. If not properly addressed, this imbalance can cause the model to favor more general labels, resulting in both false negatives and false positives when detecting rare labels or anomalous features for accurate diagnosis, underestimating less common labels, and degrading performance. To address class imbalance, feature generators can be trained with adjustable weights to influence label assignment, including applying weighted masks or weight bias when conflicts occur. Weight bias can be facilitated by assigning higher weights to less common labels, thereby balancing the training process and improving learning from underrepresented data.
[0048] Weight bias ensures the model receives more exposure to positive instances, helping it learn to distinguish features. However, even with biased weights, the overall representation of true positives may not outperform false negatives or false positives due to natural data distribution or resource limitations. Therefore, the model can be trained to effectively handle both positive and negative detection results, ensuring robust predictive accuracy for both outcomes. The degree of weight bias can be varied based on external data such as label frequency, pixel distribution, or label prevalence, ensuring the feature generator is more sensitive to rare labels and improving its relative accuracy in detecting some labels (e.g., non-tumor versus ER+). Weight bias can resolve conflicts by prioritizing labels based on predefined criteria such as false positives, false negatives, and label prevalence in transphase signal maps or ground truth annotations. Applying weights allows the feature generator to resolve label conflicts and assign the correct label to each part, thereby enhancing diagnostic prediction accuracy.
[0049] In some respects, a weight set can be incorporated into the semantic mask set during training to adjust a threshold applied during the process of defining whether cell nuclei should be identified within a given region of a particular semantic mask. By modifying the threshold, the weight set can influence the model's sensitivity in detecting the presence of cell nuclei in that region, thereby ensuring relatively more accurate segmentation based on predefined criteria. Modifications can also allow the model to prioritize specific parts (e.g., regions or features), thereby enhancing the accuracy of cell nuclei identification for each label.
[0050] In some respects, a different set of weights can be applied to the nuclei in one inter-internal nucleus signal map compared to another, allowing for efficient handling of different scenes or labels based on variations in the dataset. A baseline nucleus map can be generated by analyzing the location of all nuclei in multiple inter-internal nucleus signal maps to determine whether detected nuclei belong to the same or different parts within multiple images. If the analysis determines that multiple detected nuclei correspond to the same cell, a single, uniform location can be identified. However, when assigning multiple inconsistent labels (for a specific label type) to a given cell, the weights or probabilities of conflicting labels can be compared. Comparisons can help resolve inconsistencies and select appropriate labels for each cell or part, ensuring that the final label assignment is accurate and consistent with the underlying data.
[0051] Binary cross-entropy can be used as a loss function to train a feature generator by computing the difference between the predicted and true labels, particularly for binary classification tasks such as determining whether a pixel or region belongs to a specific label (e.g., tumor or non-tumor). Cross-entropy measures the difference between the predicted and actual labels and aims to minimize the gap by penalizing the predictions based on confidence and accuracy. The loss may be relatively small when the match is closer, and relatively large when the match is significantly off. Cross-entropy can provide steep gradients for large discrepancies, helping the model converge faster and making it more sensitive to errors. Compared to other loss functions such as mean squared error (MSE), cross-entropy may be more effective for binary classification because it produces relatively more reactive gradients. During backpropagation, the loss can be used to generate one or more weights (or adjusted weights) based on a set of midpoint signal maps and a superimposed image created by combining a triple image with a semantic mask. One or more weights can be used based on a weight adjustment unit and to update the feature generator, thereby improving its performance. In some respects, the weight bias used to address class imbalance can be combined with one or more weights into a single backpropagator to improve model training.
[0052] In some respects, the labeled portions of multiplexed or digital pathology images can be smaller than physiological cells or cellular components. Furthermore, in some respects, spatial clustering of given labeled elements (e.g., biomarkers or cell types) can be anticipated based on physiological characteristics. For example, biomarkers or cellular features can naturally cluster in tissue samples, reflecting biological structures or patterns. In such cases, where the portions are relatively small or physiologically expected to cluster, one or more masks can be applied to the signal map during the process. Masks can help transform specific regions of the map, allowing the model to accurately identify predicted nucleus locations (or other cell-specific points). This process helps refine the segmentation of individual cells, even if these cells are small or tightly clustered.
[0053] For example, for markers confined to a specific cell nucleus (e.g., ER or PR), a relatively small semantic mask (e.g., a 5x5x5 mask) can be applied to identify the target region. The mask can be configured such that the signal centroid (i.e., the center point of the detected marker) is characterized as the cell nucleus. However, the characterization can be based on setting a threshold for the colocalization signal, meaning that the signal meets a threshold in terms of intensity and spatial superposition, which can be classified as a valid cell nucleus location. While an explicit thresholding step may not be necessary during image processing, it remains an integral part of the mask application results. Alternatively, for markers associated with cell membrane proteins (e.g., HER2), a larger mask (e.g., a 20x20x20 mask) can be used to accommodate the relatively large size and different spatial arrangement of the cell membrane. In some respects, the threshold used to detect the cell membrane using this mask may differ from that used for the cell nucleus because the signal intensity and spatial distribution of cell membrane protein markers may differ.
[0054] In some aspects, a graphical user interface (GUI) can be provided to display at least two interim signal maps (e.g., corresponding to different labels or label types) or interim nucleus signal maps. The GUI allows users to interact with the maps and provide input, such as rejecting, approving, or modifying image analysis results. Users may be able to adjust labels (e.g., modifying predicted labels for specific regions or pixels), nuclei (e.g., updating or refining detected nuclei), or nucleus locations (e.g., changing the predicted location of nuclei within tissue). The GUI can automatically update the displayed maps in real time to reflect user changes, enabling effective feedback and refinement for greater accuracy. Furthermore, another GUI can display ground truth annotations and semantic masks for each label to allow user interaction and feedback when discrepancies arise. Another interface can also allow users to modify, approve, or reject aspects of the image analysis results.
[0055] Feature generators can leverage various machine learning models, including convolutional neural networks (CNNs), deep neural networks (DNNs), or specialized architectures such as UNet or VNet. These models are trained on labeled data (e.g., sets of ground truth annotations and / or sets of semantic masks) to automatically predict target cell boundaries and segment images, thereby generating a set of mid-range signal maps. These models enable accurate spatial predictions in image segmentation, allowing multi-label classification pipelines to precisely label regions in multiple images, thus improving diagnostics and biomarker detection.
[0056] Consider an example of a triplet image, which displays staining from three biomarkers (e.g., ER, PR, and HER2), which can help predict various biological conditions. The three biomarkers among the multiple biomarkers can be biological indicators or biomarkers corresponding to cell membrane proteins (e.g., HER2) or specific nuclear proteins (e.g., ER or PR) identified by a set of label types. ER can detect hormone-related conditions such as breast cancer, osteoporosis, and thyroid disorders, with outputs of ER+, ER-, and non-tumor labels for the ER label type. PR can help identify breast cancer, uterine fibroids, and reproductive health problems, with outputs of PR+, PR-, and non-tumor labels for the PR label type. HER2 can aid in the diagnosis of breast cancer and other cancers, with outputs of HER2+, HER3+, HER1+, non-tumor, and HER2- labels for the HER2 label type.
[0057] A preprocessed image can be generated from the triple image and fed into the feature generator to produce twelve midpoint signal maps (eleven for the label and one for the background). These midpoint signal maps can be used to generate twelve midpoint nucleus signal maps, which can be further refined to generate twelve cell-specific point locations. Finally, three annotated images can be generated, each corresponding to a label type. Each annotated image can have labels corresponding to multiple labels for that label type. To train the feature generator, three pre-trained detection models can generate twelve ground truth annotations (eleven for the label and one for the background) from the corresponding set of single-image images for each marker or label type. Based on these annotations, twelve semantic masks representing multiple labels and the background can be generated and initialized with a pixel value of 0 (no marker detected). For example, the ER+ mask can mark ER+ regions in white and other regions in black. The feature generator can process masks to create markers based on intensity levels, enabling the detection of ER, PR, or HER2 and enhancing classification accuracy. Some labels can be independent of each other, such as classifying ER-positive and ER-negative cells in the same sample, or identifying cytopathology that simultaneously shows both tumor and non-tumor characteristics.
[0058] Figure 1 illustrates an exemplary block diagram 100 for multi-label detection and classification of multiple phenotypes according to some aspects of this disclosure. Multiple phenotypes can be defined based on multiple biomarkers or biological indicators detectable from multiple images depicting a biological sample in a tissue sample. Multiple phenotypes can be characterized by the spatial distribution and intensity of multiple biomarkers (such as proteins, genes, or cellular components) highlighted in the multiple images. By analyzing patterns of these biomarkers, multiple phenotypes can provide meaningful insights into complex biological processes, cell interactions, or disease states. The insights provided can be used to understand the mechanisms of various biological conditions, improve diagnostic accuracy, and develop targeted therapies, particularly in fields such as cancer research, immunology, and pathology.
[0059] Multiple markers can correspond to a set of label types, where each label type has multiple associated labels. Exemplary block diagram 100 outlines a process by which multiple markers can be simultaneously identified, analyzed, and classified. Exemplary block diagram 100 may include an image generation system 102, a network 106, a multi-label classification pipeline 108, a database 110, and a computer system 114. Image generation system 102 can generate multiple images 104 labeled with the set of label types. By using advanced imaging techniques, multiple images can capture complex biological data associated with biological samples. Image generation system 102 can be connected to network 106, enabling the transfer of multiple images 104 to other components in exemplary block diagram 100 (e.g., computer system 114 and multi-label classification pipeline 108). Network 106 can act as a bridge, ensuring seamless communication and integration across exemplary block diagram 100.
[0060] A multi-label classification pipeline 108 analyzes multiple images 104, assigning multiple labels associated with each label type to generate an annotated image set 112, thereby displaying detection results for various phenotypes or biomarkers. Each annotated image in the annotated image set 112 can correspond to a label type in a label type set. Each annotated image associated with a specific label type can indicate each part of the image (one or more pixels or cell locations), and the corresponding labels for multiple label types associated with that specific label type. Pipeline 108 can utilize machine learning models, image processing techniques, and advanced algorithms to accurately identify and classify different biological indicators in multiple images. A computer system 114 can connect to a database 110, which can store images generated by the image generation system 102. Database 110 allows for easy retrieval and management of multiple images, enabling further processing.
[0061] For example, image generation system 102 can generate triplet images labeled with three tag types (ER, PR, and HER2) associated with one or more biological conditions (e.g., cancer). The triplet images can be fed into a multi-label classification pipeline 108 via network 106. Pipeline 108 can generate three images, each corresponding to a tag type, as shown in the output (i.e., annotated image 112). For example, regarding the HER2 protein, the first annotated image in the annotated image set 112 can indicate one of five tags (HER2 2+, HER2 3+, HER2 1+, non-tumor, and HER2-) for each part or cell location. Each of the five tags can correspond to different expressions of the HER2 protein; for example, HER2- can correspond to a prediction of no overexpression of the HER2 protein or gene, while HER2 3+ can indicate overexpression. HER2 1+ and HER2 2+ can represent different expression levels, and their spatial distribution around the cell membrane can be used to determine tumor characteristics. Furthermore, non-tumor regions of HER2 can be identified, reflecting areas where the HER2 protein is not overexpressed. As another example, regarding the ER protein, the second annotated image in the annotated image set 112 can indicate one of three categories (ER positive, ER negative, or ER non-tumor) for each part or cell location. Similarly, regarding the PR protein, the third annotated image in the annotated image set 112 can indicate one of three categories (PR positive, PR negative, or PR non-tumor) for each part or cell. Each part or cell location can correspond to one or more pixels. The annotated image set 112 enables comprehensive tissue analysis, capturing the diverse expression and relationships of the tag-type set. This analysis can enhance clinicians' understanding of underlying biological conditions and their progression, thereby supporting more effective intervention strategies.
[0062] Figure 2 illustrates an exemplary block diagram for generating digital pathology images according to some aspects of this disclosure. Image generation system 102 can be used to generate digital pathology images 212, including single or multiple images. Image generation system 102 may include a fixation / embedding system 204, a tissue slicer 206, a staining system 208, and an imaging system 210. Fixation / embedding system 204 fixes and / or embeds tissue samples (e.g., liquid fixatives, such as formaldehyde solution) and / or embedding materials (e.g., histological wax, such as paraffin, and / or one or more resins, such as styrene or polyethylene). Each slice can be fixed by exposing the slice to the fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the slice (e.g., by exposing it to an ethanol solution and / or a clarifying intermediate). When the embedding material is in a liquid state (e.g., when heated), the embedding material can infiltrate the slice.
[0063] Tissue slicer 206 then sections the fixed and / or embedded tissue sample (e.g., a tumor sample) to obtain a series of sections, each section having a thickness of, for example, 4 to 5 micrometers. This sectioning of tissue can be performed by first cooling the sample and then sectioning it in a warm water bath. Tissue sections can be sectioned using, for example, a vibratory microtome or a compression microtome. Because tissue sections and the cells therein are transparent, section preparation typically involves staining the tissue sections (e.g., automated staining) to make the relevant structures more visible. In some respects, staining can be performed manually. In some other respects, staining can be performed semi-automatically or automatically using staining system 208.
[0064] Staining can include exposing individual sections of tissue to one or more different staining agents (e.g., sequentially or simultaneously) to express different characteristics of the tissue. For example, each section may be exposed to a predefined volume of staining agent for a predefined time period. Staining agents may include, for example, RNA probes, protein probes (e.g., nuclear protein probes or cytoplasmic protein probes), immunohistochemical staining agents, or probes targeting secreted substances. In some aspects, the staining agent is a staining agent targeting KAPPAm RNA or LAMBDA mRNA.
[0065] An exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining can be used to indicate general aspects of tissue morphology and / or cellular microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). An example of a histochemical staining agent is hematoxylin and eosin (H&E). Other examples of histochemical staining agents include trichrome staining agents (e.g., Masson's trichrome), periodic acid-Scheffler (PAS), silver staining agents, and iron staining agents. The molecular weight of histochemical staining agents (e.g., dyes) is typically about 500 kilodaltons (kD) or less, although some histochemical staining agents (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have molecular weights as high as two or three thousand kD. An example of a high molecular weight histochemical staining agent is α-amylase (about 55 kD), which can be used to indicate glycogen.
[0066] Another type of tissue staining is immunohistochemistry (IHC, also known as "immunohistochemistry"), which uses a primary antibody that specifically binds to a target antigen (biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a tag (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to the tag (e.g., a chromophore or fluorophore) binds to the primary antibody. IHC reagents have a much higher molecular weight than histochemical staining reagents because antibodies have a molecular weight of approximately 150 kDa or higher.
[0067] The slides can then be individually mounted onto their respective slides. Imaging system 210 can then scan the slides to generate digital pathological images 212. Each slide can be mounted on a slide and then scanned to create a digital image, which can then be examined via digital pathological image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). Imaging system 210 can use bright-field imaging, dark-field imaging, or fluorescence imaging to digitize pathological slides (whole slides or sections). Imaging system 210 can include, but is not limited to, microscopes with digital cameras, robotic microscopes, or WSI scanners such as the Ventana iScan HT, Ventana DP 200, or Ventana DP 600. In some respects, pathologists can review and manually annotate the digital images of the slides (e.g., tumor areas, necrosis, etc.). Annotation of target areas can be automated using computer vision techniques. Digital pathological images 212 can be converted to other domains for further processing.
[0068] Digital histopathological images (e.g., 2^12) typically consist of arrays of pixels, usually rectangular matrices. Each “pixel” is an image element and a digital quantity representing some attribute of the image at a location in the array corresponding to a specific position in the image. Typically, in continuous-tone black and white images, pixel values represent grayscale values. Pixel values in digital images usually conform to a specified range. For example, each array element could be a byte (i.e., eight bits) representing a pixel value in the range of 0 to 255. In a grayscale image, the value “255” can represent white, and the value zero (“0”) can represent black. Color images can include those corresponding to red, green, and blue (RGB). For a given pixel, each of the three color planes has a value (i.e., a first value representing the red component, a second value representing the green component, and a second value representing the blue component). By varying the intensity of these three components, all colors in the color spectrum can be produced. Samples stained with multiple IHC can be sequentially illuminated with multiple light channels matched to the absorption bands of the chromatographs to capture brightfield images. In the case of multiplex immunofluorescence, fluorescence microscopes with different filters can be used to capture fluorescence or emitted light from fluorophores associated with each biomarker.
[0069] Figure 3 is a block diagram illustrating an exemplary architecture 300 for generating a set of ground truth annotations 306 from a set of detection models 304, according to some aspects of this disclosure. The set of detection models 304 can be trained to detect multiple labels corresponding to label types. A set of single images 302 generated by the image generation system 102 can be fed into the set of detection models. Each single image in the set of single images 302 can correspond to a specific label type in the set of label types. The detection models in the set of detection models 304 can analyze the single images in the set of single images 302 to identify and classify multiple labels corresponding to a specific label type by extracting relevant features from the corresponding single images.
[0070] The results of the ground truth annotation 306 can be presented on the graphical user interface (GUI) 308 of the computer system 114. The GUI 308 provides an intuitive and user-friendly interface to visualize the results, which can aid in further analysis and decision-making. The GUI 308 can include multiple images, each corresponding to a specific label type, which can be labeled with multiple labels corresponding to that specific label type. The pipeline 108 can then perform advanced feature generation and classification tasks, allowing the simultaneous detection and classification of multiple labels from multiple images 104. For example, the GUI 308 can display results for label types such as ER, PR, and HER2, each labeled with its corresponding label, as shown. Each detection model in the detection model set may include traditional machine learning models (e.g., SVM, KNN, random forest, or logistic regression), convolutional neural networks (e.g., LeNet, AlexNet, DenseNet, MobileNet, or EfficientNet), transfer learning models (e.g., VGG16, ResNet50, InceptionV3, or Xception), or other specialized models (e.g., U-Net or DeepLab).
[0071] Figure 4 is a block diagram illustrating an exemplary architecture 400 for generating a set of semantic masks 410 for each of a plurality of labels from a ground truth annotation 306, according to some aspects of this disclosure. The set of semantic masks 410 can be generated based on the ground truth annotation 306 to highlight the presence of multiple labels to distinguish them from the background. The semantic masks 410 may include a semantic mask for each of the multiple labels, and an additional mask for the background representing regions that do not contain any biomarkers or label types.
[0072] The exemplary architecture 400 can begin with an initialized mask 402, where each mask in a set of blank masks (a blank mask for each of a plurality of labels and an additional mask for the background) can be initialized with zero to indicate that no label was detected, and each mask corresponds to either a plurality of labels or the background. The initialized mask 402 can refer to creating an initial binary mask in image processing or computer vision to highlight a target region in the original image. In medical imaging, the initialized mask 402 can be set to blank and can later be refined to highlight specific features such as biomarkers, tumors, or cells.
[0073] Then, iterative biomarker processing can refine the initialized mask used for detecting and classifying multiple labels. Boundary detector 404 can iteratively detect boundaries or define a circular region around the predicted biomarker location on a ground truth annotation set 306 within a specified radius. The radius of the circular region can be determined using prior knowledge or the ground truth annotation set 306, thereby locating multiple labels in the biological sample. Mask creator 406 can iteratively generate a set of binary masks for multiple labels, assigning a value of 1 (indicating label presence) to pixels within the circular region and a value of 0 (indicating absence) to pixels outside the region, thereby locating the label positions and accurately representing the biomarker boundaries. Mask creator 406 can use the binary mask set to identify biomarker regions to create a background mask. Mask creator 406 can generate the background mask by using the binary mask set to identify label regions, setting pixels within the label regions to zero, and assigning a value of 1 to all other pixels, thereby creating a binary mask for the background.
[0074] Mask stacking unit 408 iteratively merges a set of binary masks for multiple labels with a background mask to distinguish biomarker regions from non-biomarker regions, generating a semantic mask set 410. The semantic mask 410 can play a crucial role in downstream analysis, such as evaluating label distribution and intensity, and enabling precise localization and detection of cellular features for diagnostic applications such as cancer detection and treatment planning. The generated semantic mask set 410 can be used to train a feature generator or fed as part of an iterative process into a boundary detector 404, allowing for further refinement and enhancement of the semantic mask 410 after each iteration.
[0075] Figure 5 illustrates an exemplary workflow 500 of a multi-label classification pipeline 108 for detecting and classifying multiple phenotypes according to some aspects of this disclosure. The multi-label classification pipeline 108 may include a preprocessor 502, a feature generator 504, a postprocessor 508, and an annotator 510. Multiple images 104 may undergo a series of preprocessing steps (which may vary depending on the image type and model used) by the preprocessor 502 to generate preprocessed images for further downstream analysis. Preprocessing steps may include resizing, normalization, contrast adjustment, background subtraction, noise reduction, and thresholding to improve image quality and enhance the accuracy and reliability of the classification pipeline 108. Normalization may adjust the intensity to compensate for variations in illumination or exposure, thereby ensuring consistent features across images. Furthermore, contrast adjustment techniques (e.g., histogram equalization or adaptive contrast enhancement) may help highlight key features (e.g., cell boundaries) by making them more distinguishable from each other. Additionally, background subtraction may remove unwanted noise or irrelevant information, thereby improving the visibility of important features. Furthermore, noise reduction methods (e.g., Gaussian filtering or median filtering) can smooth images and eliminate interference, thus enabling better classification. Binary transformation techniques such as the Otsu method or adaptive thresholding can also be applied. Advanced preprocessing techniques can also be employed based on the complexity of the multiple images, the number of markers to be examined, and the specific requirements of the labeling process.
[0076] Once preprocessing is complete, the preprocessed image can be segmented using feature generator 504 to generate a set of mid-term signal maps 506. Each mid-term signal map in the set of mid-term signal maps 506 can include multiple parts or cell locations. Each part may represent a prediction for that part, indicating whether multiple parts can be consistent with the corresponding label. Taken together, the mid-term signal maps 506 constitute a multidimensional representation of the multiple images 104, such that each map shows the predicted distribution of a specific label across image patterns among multiple labels. The mid-term signal maps 506 can help understand the spatial relationships within each part or cell location that is generally consistent with multiple labels for various target regions, where each mid-term signal map image can represent a specific label. They can isolate and identify regions in which label types can exist, thus enabling precise analysis. In some aspects, feature generator 504 can also generate probabilities for each part or cell location corresponding to a label across the set of mid-term signal maps 506. Feature generator 504 can be trained using ground truth annotations 306 and a set of semantic masks 410 to provide the correct label for each pixel or segment, allowing the model to improve its ability to distinguish different biomarkers. Feature generator 504 can be trained using backpropagation 514, where one or more weights are assigned to each label. The weights of each label can then be adjusted, and the adjusted weights of each label are sent back to feature generator 504 via adjusted weights 516.
[0077] In some respects, class imbalance (where some labels are more common than others) can cause the pipeline 108 to perform better at classifying common labels but poorer at classifying rare ones. To address this, the feature generator 504 can be trained with adjustable weights, thus assigning weight biases to less common labels to improve detection accuracy. Weight biases can adjust the semantic mask threshold for better detection of cell nuclei and resolve label conflicts by selecting accurate labels for each region. Assigning weight biases can involve giving higher weights to rare samples in the training set, thereby improving the model's ability to detect underrepresented labels.
[0078] The interphase signal map 506 can undergo multiple post-processing steps in the post-processor 508 to generate a set of cell-specific point locations. These post-processing steps may include processing the interphase signal map 506 to create a set of interphase nuclear signal maps for each label (e.g., for ER label types: ER positive, ER negative, or non-tumor labels). The interphase nuclear signal maps can identify the predicted cell location corresponding to each label, with the interphase nuclei appearing as distinct regions within the cell. For example, the nucleus may appear as a white dot (indicating the presence of a label or 1), while the background may appear as a black dot (indicating the absence of a label or 0), thus producing a binary image. Nucleus locations can be detected using techniques such as contour detection or connected component labeling (e.g., using the `skimage.measure.label` function in Python).
[0079] Detected nucleus locations, represented by their centroids or enclosed within bounding boxes or ellipses, can be overlaid onto the interphase signal map 506. For visualization, the binary mask set or interphase nucleus signal map can highlight the detected nuclei in white (or another color) against a black or transparent background. Alternatively, an RGB image can be generated where nucleus locations corresponding to specific labels are marked using different markers (such as dots, crosses, or other symbols), all rendered in the same color. Highlighting allows for clear and precise visualization of nucleus locations based on segmented cell images, with the option to further refine the image based on image quality.
[0080] The post-processing step may further include generating a set of cell-specific point locations by processing at least two of the interphase signal maps and / or at least two of the interphase nuclear signal maps using one or more mapping functions. The processing may include identifying precise, spatially resolved locations within the multiple images 104, which may correspond to portions associated with multiple labels. By accurately determining cell locations, the multi-label classification pipeline 108 can deliver relatively accurate insights into the distribution and concentration of biomarkers across tissues, aiding in the development of relatively precise diagnostic tools and personalized treatment strategies, thereby enabling clinicians to better assess how biomarkers are expressed at the cellular level within tissues. The mapping functions used to generate cell-specific points may involve mathematical operations on data structures (e.g., matrices or arrays), whose coefficients can be adjusted based on criteria such as signal intensity or distribution in the interphase signal map set 506 or the nuclear signal map set. Mathematical operations can help identify the precise location of individual cells, improving the sensitivity and resolution of the model for diagnostic and therapeutic applications.
[0081] In some aspects, each cell location corresponding to a specific label can be assigned a different color, creating an RGB image where the nucleus location for each label is rendered in one color, thus providing clear visualization of the nucleus location for a specific label. In other aspects, at least two of the inter-annual signal maps and / or inter-nuclear signal maps can be used to generate cell-specific point locations that can correspond to multiple labels relative to the same label type. For example, signal maps for labels such as ER+ (estrogen receptor positive), ER- (estrogen receptor negative), or non-tumor labels for the ER label type can be combined. The label with the highest probability of being associated with a given cell location or portion can be selected from multiple labels within the same label type. By selecting the label with the highest probability, the pipeline 108 can improve its ability to distinguish individual labels, thereby enhancing the accuracy of diagnostic, prognostic, and treatment planning approaches. This selection can also support relatively accurate mapping of multiple labels, providing advanced diagnostic and personalized medicine insights.
[0082] The post-processor 508 can apply thresholding to classify each interphase signal or cell nuclear signal map to predict the presence of a specific label. Thresholding can be applied to the probability value of each cell location or region to distinguish multiple labels from the background. For example, a threshold of 0.7 can classify values above this threshold as labels and values below this threshold as background, thus considering high-confidence predictions and reducing the impact of noise. Thresholding can improve the accuracy of pipeline 108 by focusing on reliable predictions.
[0083] Thresholding is also crucial for non-maximum suppression (NMS) by removing redundant detections and ensuring clearer output. After thresholding, NMS can eliminate overlapping or low-confidence detections. Based on a predefined intersection-over-union (IoU) threshold, NMS can preserve reliable predictions, reduce false positives, enhance label clarity, and help identify the centroid of predicted cell nuclei. Furthermore, NMS enables the prediction of multiple labels and the label type associated with each cell nucleus or cell, providing deeper insights and improving the accuracy of the analysis.
[0084] Furthermore, an annotated image set 112 can be generated by overlaying cell-specific locations onto the multi-image 104 using an annotator 510. In the annotated image set 112, each annotated image can correspond to a label type, allowing multiple labels to be assigned based on a specific type. The annotator 510 can acquire the multi-image 104 and overlay identified features or cell-specific location sets (e.g., cell location, presence of biomarkers) onto the original triplet image (or other base image).
[0085] Overlaying can highlight areas with multiple labels, typically using different color codes or markers for each label. For example, multiple labels can be represented by red, green, or blue to ensure visual differentiation. For instance, in an annotated image of the ER label type, red could represent "ER+", yellow could represent "ER-", and black markings could represent "non-tumor" cell locations or portions. A similar classification can be applied to PR in another annotated image. For a third annotated image, the HER2 label type can be displayed using different color-coded intensity levels: red for "HER2 2+", maroon for "HER2 3+", green for "HER2 1+", yellow for "non-tumor" cell locations, and orange for "HER2-". Color coding allows clinicians and researchers to easily distinguish between multiple biomarkers that may be present in a biological sample through better and clearer visualization of the annotated image 112.
[0086] In some respects, annotation tools can be used to collect input from users or experts to manually or automatically assign labels to specific parts or cell locations in multiple images 104. Annotation tools can help highlight areas containing biomarkers, label target areas, or add text descriptions. The tool allows users to associate multiple labels (e.g., ER positive, ER negative, or non-tumor for ER label types) with specific parts or cell locations. The tool can also be used to draw boundaries, add text labels, or modify colors to highlight certain features. The resulting annotated image 112 can provide a comprehensive view of the spatial distribution of multiple labels, enabling clinicians and researchers to precisely locate multiple biomarkers within the tissue for analysis, diagnosis, or treatment planning. Annotated image 112 can also support studies on the relationships between biomarkers, cell structures, and tissue properties.
[0087] Figure 6 illustrates an exemplary workflow 600 of the backpropagator 514 of a multi-label classification pipeline 108 according to some aspects of this disclosure. In some aspects, the feature generator 504 can be trained using a cross-entropy module 602 to optimize model performance by minimizing the error between the predicted and actual labels based on ground truth annotations 306. The cross-entropy module 602 can be effective for classification tasks that may involve binary outcomes, such as determining whether a part or cell location belongs to a specific label (e.g., tumor or non-tumor). The cross-entropy module 602 can measure the difference between the actual label or probability and the predicted label or probability, aiming to minimize this difference for more accurate predictions. The cross-entropy module 602 can be used as a direction mechanism during training, enabling the feature generator 504 to learn effectively by adjusting its parameters or weights by the backpropagator 514. By iteratively refining the network parameters based on the error, the model can enhance the accuracy of its predictions, resulting in better consistency with the actual labels and improving overall model performance.
[0088] When there is a relatively large observation error between the predicted and actual labels, the cross-entropy module 602 can provide a steep gradient, making it more sensitive to error and helping the model converge faster during training. Binary cross-entropy performs better for binary classification tasks than other loss functions such as mean squared error (MSE) because MSE provides asymptotic gradients, which can potentially slow down the learning process. The cross-entropy module 602 can generate the output loss using the midpoint signal map 506 and the overlay image set. The overlay image set can be generated by combining the multiple images 104 with the semantic mask set 410 using the overlay module 604. Adjusted weights 516 can then be generated using the weight adjustment unit 606 based on the loss generated by the cross-entropy module 602. The adjusted weights 516 can be used to update the feature generator 504 using the backpropagator 514, refining the feature generator 504, thereby enhancing the performance and accuracy of the multi-label classification pipeline 108 over a certain period of time.
[0089] Figure 7 illustrates single-image sets (corresponding to ER, PR, and HER2) and semantic mask sets according to some aspects of this disclosure. The set of label types (i.e., ER, PR, and HER2) can play an important role in detecting various hormone-related conditions, including breast cancer, endometriosis, osteoporosis, uterine fibroids, cardiovascular disease, polycystic ovary syndrome (PCOS), menstrual disorders, neurological disorders, and some thyroid conditions. ER and PR labels can generate three possible labels: ER+ (estrogen receptor positive), ER- (estrogen receptor negative), and non-tumor; PR is classified similarly. On the other hand, based on the intensity of the biomarker, HER2 can generate five possible labels: HER2+, HER3+, HER1+, non-tumor, and HER2-. Multiple labels can provide meaningful insights into the molecular characteristics of tissues, thereby enabling more accurate diagnostic and treatment plans for the aforementioned conditions.
[0090] Each semantic mask in the semantic mask set 410 visualizes the presence of multiple labels on a single-weighted image, which can be generated by the exemplary architecture 400. For example, box 702 shows the corresponding single-weighted image for the ER label type and displays its corresponding semantic mask (ER+, ER-, and non-tumor). Similarly, for the PR label type, box 704 shows the corresponding single-weighted image for the PR label type and displays the semantic mask corresponding to the label type (i.e., PR+, PR-, and non-tumor). For HER2, box 706 shows various masks for HER2 1+, HER2 2+, HER2 3+, HER2-, and non-tumor. Generating the semantic mask set 410 for each label type allows for detailed and precise analysis of multiple labels, thereby improving the understanding of their roles in various biological conditions.
[0091] Figure 8 illustrates an input triple image with a set of semantic masks 410 and a set of midpoint signal maps for three label types (ER, PR, and HER2) generated by the multi-label classification pipeline 108. The semantic mask set 410 can be used to train a feature generator 504, and a set of midpoint signal maps 506 can be generated by the feature generator 504, making the midpoint signal map set 506 the output derived from the semantic masks 410. Each label type can correspond to multiple labels, which can be combined to help detect one or more biological conditions. For example, multiple labels can include three labels for ER (ER+, ER- and non-tumor), three labels for PR (PR+, PR- and non-tumor), and five labels for HER2 (HER2 1+, HER2 2+, HER2 3+, HER2- and non-tumor). Twelve semantic masks for multiple labels, plus an additional mask for the background, can be generated.
[0092] Box 802 displays the triplet (i.e., the multiple images 104 fed into pipeline 108 as input), which can be used to generate a semantic mask set 410 and a mid-term signal map set 506. Box 804 displays the semantic mask set 410 corresponding to the PR (progesterone receptor) tag type on the right. The corresponding mid-term signal map for the PR tag type is displayed on the left. Similar to Box 804, Box 806 displays the semantic mask set 410 for the ER (estrogen receptor) tag type on the right. Mid-term signal map 506 information related to the ER tag type is displayed on the left. Furthermore, in Box 808, both the semantic mask 410 and the mid-term signal map 506 corresponding to the HER2 tag type can be displayed. Each mask and map can be labeled with text to identify specific features or various properties detected by the feature generator 504, thereby providing a comprehensive overview of HER2-related features in the multiple images 104.
[0093] In some respects, the semantic mask set 410 can be referred to as a reference set of interphase nuclear signal maps, since the semantic mask 410 can include a corresponding set of nuclear nuclei generated by the feature generator 504. The reference set of nuclear signal maps can be compared with the set of nuclear signal maps derived from the interphase signal map 506 to evaluate the model's performance or loss function, thereby calculating the model's accuracy. The comparison can demonstrate how different masks and thresholds can be applied to generate different signal representations for each label, enabling relatively accurate identification of specific cellular structures and biomarkers.
[0094] Figure 9 illustrates an exemplary output 900 of an input triple image with biomarkers for three label types (ER, PR, and HER2) generated by a multi-label classification pipeline 108, according to some aspects of this disclosure. As shown in exemplary block diagram 200, the triple image 902 can be generated using image generation system 102. Each annotated image can represent output for a specific label type, such that different biomarkers (circles, dots of various colors) for each of the multiple labels correspond to the specific label type. For example, in annotated image 904, each cell can be labeled according to its ER status, where red represents “ER+”, yellow represents “ER-”, and black marks “non-tumor” cells or portions. A similar classification can be applied to PR in another annotated image 906. For the third annotated image 908, the HER2 biomarker output can be displayed using different color-coded intensity levels: red represents “HER2 2+”, maroon represents “HER2 3+”, green represents “HER2 1+”, yellow represents “non-tumor” cells, and orange represents “HER2-”.
[0095] Different markers can be generated based on various intensity levels, which can be derived by the feature generator 504 processing the semantic mask set 410 (corresponding to each label type). In some aspects of this disclosure, more than three classification types for ER or PR, or more than five classification types for HER2, can be detected based on the intensity level of the markers. The increased sensitivity can be identified by experts in the field or by advanced imaging techniques capable of distinguishing more subtle intensity variations, thereby enhancing the accuracy of the classification process.
[0096] Figure 10 illustrates an exemplary figure 1000 according to some aspects of this disclosure, showing one or more performance metric plots on training and validation sets. Exemplary figure 1000 may show multiple plots measuring the performance of a multi-label classification pipeline 108 applied to a triple image 902. The triple image 902 may contain markers from ER, PR, and HER2 label types. Figure 1002 shows validation accuracy related to multiple steps. Validation accuracy measures how well the feature generator 504 performs on a separate subset of data called the validation set. Unlike the training set, the validation set cannot be used to train the feature generator 504 and can be used to predict the model's performance on unseen data. Validation accuracy can be calculated by determining the percentage of correctly labeled data on the validation set. Validation accuracy can identify overfitting (better performance on the training dataset but worse performance on the validation dataset) or underfitting (poor performance on both the training and validation datasets). Monitoring validation accuracy may be used to adjust the general performance of the feature generator 504 for new, unseen real-world datasets. In Figure 1002, the validation accuracy reaches approximately 0.85, indicating that its performance is at an acceptable level for real-world applications. Notably, the significant improvement in accuracy from 0.6 to 0.8 highlights the model's ability to generalize its learning process.
[0097] Furthermore, Figure 1004 shows the training loss associated with multiple steps. The training loss measures how well the model performs on the training dataset. It can be calculated by determining the difference between the predicted and true labels based on a chosen loss function (e.g., cross-entropy for classification). A lower training loss indicates that the predictions based on feature generator 504 are closer to the actual values in the training dataset. Monitoring the training loss during training helps assess how effectively the model learns from the training data. As the generalization ability of feature generator 504 improves, the training loss can decrease over time. However, if the training loss decreases significantly while the validation loss remains high, it may indicate overfitting, meaning the model performs well on the training dataset but may not generalize to unseen validation datasets. In Figure 1004, the training loss reaches approximately 0.4, indicating that its performance is acceptable for real-world applications. Notably, the training loss decreases dramatically from approximately 1.2 to 0.4, reflecting a significant improvement in the model's performance.
[0098] Furthermore, Figure 1006 shows the validation loss associated with multiple steps. The validation loss can be used to evaluate the performance of feature generator 504 on the validation set. Unlike accuracy (which measures the proportion of correctly predicted values), the validation loss measures how well the predicted labels align with the actual results based on the chosen loss function (e.g., cross-entropy). A lower validation loss can indicate better performance because it means the model's predictions are closer to the true values in the validation data. Monitoring the validation loss during training may be related to identification problems such as overfitting, where feature generator 504 may perform better on the training data but poorer on unseen validation data. By minimizing the validation loss, feature generator 504 can be trained to generalize, thereby improving its performance on real-world datasets. In Figure 1004, the validation loss reaches approximately 0.3, indicating that its performance is at an acceptable level for real-world applications. Notably, the significant reduction in validation loss from 0.5 to 0.3 during the training steps reflects a substantial improvement in model performance.
[0099] Validation accuracy serves as a better metric for evaluating the overall performance of the model in detecting ER / PR / HER2 biomarkers. To calculate accuracy, ER, PR, and HER2 can be evaluated separately, with a weighting of 0.3 for ER, 0.3 for PR, and 0.4 for HER2. The optimal accuracies for ER, PR, and HER2 are 0.89, 0.93, and 0.80, respectively. Therefore, the optimal overall accuracy is 0.86.
[0100] Figure 11A shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of the detection model set 304 during training for the ER, PR, and HER2 label types. A single-image set 302 corresponds to the three label types (ER, PR, and HER2) that can be generated by the image generation system 102 and fed into the detection model 304. Each image in the single-image set 302 can be characterized by a label of a specific label type (e.g., ER, PR, or HER2). The confusion matrix is a table that can be used to evaluate the performance of the classification model. The confusion matrix can display true positives, true negatives, false positives, and false negatives, assessing the consistency between the model's predictions and the true results. Rows in the confusion matrix can represent true labels, and columns can represent predicted labels, providing insights into accuracy, precision, recall, and other key metrics while highlighting potential error or misclassification regions.
[0101] Multiple confusion matrices 1102 can include three tables trained with a set of each of the three label types. For example, consider the results generated by detection model 304 corresponding to the ER (estrogen receptor) label type, denoted as 1102-A. According to confusion matrix 1102-A, 20,974 cells are correctly labeled as "ER+", while 3,651 cells, although belonging to the "ER+" group, are incorrectly labeled as "ER-". Furthermore, 2,758 cells are incorrectly labeled as "non-tumor" instead of "ER+". Similarly, the results for the PR (progesterone receptor) label type can be represented by confusion matrix 1102-B, and the results for HER2 (human epidermal growth factor receptor 2) can be represented by confusion matrix 1102-C, which shows similar patterns of correct and incorrect classification for each label across multiple labels within the label types. Therefore, the overall accuracy for the ER label type is 0.86, for the PR label type is 0.91, and for the HER2 label type is 0.80, respectively. Furthermore, the balanced accuracy for ER is 0.89, for PR is 0.86, and for HER2 is 0.77, respectively. The overall accuracy and balanced accuracy metrics reflect the ability of the detection model ensemble 304 to correctly classify multiple label categories within the label types during the training phase, thus helping to assess the performance and robustness of the detection model 304.
[0102] Figure 11B shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of the detection model set 304 on the validation set for the ER, PR, and HER2 label types. A single image set 302 corresponds to the three label types (ER, PR, and HER2) that can be generated by the image generation system 102 and can be fed into the detection model 304. Each image in the single image set 302 can be characterized by a label from a set of multiple labels, which can correspond to a specific label type (e.g., ER, PR, or HER2) in the set of label types.
[0103] The multi-confusion matrix 1104 may include three tables for the validation phase, where each set corresponds to a label type among the three label types. For example, consider the results generated by the detection models of the detection model set 304 corresponding to the ER (estrogen receptor) label type, denoted as 1104-A. According to the confusion matrix 1104-A, 5800 cells were correctly labeled as "ER+", while 1050 cells, although belonging to the "ER+" group, were incorrectly labeled as "ER-". Furthermore, 351 cells were incorrectly labeled as "non-tumor" instead of "ER+". Similarly, the results for the PR label type are represented by 1104-B, and the results for HER2 are represented by 1104-C. Therefore, the overall accuracy for ER is calculated to be 0.88, the overall accuracy for PR is calculated to be 0.94, and the overall accuracy for HER2 is calculated to be 0.80, respectively. Furthermore, the balance accuracy for ER was 0.90, for PR it was 0.95, and for HER2 it was 0.75. Overall accuracy and balance accuracy metrics reflect the ability of the detection model ensemble 304 to correctly classify multiple label categories within the label types during the validation phase, thus helping to assess the performance and robustness of detection model 304.
[0104] Figure 12 shows a confusion matrix according to some aspects of this disclosure, illustrating the performance of a multi-label classification pipeline 108 from a triple image 902. The triple image 902 generated by the image generation system 102 may have labels from three different label types (i.e., ER, PR, and HER2). The triple image 902 can be processed by the multi-label classification pipeline 108, which generates three annotated images, each corresponding to a different label type. The annotated images 112 can provide predictions for each region of the image, corresponding to multiple labels (e.g., labels “ER+”, “ER-”, or “non-tumor” for the ER label type; labels “PR+”, “PR-”, or “non-tumor” for the PR+ label type; and labels “HER2 2+”, “HER2 3+”, “HER2 1+”, “non-tumor”, or “HER2 -” for the HER2 label type). A confusion matrix 1202 can be generated to represent the classification results for each label type, thus providing a detailed comparison of predicted labels with true labels across multiple labels. The confusion matrix 1202 can help assess the overall accuracy and effectiveness of the multi-label classification pipeline 108 by measuring the true positive, false positive, true negative, and false negative predictions for each label.
[0105] The confusion matrix 1202 can include a set of categories generated based on multiple labels. The set of categories can include one category for the "non-tumor" category, and the remaining twelve categories can correspond to various label combinations, with a label coming from each of the label types (such as ER, PR, and HER2). For example, 6403 cells can be correctly labeled as "non-tumor," and the category from the set of categories in the matrix can be represented as a combination of the labels <"ER-", "PR+", "HER2 1+">. Within this category, 843 cells can be correctly identified based on the specified label combination. The confusion matrix 1202 can also highlight misclassified cells, where errors originate from various label combinations across different label types. For example, 127 labels are incorrectly labeled as <"ER-", "PR-", "HER2 1+">, while 119 labels are labeled as <"ER-", "PR-", "HER2 1+">. The confusion matrix 1202 can provide detailed insights into the performance of the multi-label classification pipeline 108 across multiple label combinations, thus highlighting areas of misclassification. These insights may be significant for refining and optimizing feature generator 504 to improve accuracy.
[0106] Figure 13 illustrates an exemplary flowchart 1300 for detecting and classifying multiple phenotypes according to some aspects of this disclosure. The boxes in the flowchart are shown in a specific order, but this order can be modified; for example, some boxes may be performed before others, and some boxes may be performed simultaneously. These boxes can be performed by hardware or software, or a combination thereof. At box 1302, the process can begin by accessing a multiplexed image 104 generated by the image generation system 102, which depicts a biological sample that can be labeled with multiple biomarkers. Multiplexing multiple biomarkers can identify various biomarkers and cellular components, thereby enabling visualization and comprehensive analysis of spatial relationships, thus providing information for diagnosing biological conditions and creating personalized treatment plans.
[0107] In box 1304, a set of label types can be identified, such that each label type can correspond to a marker among multiple markers corresponding to the multiple images 104. The set of label types can be used to classify biological or molecular characteristics, thereby representing the expression of specific markers, such as the presence, expression level, or location of a target molecule. Label types can correspond to proteins, such as cell membrane proteins (e.g., HER2) or nuclear proteins (e.g., ER or PR), providing detailed insights into the cellular composition and function within a tissue sample. In box 1306, multiple labels can be identified for each label type in the set of label types. Multiple labels can represent predictions about cells (e.g., negative, intermediate-confidence, or positive expression for each label type, or a depiction of non-tumor biological components). In box 1308, a set of interphase signal maps 506 for each label can be generated by processing the multiple images 104 using a machine learning model or feature generator 504. Each interphase signal map can include a prediction of each part or cell location, indicating whether it corresponds to the corresponding label. This metric can help assess pixel consistency, thereby enabling more accurate mapping of cell features and biomarker distribution.
[0108] In box 1310, a set of interphase nuclear signal maps can be generated by processing the interphase signal maps for each label (e.g., ER positive, ER negative, or non-tumor). The set of interphase nuclear signal maps can identify partial or cell locations, predicting each label corresponding to a nuclear location, which can be detected using methods such as contour detection or component labeling. Nuclear locations can be labeled on the map as centroids, bounding boxes, or ellipses, or visualized as binary or RGB images, allowing for the highlighting of detected nuclei for clear representation. In box 1312, a set of cell-specific point locations can be generated by processing at least two interphase signal maps and / or interphase nuclear signal maps using one or more mapping functions. One or more mapping functions can be based on mathematical operations on data structures such as matrices to adjust the data based on signal intensity or distribution, thereby enhancing the sensitivity and resolution of the feature generator 504. The mapping functions can help identify precise locations in the multiple images 104 that can correspond to individual cells or specific regions, allowing for relatively accurate insights into the distribution and concentration of biomarkers. In RGB images, a distinct color can be assigned to each cell at a specific location for clear visualization. In some respects, interim signal maps for multiple labels can correspond to the same label type (e.g., ER+, ER-, non-tumor) and can be processed together, allowing the selection of the label with the highest probability to improve the accuracy of diagnosis and treatment planning.
[0109] In box 1314, output or annotated image set 112 can be generated for each cell-specific point location, indicating at least one predicted label corresponding to that location. The annotated image set 112 can be created by overlaying cell-specific locations onto multiple images 104 using an annotator 510. On multiple images 104, each label can be represented by a different color, corresponding to a specific biomarker that can be assigned to each part (e.g., cell location, pixel, or segmentation region). In box 1316, the output can be used to identify one or more biological conditions that can be associated with different biomarkers, such as cancer, osteoporosis, uterine fibroids, autoimmune diseases, hormone-related conditions, cardiovascular diseases, PCOS, menstrual disorders, neurological diseases, and some thyroid conditions.
[0110] Figure 14A illustrates a side-by-side comparison of superimposed estimates of PR and ER on a first triplet image and synthesized single PR and ER images according to some aspects of this disclosure. The images on the left are generated by superimposing cell-specific point locations generated by the multi-label classification pipeline 108 onto first triplet images for PR 1402 and ER 1406 label types, respectively. The images on the right are generated by superimposing cell-specific point locations onto synthesized images for PR 1404 and ER 1408 images, respectively. The synthesized images may be single images generated by a color demixing technique that demixes the triplet image into its constituent single images.
[0111] Figure 14B illustrates a side-by-side comparison of superimposed estimates of PR and ER on second triple images and augmented images according to some aspects of this disclosure. The images on the left are generated by superimposing cell-specific point locations generated by the multi-label classification pipeline 108 onto second triple images for PR 1410 and ER 1414 label types, respectively. The images on the right are generated by superimposing cell-specific point locations onto augmented images for PR 1412 and ER 1416, respectively. Augmented images can be generated by applying various transformations to the triple images to increase the number of training samples.
[0112] Figure 15 illustrates a side-by-side comparison of HER2 overlay estimates on a third triplet image and an enhanced image, according to some aspects of this disclosure. The image on the left can be generated by overlaying cell-specific point locations generated by the multi-label classification pipeline 108 onto a third triplet image on the HER2 1502 image. The image on the right can be generated by overlaying cell-specific point locations onto an enhanced image 1504, which can be generated by applying various transformations to the triplet image to increase the number of training samples.
[0113] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.
[0114] Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of the methods and / or part or all of the processes disclosed herein.
[0115] The terms and expressions used are descriptive rather than restrictive, and in using such terms and expressions, no equivalent of the features shown and described or any part thereof is intended to be excluded. However, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, those skilled in the art can employ modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.
[0116] This description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, this description of preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing various embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.
[0117] Specific details are set forth in this description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without specific details. For example, circuits, systems, networks, processes, and other components may be shown as parts in block diagram form to avoid obscuring the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.
Claims
1. A computer-implemented method, comprising: Access multiple images depicting biological samples labeled with a variety of different markers, wherein the various different markers identify different biomarkers or cellular components relative to each other; Identify a set of label types, wherein each label type in the set of label types corresponds to a marker among the multiple different markers; For each tag type in the set of tag types: Identify multiple labels, where each of the multiple labels represents a prediction for the following: Negative expression for the marker corresponding to the label; A given medium confidence expression for the marker corresponding to the label; Positive expression of markers corresponding to the label; or Description of biological components that are not tumor cells; For each of the plurality of labels, a set of midterm signal maps is generated by processing the multiple images using a machine learning model associated with the label type, wherein each midterm signal map in the set of midterm signal maps includes a prediction of whether each of one or more portions of the midterm signal map is consistent with the label; For each of the plurality of labels, a set of interim nuclear signal maps corresponding to the set of interim signal maps is generated, wherein each interim nuclear signal map in the set of interim nuclear signal maps identifies a representative location of a cell predicted to correspond to the label; A set of cell-specific point locations is generated by processing at least two of the mid-phase signal map set or at least two of the mid-phase nucleus signal map set using one or more mapping functions. Generate output, the output comprising: for each of at least some of the point locations in the set of cell-specific point locations, an indication of at least one label predicted to correspond to the point location, wherein the at least one label comprises a label from a set of labels corresponding to one or more label types; as well as The output is used to identify one or more biological conditions associated with the various different biomarkers.
2. The computer-implemented method according to claim 1, wherein the tag types in the tag type set are: A machine learning model associated with the label type is trained to assign bias weights to training elements corresponding to the first label among the plurality of labels relative to the second label among the plurality of labels.
3. The computer-implemented method of claim 1, wherein the machine learning model associated with at least one label type is trained using a binary cross-entropy loss function.
4. The computer-implemented method of claim 1, wherein the one or more mapping functions for generating the set of cell-specific point locations comprise a matrix or array, the matrix or array comprising coefficients for each element in the matrix for the array.
5. The computer-implemented method according to claim 1, wherein the set of tag types includes at least three tag types.
6. The computer-implemented method of claim 1, wherein at least one tag type in the tag type set includes a tag type corresponding to a specific cell membrane protein; and At least one other tag type in the set of tag types includes a tag type corresponding to a specific nuclear protein.
7. The computer-implemented method of claim 1, wherein the machine learning model includes the UNet model.
8. A system comprising: One or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform a set of operations, the set of operations including: Access multiple images depicting biological samples labeled with a variety of different markers, wherein the variety of different markers identify different biomarkers and / or cellular components relative to each other; Identify a set of label types, wherein each label type in the set of label types corresponds to a marker among the multiple different markers; For each tag type in the set of tag types: Identify multiple labels, where each of the multiple labels represents a prediction for the following: Negative expression for the marker corresponding to the label; A given medium confidence expression for the marker corresponding to the label; Positive expression of markers corresponding to the label; or Description of biological components that are not tumor cells; For each of the plurality of labels, a set of midterm signal maps is generated by processing the multiple images using a machine learning model associated with the label type, wherein each midterm signal map in the set of midterm signal maps includes a prediction of whether each of one or more portions of the midterm signal map is consistent with the label; For each of the plurality of labels, a set of interim nuclear signal maps corresponding to the set of interim signal maps is generated, wherein each interim nuclear signal map in the set of interim nuclear signal maps identifies a representative location of a cell predicted to correspond to the label; A set of cell-specific point locations is generated by processing at least two of the mid-phase signal map set and / or at least two of the mid-phase nucleus signal map set using one or more mapping functions; Generate output, the output comprising: for each of at least some point locations in the set of cell-specific point locations, an indication of at least one label predicted to correspond to the point location, wherein the at least one label comprises labels from a set of labels corresponding to one or more label types; and The output is used to identify one or more biological conditions associated with the various different biomarkers.
9. The system of claim 8, wherein for the tag types in the tag type set: A machine learning model associated with the label type is trained to assign bias weights to training elements corresponding to the first label among the plurality of labels relative to the second label among the plurality of labels.
10. The system of claim 8, wherein the machine learning model associated with at least one label type is trained using a binary cross-entropy loss function.
11. The system of claim 8, wherein the one or more mapping functions for generating the set of cell-specific point locations comprise a matrix or array, the matrix or array comprising coefficients for each element in the matrix for the array.
12. The system of claim 8, wherein the set of tag types includes at least three tag types.
13. The system of claim 8, wherein at least one tag type in the set of tag types includes a tag type corresponding to a specific cell membrane protein; and At least one other tag type in the set of tag types includes a tag type corresponding to a specific nuclear protein.
14. The system of claim 8, wherein the machine learning model includes the UNet model.
15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform a set of operations, the set of operations including: Access multiple images depicting biological samples labeled with a variety of different markers, wherein the variety of different markers identify different biomarkers and / or cellular components relative to each other; Identify a set of label types, wherein each label type in the set of label types corresponds to a marker among the multiple different markers; For each tag type in the set of tag types: Identify multiple labels, where each of the multiple labels represents a prediction for the following: Negative expression for the marker corresponding to the label; A given medium confidence expression for the marker corresponding to the label; Positive expression of markers corresponding to the label; or Description of biological components that are not tumor cells; For each of the plurality of labels, a set of midterm signal maps is generated by processing the multiple images using a machine learning model associated with the label type, wherein each midterm signal map in the set of midterm signal maps includes a prediction of whether each of one or more portions of the midterm signal map is consistent with the label; For each of the plurality of labels, a set of interim nuclear signal maps corresponding to the set of interim signal maps is generated, wherein each interim nuclear signal map in the set of interim nuclear signal maps identifies a representative location of a cell predicted to correspond to the label; A set of cell-specific point locations is generated by processing at least two of the mid-phase signal map set and / or at least two of the mid-phase nucleus signal map set using one or more mapping functions; Generate output, the output comprising: for each of at least some of the point locations in the set of cell-specific point locations, an indication of at least one label predicted to correspond to the point location, wherein the at least one label comprises a label from a set of labels corresponding to one or more label types; as well as The output is used to identify one or more biological conditions associated with the various different biomarkers.
16. The computer program product of claim 15, wherein the tag types in the set of tag types are: A machine learning model associated with the label type is trained to assign bias weights to training elements corresponding to the first label among the plurality of labels relative to the second label among the plurality of labels.
17. The computer program product of claim 15, wherein the machine learning model associated with at least one label type is trained using a binary cross-entropy loss function.
18. The computer program product of claim 15, wherein the one or more mapping functions for generating the set of cell-specific point locations comprise a matrix or array, the matrix or array comprising coefficients for each element in the matrix for the array.
19. The computer program product of claim 15, wherein the set of tag types includes at least three tag types.
20. The computer program product of claim 15, wherein at least one tag type in the set of tag types includes a tag type corresponding to a specific cell membrane protein; and At least one other tag type in the set of tag types includes a tag type corresponding to a specific nuclear protein.
