Breast cancer classification

Abstract

The present invention relates to breast cancer classification.SOLUTION: The method includes applying a first machine-learning (ML) model to copy number variation (CNV) information associated with a subject to generate first output-logits that each correspond to a plurality of breast carcinoma subtypes, applying a second 2ML model to histopathology image information associated with the subject to generate second output-logits that each correspond to the plurality of breast carcinoma subtypes, applying a third 3ML model to medical information associated with the subject that includes information related to a diagnosis of breast carcinoma in the subject to generate third output-logits that each correspond to the plurality of breast carcinoma subtypes, computing a weighted combination of the first output-logits, the second output-logits, and the third output-logits, and classifying the subject's breast carcinoma as a subtype of the plurality of breast carcinoma subtypes based on the weighted combination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to breast cancer classification, and in particular to computer-implemented methods, computer programs and information programming devices. [Background technology]

[0002] Breast cancer is the most commonly diagnosed cancer among women worldwide. However, simply diagnosing breast cancer is a gross simplification, as breast cancer exhibits genetic and clinical heterogeneity that affects prognosis and treatment planning.

[0003] Cancer is a heterogeneous disease that exhibits genetic and molecular diversity. Classifying cancers into more detailed subtypes can help tailor treatment plans that can improve patient outcomes. The PAM50 breast cancer subtype classification is a widely used genetic classification based on 50-gene RNA analysis of tumors and is actually used for prognosis prediction and treatment adjustment. However, RNA profiling is an expensive and challenging process and is not always a routine procedure in clinical practice.

[0004] In light of the above, a method for classifying breast cancer is desirable. Summary of the Invention

[0005] The invention is defined in the independent claims, to which reference should now be made. Particular embodiments are set out in the dependent claims.

[0006] According to one embodiment of the first aspect, there is provided a computer-implemented method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; A method is disclosed that includes:

[0007] By way of example, reference is made to the accompanying drawings in which: [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of a system. [Figure 2] FIG. 1 is a diagram illustrating an overview of the system. [Figure 3] FIG. 1 is a diagram illustrating an overview of the system. [Figure 4] FIG. 10 is a diagram illustrating training data. [Figure 5] FIG. 1 is a diagram illustrating SELU. [Figure 6] FIG. 1 illustrates a first ML model architecture. [Figure 7] This is a diagram that helps understand the 2ML model. [Figure 8] FIG. 2 illustrates a second ML model architecture. [Figure 9] FIG. 1 illustrates a third ML model architecture. [Figure 10] FIG. [Figure 11] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 12] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 13] FIG. 1 illustrates a WSI in an operational example. [Figure 14] FIG. 1 illustrates a WSI in an operational example. [Figure 15] This is a diagram that helps understand the 2ML model. [Figure 16] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 17] This is a diagram that helps understand the 2ML model. [Figure 18] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 19] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 20] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 21] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 22] FIG. 10 is a diagram illustrating the results of an operation example. [Figure 23] FIG. 1 illustrates a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following definitions may be used in the description but are not exhaustive.

[0010] Copy Number Variation (CNV): Copy number variation is the quantification of variations, repeats, and duplications that affect a significant number of base pairs of DNA.

[0011] Electronic Health Records (EHR): An electronic health record is a collection of a patient's clinical data in digital form. This includes information such as medical history, medications, lab tests, and demographics.

[0012] Whole-Slide Image (WSI): A whole-slide image is a digital, high-resolution scan of a histopathology slide. WSIs are typically full-resolution, multi-gigapixel images stored with down-sampled versions, if necessary.

[0013] Ribonucleic Acid (RNA): RNA is a macromolecule essential for cellular function, either by itself (e.g., certain non-coding RNAs) or as a template for protein synthesis.

[0014] RNA-Seq: RNA sequencing is a technique that uses next-generation sequencing to provide a snapshot of gene expression, or the transcriptome, within a tissue sample.

[0015] Convolutional Neural Network (CNN): A CNN is a neural network model that combines linear filters and nonlinear activations to perform machine learning tasks. It is most commonly used in computer vision applications.

[0016] Support Vector Machine (SVM): SVM is a classic supervised machine learning model that can be used for classification and regression. It is motivated by finding the maximum margin hyperplane.

[0017] Circulating Tumor DNA (ctDNA): ctDNA refers to fragmented DNA in the bloodstream that is not associated with cells and is derived from the tumor.

[0018] Tumor Micro-Environment (TME): The TME refers to the complex ecosystem and cellular makeup that surrounds a tumor and influences its behavior.

[0019] Fusion: In machine learning, fusion refers to combining multiple data sources to perform a specific downstream task. It can generally be categorized into intermediate fusion and late fusion. Intermediate fusion is based on combining intermediate representations from multiple models to perform a downstream task, while late fusion relies on the output of independent models.

[0020] PAM50: PAM50 is a molecular subtyping method for breast cancer based on a 50-gene signature. There are typically five subtypes: Luminal A, Luminal B, Her2-enriched, Basal-like, and Normal-like. Often, only four of the subtypes are used (normal-like is omitted).

[0021] Self-Normalizing Network (SNN): A self-normalizing network is a forward-propagation fully connected neural network designed to preserve activation normalization, making it useful for low sample size data with depth and high dimensionality.

[0022] Integrated gradients (IG): Integrated gradients is an input attribute and explainability method for differentiable machine learning models. It is based on taking the path integral of the gradient from the baseline to the input.

[0023] FIG. 1 is a schematic diagram of a system 100 that includes a CNV module 10, a WSI module 20, an EHR module 30, and an integration module 60. Modules 10, 20, and 30 each apply a machine learning (ML) model to a corresponding type of data associated with a subject. The ML model is for classifying the type of data associated with the subject to correspond to a subtype among multiple breast cancer subtypes (e.g., PAM50 subtypes). Modules 10, 20, and 30 provide output logits indicative of their predictions to integration module 60, which performs late fusion and classifies the subject's breast cancer as a subtype among multiple breast cancer subtypes based on the output logits.

[0024] The CNV module 10 is responsible for CNV data preprocessing and modality-specific prediction. After preprocessing the CNV data, module 10 extracts features from the preprocessed data. This is followed by providing prediction and explainability for the data. Module 10 provides an indication of the attributes of the different input genes, indicating the importance of each to the module's 10 decision.

[0025] The WSI module 20 is responsible for processing WSI data. WSIs are multi-gigapixel images that are difficult (time- and / or resource-expensive) to process as a single unit. This module 20 is responsible for segmentation and tile extraction from the WSI, allowing neural networks (and associated ML models) to be applied to the data. After dividing the WSI into smaller patches, module 20 filters empty patches based on their tissue content. Module 20 can operate on patches that are downsampled by 20x, for example, to help with computational efficiency. After preprocessing and feature extraction, the WSI module 20 generates predictions for each patch. Module 20 is responsible for aggregating patch predictions into a single prediction for a given WSI (and therefore a given subject) using majority voting. Module 20 provides explainability of the WSI as a heatmap showing regions that are important for the model's decision.

[0026] The EHR module 30 is responsible for local processing and prediction of clinical health records. EHR features can be classified as numeric (e.g., age at diagnosis), ordinal (e.g., IHC (Immunohistochemistry) score), and categorical (e.g., ICD (International Classification of Diseases)-10 classification). Module 30 is responsible for preprocessing and converting ordinal and categorical features into numeric form. It also performs feature normalization before applying the predictive model. Module 30 generates clinical data-specific subtype predictions and provides feature attributes that indicate the contribution of clinical features to the prediction.

[0027] The integration module 60 performs late fusion to combine the outputs of the distributed modules 10, 20, and 30 into a multimodal prediction. Each module 10, 20, and 30 generates logits that encode probability predictions for the four PAM50 classes. The module outputs are serialized and forwarded to the integration module 60, which assigns weights to each modality and each class (subtype) logit. Module 60 then integrates the logits by weighting the outputs of each modality and each class, summing the contributions from all modules 10, 20, and 30. This is followed by a softmax function that normalizes the logits to a valid probability distribution that encodes the PAM50 classification probabilities.

[0028] Modules 10, 20, 30, and 60 can operate in a distributed manner on local data without requiring central data collection. Furthermore, they can operate in a heterogeneous environment, utilizing CPU or GPU processing based on modality load. Because they do not rely on RNA sequencing, the system is viable in clinical settings. The distributed nature of system 100 is supported by the integration module 60, which relies on the output (logits) of modules 10, 20, and 30, but does not necessarily rely on raw data. System 100 can provide high performance for PAM50 breast cancer subtype classification. WSI module 20 can operate on low-magnification, 20x downsampled patches, improving resource efficiency. Finally, each module 10, 20, and 30 can provide explainability for its modality input, which helps understand its decisions and provides transparency to system users.

[0029] The distributed nature of the system 100 is illustrated in FIG. 1 in that the modules 10, 20, 30, and 60 are located in different physical locations, e.g., different labs or hospitals. For example, the modules can be located where the necessary data is stored, thus minimizing / avoiding data communication. That is, patient data can reside in different systems and different locations. Single-modality modules (CNV, WSI, or EHR) can be independently located in different locations to process local data. Each module can provide modality-specific outputs and modality explainability. The integration module 60 is responsible for integrating modality-specific model outputs to provide multimodal predictions for the patient / subject.

[0030] 1 can be considered as at least one processor configured to perform the described functions of the module. The at least one processor can perform the operations of all the modules.

[0031] FIG. 2 illustrates the method steps / operations performed by each module 10, 20, 30, 60 in our implementation. Each single-modality module 10, 20, 30 performs preprocessing steps S12, S22, S32 to preprocess the data. The WSI module performs step S23 to divide the WSI into tiles and extract relevant tiles. The EHR module performs step S33 to convert ordinal and categorical features into numerical features. Each module 10, 20, 30 performs feature extraction (S14, S24, S34) to extract features from relevant data and classification steps (S16, S26, S36) to predict the subject's breast cancer subtype. Since the WSI module performs classification tile-by-tile, a pooling step (S28) follows to pool the predictions per tile. The output logits from each module 10, 20, 30 are used in a late fusion step S60 by a fusion module 60 to predict the PAM50 classification of breast cancer for the subject.

[0032] Training of the system 100 occurs in two stages. First, each modality-specific ML model is trained for that modality using the classification target. After convergence, the weights of the single-modality ML model are fixed. Next, the output of the modality-specific model is used to train the weights used in the fusion module 60 with the same classification target.

[0033] 3 is a schematic diagram illustrating the functionality of modules 10, 20, 30, and 60. Modules 10, 20, and 30 each perform a single-modality prediction, and the resulting output logits are used in fusion module 60 to generate a multi-modality prediction.

[0034] As mentioned above, modules 10, 20, 30, and 60 are not required, but provide a useful division of operations for purposes of explanation. Processing of CNV data (i.e., the operation of CNV module 10) can be thought of as using the first ML model, processing of WSI data (i.e., the operation of WSI module 20) as using the second ML model, and processing of EHR data (i.e., the operation of EHR module 30) as using the third ML model. The processing of CNV, WSI, and EHR data and the first through third ML models are described according to the following implementation example.

[0035] CNV data processing CNV data quantify duplications and deletions of DNA segments. For each gene, CNVs provide discrete values ​​that describe this information. Deep deletion: -2 Shallow deletion: -1 Diploid: 0 Gain: 1 Amplification: 2

[0036] A value of 0 means no repeat or deletion, positive values ​​indicate repeats, and negative values ​​indicate deletions. Below is a random sample of 15 genes and their indices taken from the XXXXXX dataset (described below).

number

[0037] Preprocessing includes removing genes with missing CNV values, and further includes z-scoring the CNV values ​​using the mean and standard deviation of the training set (on which the first ML model was trained) by:

number

[0038] The first ML model includes a Self-Normalizing Network (SNN). An SNN is a forward-propagation fully connected network designed to preserve activation normalization. This can be useful for low-sample size data with depth and high dimensionality, such as those commonly found in CNV data. SNNs differ from standard forward-propagation networks in two ways. First, they use the Scaled-Exponential Linear Unit (SELU) as their activation function (Figure 5 shows a graph of the Scaled-Exponential Linear Unit). Second, they use a variation of Dropout called Alpha-Dropout, which is better suited to SELU activations and preserves self-normalization.

[0039] Figure 6 shows the specific architecture used in a specific implementation of the 1ML model. This architecture includes an SNN with five hidden layers with 8192, 4096, 2048, 512, and 128 channels. The CNV data for this specific implementation contains 23k records.

[0040] Integrated Gradients (IG) can be used to provide explainability for CNV data processing. IG is a machine learning explainability methodology that assigns attributes to each feature in the input by accumulating predictive gradients along the path integral from the baseline to the input. IG can be used to obtain attribute scores for each gene in the input. The absolute value of the attribute scores can be used to rank genes based on their contribution to the first ML model's decision. This provides the system user with the importance of each gene for model prediction, which is useful for explainability.

[0041] CNV data can be considered a genetic modality. Another genetic modality that can be used is RNA-seq. A first auxiliary ML model with the same architecture as the first ML model can be used to generate output logits based on the RNA-seq data. Preprocessing of the RNA-seq data includes removing genes with missing values ​​and transforming the data using a shifted Log2. The first auxiliary ML model can be implemented by the CNV module 10 or another module similar to modules 10, 20, and 30, such as an RNA module. The integration module 60 can include the output logits from the RNA-seq processing (first auxiliary ML model) in its processing.

[0042] WSI data processing WSIs are typically multi-gigapixel images of tissue samples. Using WSIs for machine learning presents several challenges. Due to their large size and resolution, inputting the entire WSI into a machine learning model in one pass may be infeasible due to memory and storage constraints. Another complication is that large regions of the WSI are typically background that do not provide information to the learning process.

[0043] Preprocessing of the WSI data involves dividing the WSI into non-overlapping patches. After downsampling each patch, we 1) convert the patch to Hue-Saturation-Value (HSV) color space, 2) apply a median blur to the saturation channel (e.g., using a kernel size of 7), and 3) generate a tissue mask by thresholding the saturation channel using a binarization threshold (e.g., 20 for pixel values ​​between 0 and 255). Then, based on the generated mask, patches are selected only if their tissue content exceeds 5%.

[0044] In other words, a WSI is a digitized, multi-gigapixel image of pathological tissue. Because processing the entire WSI raw is time- and resource-intensive, the WSI is first tiled into small patches that can be processed individually by the neural network of a 2D ML model. However, large regions of the WSI are typically background, providing no information to the prediction process and only increasing computational complexity. Therefore, patches that do not contain sufficient information are removed as described above.

[0045] An example of WSI preprocessing is shown in Figure 7. "A" shows an example of an input WSI (shown in black and white, but in reality it is in color). First, the tissue content is separated from the background using the following segmentation algorithm: The WSI is tiled into non-overlapping patches (e.g., 256x256). The image is converted from RGB to HSV color space, which breaks down the color space into Hue, Saturation, and Value or Brightness. The chroma channel is filtered using a median blur. Median blur uses a moving window or kernel to replace the target pixel with the median value within the window. A kernel size of 7 may be used. The mask is generated by applying a threshold to the filtered saturation channel. A binarization threshold of 20 for pixel values ​​between 0 and 255 can be used.

[0046] An example of the generated mask is shown in (B). The patches are then filtered based on tissue content. The mask is used to determine the percentage of each patch's area that is occupied by tissue. Patches are only accepted if the tissue content exceeds a certain threshold (e.g., a 5% threshold). Accepted patches are then downsampled (e.g., 20x) to reduce computational complexity. (C) shows the patch filtering process. Patches indicated by solid lines are accepted, while patches indicated by dashed lines are rejected. Preprocessing further involves transforming each patch as follows: Patches may be resized (e.g. to 342x342) using bilinear interpolation. Center cropping (e.g. 299x299) can be applied. Pixel values ​​can be rescaled to [0.0, 1.0] (i.e., rescaled to lie between 0 and 1) and normalized using the mean and standard deviation of the training WSI data.

[0047] The 2ML model can include a neural network, a fully connected neural network, or a convolutional neural network. In a specific implementation, the 2ML model includes a classification head on top of a backbone of a pre-trained computer vision model (e.g., Inceptionv3 (Szegedy, C., et al. (2016), Rethinking the inception architecture for computer vision, In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826)), VGG16 (K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014)), and Dinov2 (M. Oquab, et al., Dinov2: Learning robust visual features without supervision, arXiv preprint arXiv:2304.07193 (2023))).

[0048] Each subject / patient can have multiple WSIs, and each WSI contributes to multiple patches. The ground truth labels for downstream tasks are per-patient. Therefore, a pooling strategy is used to convert patch-level predictions to patient-level predictions. For example, two common pooling strategies are based on pooling the classifier outputs and pooling the intermediate representations. For output pooling, the following examples can be used: majority voting, mean of output logits, mean of output probabilities. For intermediate representation pooling, the following examples can be used: using distance from the mean, using either the mean or weighted mean.

[0049] FIG. 8 shows the architecture of a 2ML model according to a specific implementation example, which includes a classification head (in this case an MLP 83) on top of an Inceptionv3 backbone 82 and accepts WSI data 81 as input.

[0050] The explainability of the 2ML model can be provided as a heatmap using integrated gradients. IG can be used to calculate attributes from each pixel of each accepted WSI patch. The sum of the attribute scores across channels is taken, and then the absolute value is taken. A heatmap of the top 25% of pixels contributing to the 2ML model's decision can be overlaid on a grayscale version of the (original) WSI to show the system user which regions contributed most to the decision.

[0051] EHR Data Processing A subject's (individual / patient) EHR data in the context of this disclosure may be referred to as "medical data relating to the subject, including information relating to the subject's diagnosis of breast cancer."

[0052] EHRs typically contain clinical data, history, lab tests, demographics, and other patient clinical information. Preprocessing may include removing features for which more than a threshold amount of training data records / instances are absent (e.g., removing features with more than 75% missing values). Uninformative features, such as identifiers and constant fields from the EHR, are filtered out. Remaining features are classified as numeric (e.g., age at diagnosis), ordinal (e.g., IHC score), or categorical (e.g., ICD-10 classification). Categorical and ordinal values ​​are converted to numeric values. Preprocessing further includes one-hot coding categorical values ​​and applying z-scoring (based on the mean and standard deviation of the training data used to train the 3ML model) according to:

number

[0053] Examples of features that may be included in medical data related to a subject are provided below, along with descriptions of some exemplary values. Age at diagnosis Age at which subject was first diagnosed with breast cancer 62.0, 63.0, 61.0, 54.0, 50.0 Days until sample collection The time between the subject's initial diagnosis of breast cancer and the time the tissue sample was taken 57.0, 63.0, 394.0, 637.0, 162.0 Disease-free period (months) The period of time a subject was disease-free between a previous breast cancer diagnosis and the current breast cancer diagnosis 0.33, 0.0, 13.01, 12.32, 12.65 Fraction Genome Altered Abnormal gene fraction (FGA) measurement 0.0, 0.0001, 0.0002, 0.1153, 0.0004 Positive Finding Lymph Node Hematoxylin and Eosin Staining Microscopy Count Number of positive lymph nodes based on hematoxylin-eosin microscopy 0.0, 1.0, 2.0, 3.0, 4.0 Positive Finding Lymph Node Keratin Immunohistochemistry Staining Count Number of positive lymph nodes based on keratin immunohistochemical staining microscopy 0.0, 1.0, 2.0, 3.0, 4.0 Number of lymph node examinations Number of lymph nodes examined 2.0, 1.0, 3.0, 4.0, 10.0 Number of mutations Number of mutations 24.0, 22.0, 19.0, 23.0, 21.0 Sample initial weight Tissue sample weight (of tissue taken for breast cancer diagnosis / analysis) 110.0, 120.0, 100.0, 130.0, 200.0 TMB (non-synonymous) Nonsynonymous Tumor Mutation Burden (TMB) 0.633333333, 0.766666667, 0.7, 0.6, 0.833333333 ER Status IHC Percent Positive Estrogen receptor positive rate 90-99%, <10%, 70-79%, 80-89%, 10-19% HER2ICH score HER2ICH test results 1.0, 2.0, 3.0, 0.0 PR Status IHC Percent Positive Progesterone receptor positivity rate <10%, 90-99%, 70-79%, 10-19%, 80-89% American Joint Committee on Cancer Metastatic Stage Codes American Joint Committee on Cancer (AJCC) metastatic stage codes M0, MX, M1, cM0(i+) Neoplastic Disease Lymph Node Stage American Joint Committee on Cancer Code AJCC Neoplastic Disease Lymph Node Stage Codes N0, N1a, N0(i-), N1, N2a Neoplastic Disease Stage American Joint Committee on Cancer Code AJCC Neoplastic Disease Stage Codes StageIIA, StageIIB, StageIIIA, StageI, StageIA American Joint Committee on Cancer Publication Version Type A version or edition of a publication by a group established for the purpose of developing a clinical staging system for cancer that is accepted by medical professionals in the United States and compatible with other accepted classifications. 6th, 7th, 5th, 4th, 3rd American Joint Committee on Cancer tumor stage codes AJCC Tumor Stage Codes T2, T1c, T3, T1, T4b Cancer type details Types of Breast Cancer Invasive ductal carcinoma of the breast, invasive lobular carcinoma of the breast, mixed ductal lobular carcinoma of the breast, mixed invasive mucinous carcinoma of the breast, metaplastic breast cancer disease-free state Disease-free status of the subject 0: disease-free, 1: recurrence / progression ER status by IHC Breast Cancer Estrogen Receptor Status positive, negative, unknown Ethnic Category Subject's ethnicity Not Hispanic or Latino, Hispanic or Latino HER2 FISH status HER2FISH test results Negative, positive, equivocal, unclear, [unassessed] Neoplasm Histologic Type Name Histological type of tumor Invasive ductal carcinoma, invasive lobular carcinoma, other, mixed histology, mucinous carcinoma Text of type of neoadjuvant therapy administered before resection Indication of whether the subject received neoadjuvant therapy No, Yes Previous cancer diagnosis An indication of whether the subject has ever been diagnosed with cancer No, Yes ICD-10 Classification ICD-10 classification C50.9, C50.3, C50.4, C50.2, C50.8 International Classification of Diseases oncology, 3rd edition, ICD-O-3 Histology Code ICD-O-3 Histology Code 8500 / 3, 8520 / 3, 8522 / 3, 8523 / 3, 8480 / 3 International Classification of Diseases oncology (3rd edition) ICD-O-3 site code ICD-O-3 site code C50.9, C50.3, C50.4, C50.2, C50.8 IHC-HER2 HER2 IHC test results (expressed in a different format than the above information "HER2 IHC score") Negative, equivocal, positive, unknown Primary lymph node metastasis evaluation Ind-3 Indication of whether lymph node evaluation was performed YES, NO Menopause Status Menopausal status Postmenopausal (history of bilateral oophorectomy or LMP more than 12 months ago but no history of hysterectomy), premenopausal (less than 6 months ago LMP and no history of bilateral oophorectomy and not receiving estrogen replacement therapy), premenstrual (6-12 months after last menstrual period), unknown (neither premenopausal nor postmenopausal) Metastatic tumor indicator Metastatic tumor index No, Yes First Pathologic Diagnosis Biospecimen Acquisition Method Type Initial pathological diagnosis biopsy collection method type Core needle biopsy, tumor resection, fine needle aspiration biopsy, other methods, specified: excision biopsy Micromet detection by ihc Indicators of micrometastasis No, Yes OCT embedding Whether an optimal cutting temperature (OCT) process was used before slicing the tissue samples onto slides for analysis True, False Oncotree Code Oncotree Code IDC, ILC, MDLC, IMMC, MBC Disease Surgical Margin Status Surgical margin status Negative, positive, adjacent Patient's primary tumor site Primary tumor site Left upper lateral quadrant, left, right upper lateral quadrant, right, left upper inner quadrant Future harvest indicators for tissue A textual indicator of the tissue acquisition period, indicating that the tissue was acquired in parallel with or prior to the study (this information may only be included in "research" based training data) No, Yes PR status by IHC Breast cancer progesterone receptor status positive, negative, unknown Racial Categories Subject's racial information White, Black or African American, Asian, American Indian or Alaska Native Past collection indicators of tissue A textual indicator of the tissue acquisition period indicating that the tissue was acquired and stored before the start of the study (this information may only be included in "study" based training data) YES, NO Staging System The process (staging system) used to determine the extent of breast cancer growth and spread Axillary lymph node dissection only, Sentinel lymph node biopsy only, Sentinel lymph node biopsy + axillary dissection, No axillary dissection, Other First, surgical procedures breast cancer surgery Modified radical mastectomy, other, lumpectomy, simple mastectomy Tissue Source Site Tissue origin BH, A2, E2, A8, D8 Human tumor conditions Tumor status No tumor, tumor present Survival time (months) Subject survival time since breast cancer diagnosis 0, 1, 6, 12

[0054] The 3ML model can include a neural network. In an example implementation, the model architecture can include a classification head on top of an MLP backbone. The MLP network can be applied directly to numerical data. The backbone generates intermediate representations, and the classification head generates modality-specific predictions (output logits).

[0055] In the example implementation, numeric data is converted into textual key-value pairs, where the key is the feature name and the value is the corresponding feature value. In this example, the 3ML model includes a classification head on top of a Transformer encoder. The Transformer generates an intermediate representation, and the classification head on top of the intermediate representation generates clinical modality predictions (output logits). For example, if the Transformer encoder is used, pre-trained BERT and RoBERTa can be used after fine-tuning (J. Devlin, et al., Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018) and Y. Liu, et al., Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)).

[0056] In a specific implementation example, as shown in Figure 9, the third ML model includes an MLP with 128-channel and 64-channel hidden layers, ReLU activation, and instance normalization.

[0057] The integrated gradient can be used to provide explainability of clinical features. IG can be used to calculate attribution for each feature in the input. Since one-hot coding contributes multiple features per raw categorical feature, the contributions of all one-hot vectors are summed and assigned to the raw feature. Absolute values ​​are taken to rank the features according to their contribution to the prediction. This provides the system user with an indication of which features were more important in reaching the 3ML model's decision.

[0058] Fusion Processing The fusion process involves combining the outputs (logits) of the single-modality models into a multimodal prediction. The fusion process uses the output logits from the first, second, and third single-modality ML models. For each modality and each class output, the fusion process uses learnable weight parameters. The modality-specific outputs are scaled using these weights and summed to produce a multimodal weighted logit prediction, which is then normalized to a multimodal probability output by applying a softmax function.

[0059] If the output logit of model i (1st to 3rd ML models, i=1, 2, 3) is:

number

number

[0060] As mentioned above, subjects may have data in different locations. This could be CNVs performed in a specific lab, clinical history and EHRs stored at a specific hospital, or WSIs stored in yet another lab. Data does not need to be moved to central data storage to make multimodal predictions. Each institution can use relevant single-modality ML models on their own local data stored on-premise or in private storage. The single-modality ML models can process the data locally and generate single-modality predictions and explainability. The model outputs are then serialized and communicated for use in the fusion process. The fusion process can then generate multimodal predictions using only the model outputs (logits), without any information about the data used to generate these outputs.

[0061] training As mentioned above, training is performed in two stages: in the first stage, the first to third ML models are trained (sometimes referred to as the first to third ML model training processes, respectively), and in the second stage, the weights used in the fusion process are trained while the weights of the first to third ML models are fixed (sometimes referred to as the fourth training process).

[0062] Generally, the training process of each of the first to third ML models includes predicting a subtype of the training subject from among multiple breast cancer subtypes using training data (CNV / WSI / medical data) related to the training subject, comparing the predicted subtype with ground truth subtypes or ground truth data representing the training subject, and adjusting at least one weight of the ML model based on the comparison. The training of the weights for each ML model and the fusion process are described in detail below.

[0063] In the training implementation, CNV, WSI, and EHR data from the TCGA-BRCA dataset are used (WSI is obtained directly from TCGA (cancer.gov / ccg / research / genome-sequencing / tcga), while CNV and EHR are obtained from cBioPortal (cbioportal.org / )). The PAM50 breast cancer classification is obtained from an independent publication (Netanely, D., et. al. (2016), Expression and methylation patterns partition luminal-A breast tumors into distinct prognostic subgroups, Breast Cancer Research, 18, 1-16). The PAM50 classification in the publication was obtained using RNA analysis and is therefore considered a ground truth label. Instances that do not include all three data modalities or that do not have a PAM50 classification are excluded. In this training implementation, the number of cases after filtering is 977.

[0064] PAM50 classifies each case into one of four breast cancer subtypes (sometimes five, as mentioned above, but "normal-like" is omitted here): Luminal A, Luminal B, Her2-enriched, and Basal-like. Figure 4 shows the subtypes of 977 instances in the training data. Based on the PAM50 subtyping, the dataset is highly imbalanced, as shown in Figure 4. Therefore, to achieve better classification performance, a balancing strategy is used. Two balancing strategies are used: oversampling and loss weighting. Oversampling involves resampling each class (subtype) to the same size as the majority class by repeating samples. Loss weighting involves rescaling the loss of each class in inverse proportion to the frequency of the class in the dataset. For each case, the dataset is divided into 80% training, 10% validation, and 10% test. The training and fusion weights for each ML model are described in detail below in relation to the implementation example using the training data mentioned above.

[0065] First ML model training The CNV data are preprocessed by removing genes with missing values, which leaves approximately 23k genes for further processing. Preprocessing involves z-scoring the CNV values ​​using the mean and standard deviation of the training data according to:

number

[0066] The weights of the 1ML model are initialized by setting the bias to 0 and initializing the other weights according to:

number

[0067] Second ML model training The training WSI data is preprocessed as described above for more general WSI data: pixel values ​​are rescaled to [0.0, 1.0] (i.e., rescaled to lie between 0 and 1) and normalized using mean=[0.485, 0.456, 0..406] and std=[0.229, 0.224, 0.225].

[0068] Training of 2ML models is subject to weak supervision. Ground truth labels (classifications) are at the subject level, so individual patches are assigned case labels. 2ML models are trained on a per-patch basis. However, for inference, the models are evaluated by pooling the outputs to produce one prediction per case / subject.

[0069] When pooling at the intermediate representation level, two strategies (among others) can be considered. The first strategy uses two stages: In the first stage, the model is trained patch by patch. A backbone is then fixed and used to encode and pool the intermediate representation. A new classifier is then used to classify the pooled intermediate representation. The other strategy is to train the model with pooling from scratch by taking the average of the sampled patches for each subject. During inference, the average of all patches for each patient is used. When aggregating at the intermediate representation level, to make the process more computationally affordable, only the top 50 patches for each subject can be used and ranked based on the region of segmented tissue content.

[0070] For Inceptionv3, VGG16, and Dinov2, the input patches are further preprocessed by: (1) resizing them to 342x342 for Inceptionv3 and 256x256 for VGG16 and Dinov2 using bilinear interpolation, (2) applying a center crop of 299x299 for Inceptionv3 and 224x224 for VGG16 and Dinov2, and (3) rescaling pixel values ​​to [0.0, 1.0] and normalizing them using mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225].

[0071] In an alternative implementation of the 2ML model, training weights pre-trained on ImageNet (Deng, J. et al (2009) Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255), IEEE) may be used in the 2ML model (but the fully connected layers may be removed).

[0072] The second ML model is trained (e.g., for 200 epochs) using stochastic optimization, e.g., the Adam optimizer.

[0073] 3. ML Model Training The training data is preprocessed as described above for more general EHR data. Additionally, during the training phase, missing values ​​are imputed using the k-Nearest Neighbors algorithm.

[0074] According to the implementation example described above, the number of input features of the training EHR data after augmentation by one-hot coding of the training data is 271. The third ML model is trained (e.g., for 200 epochs) using stochastic optimization, e.g., the Adam optimizer.

[0075] Training weights for fusion processing After the modality-specific models are trained, the fusion module is trained to perform classification using only the outputs from the other models. That is, the first, second, and third ML models are applied to training data associated with the training subject (each model applies to a specific data type), and the output logits are fed into the fusion process. The weights are used to generate a weighted sum, and therefore a classification, which is compared to ground truth data representing the corresponding ground truth classification of the training subject. Based on the comparison, the weights are adjusted. Training is performed (e.g., for 200 epochs) using stochastic optimization, e.g., the Adam optimizer.

[0076] Training schemes other than stochastic optimization (stochastic gradient descent (SGD)) may also be used, including, for example, simulated annealing and evolutionary strategies, but in deep learning, SGD is the most efficient.

[0077] FIG. 10 shows a method including steps S10 to S50, and optionally step S90.

[0078] Step S10 includes applying a first ML model to the CNV data to generate first output logits, i.e., step S10 includes applying a first ML model to the CNV data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, where the first ML model is trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes.

[0079] Step S20 includes applying a second ML model to the histopathology image data to generate second output logits, i.e., step S20 includes applying a second ML model to the histopathology image data associated with the subject to generate second output logits, each corresponding to a plurality of breast cancer subtypes, where the second ML model is trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes.

[0080] Step S30 includes applying a 3ML model to medical data to generate third output logits, i.e., step S30 includes applying a 3ML model to medical data regarding subjects that includes information regarding the subjects' diagnosis of breast cancer to generate third output logits each corresponding to a plurality of breast cancer subtypes, the 3ML model being trained to classify the medical data that includes information regarding the breast cancer diagnosis as corresponding to a subtype among the plurality of breast cancer subtypes.

[0081] Step S40 includes calculating a weighted combination of the first through third output logits, i.e., step S40 includes calculating a weighted combination of the first output logit, the second output logit, and the third output logit.

[0082] Step S60 includes classifying the subject's breast cancer, i.e., step S50 includes classifying the subject's breast cancer as a subtype of a plurality of breast cancer subtypes based on the weighted combination.

[0083] The subject may be referred to as a patient or a target patient. The method may further include outputting a diagnosis comprising the classification.

[0084] Step S10 may be considered to correspond to the functionality of CNV module 10 and / or steps S12-S16, and descriptions that apply to CNV module 10 and steps S12-S16 may be applicable to step S10 in some implementations, and vice versa. Step S20 may be considered to correspond to the functionality of WSI module 20 and / or steps S22-S28, and descriptions that apply to WSI module 20 and steps S22-S28 may be applicable to step S20 in some implementations, and vice versa. Step S30 may be considered to correspond to the functionality of EHR module 30 and / or steps S32-S36, and descriptions that apply to EHR module 30 and steps S32-S36 may be applicable to step S30 in some implementations, and vice versa. Steps S40-S50 may be considered to correspond to the fusion of integration module 60 and / or step S60, and descriptions that apply to integration module 60 and step S60 may be applicable to step S10 in some implementations, and vice versa. In particular, the first to third ML models in the method of Fig. 10 may be the first to third ML models described with reference to modules 10, 20, and 30. Further, in particular, the histopathology image data may include WSI image data, and / or the medical data may include EHR data. Further, in particular, the weights used in step S40 are learned / trained. Further, in particular, the breast cancer subtype is a PAM50 subtype.

[0085] Step S90 is optional and involves comparing the classification output in step S50 with the true classification (i.e., ground truth classification from the training data) and adjusting the weights used in step S40. The training method for training the weights in step S40 (sometimes referred to as the fourth training process) includes step S90, and as shown, the training method returns to step S10. In this training method, the CNV data, the histopathological image data, and the medical data are training data for the training subjects. The classification and adjustment are repeated / iterated, and thus performed on the training data for multiple training subjects. A training process for training the first to third ML models is also disclosed herein. In the method of FIG. 10, the first to third ML models have already been trained (and their weights are frozen).

[0086] In an example implementation, the method of FIG. 10 is performed using multiple processors or computing devices, particularly using at least one first processor or computing device to apply a second ML model, the first at least one processor or computing device having greater and / or higher processing power than at least one second processor or computing device, which is used to perform one of the following: apply the first ML model, apply a third ML model, calculate the weighted combination, or classify the subject's breast cancer. This is advantageous because applying the second ML model is likely to be more computationally expensive than other method steps. Furthermore, at least one graphical processing unit (GPU) can be used to apply the second ML model, while at least one central processing unit (CPU) can be used for at least one other method step. It will be appreciated that a similar division of processing load can be applied to the system 100 described above. Furthermore, this division of processing load can be applied when training the ML model and weights and / or when classifying the target patient's breast cancer after training is complete.

[0087] For example, referring to FIG. 10 , according to a first embodiment, there is provided a computer-implemented method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to a plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data relating to the subjects, the third ML model comprising information regarding a diagnosis of breast cancer in the subjects, to generate third output logits each corresponding to a plurality of breast cancer subtypes, the third ML model being trained to classify the (training) medical data comprising information regarding a diagnosis of breast cancer (related to the training subjects) to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; A method is disclosed that includes:

[0088] The first output logit can indicate the probability of CNV data corresponding to each of the plurality of breast cancer subtypes, the second output logit can indicate the probability of pathological tissue image data corresponding to each of the plurality of breast cancer subtypes, and the third output logit can indicate the probability of medical data corresponding to each of the plurality of breast cancer subtypes.

[0089] The first ML model can include a neural network.

[0090] The first ML model can include a forward propagation network.

[0091] The first ML model can include a feedforward network that includes a scaled exponential linear unit (SELU) activation function and / or uses alpha-dropout.

[0092] The first ML model can include a Self-Normalizing Network (SNN).

[0093] Applying the first ML model to CNV data associated with the subject can include normalizing the CNV data using z-score normalization (based on the training CNV data).

[0094] The histopathological image data associated with the subject can include a whole scale image (WSI).

[0095] Applying the second ML model to the tissue pathology image data associated with the subject can include dividing the WSI into a plurality of patches, filtering a saturation channel of the WSI using a central blur, applying a threshold to the WSI according to the filtered saturation channel to generate a mask, and using the mask to select patches for input to the second ML model based on tissue content.

[0096] Selecting patches for input to the second ML model based on tissue content using the mask can include selecting patches having tissue content (percent) above a tissue content threshold according to the mask.

[0097] Applying the second ML model to the histopathology image data associated with the subject may include downsampling the selected patches and / or normalizing pixel values ​​of the selected patches.

[0098] The second ML model can include a neural network.

[0099] The second ML model can include a fully connected neural network.

[0100] The second ML model can include a convolutional neural network.

[0101] The medical data about the subject may include any of the following: the subject's age when the subject was first diagnosed with breast cancer; the time between the subject's first diagnosis with breast cancer and tissue sample collection; the disease-free time between the subject's previous diagnosis with breast cancer and the current diagnosis; Fraction of Genome Altered (FGA) measurement; the number of positive lymph nodes by Hematoxylin and Eosin Staining Microscopy; the number of positive lymph nodes by Keratin Immunohistochemistry Staining Method; the number of lymph nodes examined; Mutation Count; tissue specimen weight; nonsynonymous Tumor Mutational Burden (TMB); estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) Metastasis Stage Code Code); AJCC Neoplasm Disease Stage Code; AJCC Tumor Stage Code; AJCC Publication Version Type; Breast Cancer Type; Subject's Disease-Free Status; Breast Cancer Estrogen Receptor Status; Breast Cancer Progesterone Receptor Status; Subject's Ethnicity; HER2 FISH Test Results; Neoplasm Histology Type; Whether the Subject Has Received Neoadjuvant Therapy; Whether the Subject Has Previously Been Diagnosed with Cancer; ICD-10 Classification; ICD-O-3 Tissue Code; ICD-O-3 Site Code; Whether Lymph Node Assessment Was Performed; Menopausal Status; Metastatic Tumor Index; Initial Pathology Biopsy Method Type; Micrometastasis Index; Oncotree Code; Whether an Optimal Cutting Temperature (OCT) Process Was Used on the Tissue Specimen; Surgical Margin Status; Primary Tumor Site; Subject's Ethnicity; Breast Cancer Surgical Procedure;Tissue Source Site; Neoplastic Status; Subject Survival Time from Breast Cancer Diagnosis;

[0102] The type of breast cancer may include an indication of the type of breast cancer from among multiple types including invasive ductal carcinoma of the breast, invasive lobular carcinoma of the breast, mixed ductal and lobular carcinoma of the breast, invasive mixed mucinous carcinoma of the breast, and metaplastic carcinoma of the breast.

[0103] The neoplasm histological type name may include an indication of the type name from among multiple tumor histological types, including invasive ductal carcinoma, invasive lobular carcinoma, and mucinous carcinoma.

[0104] The initial pathology biopsy acquisition method type may include an indication of the method used to diagnose breast cancer from among several methods including core needle biopsy, tumor resection, fine needle aspiration biopsy, and excision biopsy.

[0105] Medical data about the subject may include the following: the subject's age when the subject was first diagnosed with breast cancer; a measurement of the Fraction of Genome Altered (FGA); the number of positive lymph nodes based on hematoxylin and eosin microscopy; the number of positive lymph nodes based on keratin immunohistochemistry staining; nonsynonymous tumor mutational burden (TMB); estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) metastasis stage code; AJCC neoplastic disease lymph node stage code; AJCC neoplastic disease stage code; AJCC tumor stage code; type of breast cancer; the subject's disease-free status; estrogen receptor status of breast cancer; progesterone receptor status of breast cancer; the subject's ethnicity; HER2 FISH test results; neoplasm histology type; whether the subject has received neoadjuvant therapy; whether the subject has previously been diagnosed with cancer; ICD-10 classification; ICD-O-3 histology code; ICD-O-3 site code; menopausal status; Oncotree code; primary tumor site; and subject race information.

[0106] The medical data relating to the subject may include any of the following: Subject's age at first breast cancer diagnosis; estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) metastatic stage code; AJCC Neoplastic Disease Lymph Node Stage Code;AJCC Neoplastic Disease Stage Code;AJCC Tumor Stage Code; Breast cancer type; subject ethnicity; HER2 FISH test result; neoplasm histology; ICD-10 classification; ICD-O-3 histology code; ICD-O-3 site code; menopausal status; Oncotree code; primary tumor site; and subject racial information.

[0107] Medical data about a subject can include electronic health record (EHR) data.

[0108] Applying a 3ML model to medical data about a subject can include converting ordinal and categorical values ​​in the medical data to numeric values ​​(prior to inputting them into the 3ML model).

[0109] Applying the 3ML model to the medical data for the subject can include filling in missing values ​​of the medical data for the subject using a nearest neighbor algorithm based on the medical data for a plurality of other subjects (prior to inputting it into the 3ML model).

[0110] Applying a 3ML model to medical data relating to a subject can include one-hot encoding values ​​of the medical data (prior to inputting them into the 3ML model) and normalizing the encoded values ​​using z-score normalization.

[0111] The 3ML model may include a neural network.

[0112] The computer-implemented method can include obtaining CNV data associated with the subject and / or histopathological imaging data associated with the subject and / or medical data associated with the subject.

[0113] The first ML model may be trained according to a first ML model training process that includes: applying the first ML model to training CNV data associated with a plurality of training subjects to predict a subtype from among a plurality of breast cancer subtypes for each training subject; comparing the predicted subtypes to ground truth data indicating ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the first ML model based on the comparison.

[0114] The first ML model training process can include iteratively applying the first ML model to the training CNV data to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the first ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0115] The computer-implemented method can include, prior to applying the first ML model to CNV data associated with the subject, performing a first ML model training process to train the first ML model to classify the CNV data as corresponding to a subtype among a plurality of breast cancer subtypes.

[0116] The second ML model may be trained according to a second ML model training process that includes: applying the second ML model to training histopathology image data associated with a plurality of training subjects to predict a subtype from among a plurality of breast cancer subtypes for each training subject; comparing the predicted subtypes to ground truth data indicative of ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the second ML model based on the comparison.

[0117] The second ML model training process can include iteratively applying the second ML model to training histopathology image data to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the second ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0118] The second ML model training process can include training the second ML model patch by patch.

[0119] Predicting a subtype from among the plurality of breast cancer subtypes for each training subject can include predicting a subtype for each patch for each training subject, and comparing the predicted subtype to ground truth data can include performing a comparison for each patch.

[0120] The computer-implemented method can include, prior to applying the second ML model to the histopathological image data associated with the subject, performing a second ML model training process to train the second ML model to classify the histopathological image data as corresponding to a subtype among a plurality of breast cancer subtypes.

[0121] The 3ML model may be trained according to a 3ML model training process that includes: applying the 3ML model to training medical data that includes information about breast cancer diagnoses associated with a plurality of training subjects to predict a subtype from among a plurality of breast cancer subtypes for each training subject; comparing the predicted subtype to ground truth data indicative of ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the 3ML model based on the comparison.

[0122] The 3ML model training process can include iteratively applying the 3ML model to training medical data containing information regarding breast cancer diagnoses to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the 3ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0123] The computer-implemented method can include, prior to applying the 3ML model to training medical data associated with the subject, performing a 3ML model training process to train the 3ML model to classify the training medical data, including information regarding a diagnosis of breast cancer, as corresponding to a subtype among a plurality of breast cancer subtypes.

[0124] In any of the first through third ML model training processes, comparing the predicted subtype to ground truth data may include calculating a loss based on the comparison, and adjusting at least one weight of the ML model may include adjusting the at least one weight based on the calculated loss.

[0125] Classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes can include generating a probability distribution using a softmax function based on the weighted combination, and classifying the subject's breast cancer according to the probability distribution.

[0126] The step of calculating a weighted combination of the first output logit, the second output logit, and the third output logit can include using first weights, second weights, and third weights corresponding to the first output logit, the second output logit, and the third output logit, respectively, and the first weights, second weights, and third weights can each include weights corresponding to a plurality of breast cancer subtypes, respectively.

[0127] The first, second, and third weights may have been trained according to a (fourth) training process.

[0128] (4) The training process may include calculating a training weighted combination of first, second, and third training output logits, where the first training output logit is a result of applying a first ML model to training CNV data associated with the training subject, the second training output logit is a result of applying a second ML model to training histopathology image data associated with the training subject, and the third training output logit is a result of applying a third ML model to training medical data associated with the training subject, the training medical data including information regarding a diagnosis of breast cancer in the training subject; classifying the breast cancer of the training subject as a subtype among a plurality of breast cancer subtypes based on the training weighted combination; and comparing the classification with ground truth data indicating a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison.

[0129] The first weight, second weight, and third weight may be trained by repeating the (fourth) training process for a threshold number of iterations for multiple training subjects or until a loss threshold is met.

[0130] (Fourth) The training process may include iteratively computing a training weighted combination of the first, second, and third training output logits for a plurality of training subjects to classify the breast cancer of the training subjects as a subtype, comparing the classification to a ground truth subtype, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison.

[0131] The computer-implemented method may include, prior to calculating the weighted combination, training the first weight, the second weight, and the third weight according to a (fourth) training process.

[0132] The computer-implemented method may include performing first, second, and third ML model training processes to train the first, second, and third ML models before performing the (fourth) training process.

[0133] The subject may be a training subject, and the method may include comparing the classification to ground truth data indicative of a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison.

[0134] The subject may be a target patient, and the computer-implemented method may include outputting a diagnosis of the target patient, the diagnosis including a classification of the subject's breast cancer as a subtype among a plurality of breast cancer subtypes.

[0135] The multiple breast cancer subtypes can include multiple PAM50 breast cancer subtypes.

[0136] Multiple breast cancer subtypes can include Luminal A, Luminal B, Her2-enriched, and Basal-like.

[0137] The computer-implemented method can include using a first at least one processor or computing device to apply a second ML model, where the first at least one processor or computing device has larger and / or higher processing power than a second at least one processor or computing device, where the second at least one processor or computing device is used for any of the following: applying the first ML model, applying a third ML model, calculating a weighted combination, or classifying breast cancer in the subject.

[0138] The computer-implemented method can include using a first at least one processor or computing device for applying the second ML model and a second at least one processor or computing device for calculating the weighted combination and classifying the subject's breast cancer, wherein the first at least one processor or computing device has larger and / or higher processing power than the second at least one processor or computing device.

[0139] The computer-implemented method may include using multiple processors and / or computing devices located in different / separate (geographical) locations and / or in different medical facilities.

[0140] The computer-implemented method may include using a first at least one processor or computing device for applying the second ML model, using a second at least one processor or computing device for applying the first ML model, using a third at least one processor or computing device for applying the third ML model, and using a fourth at least one processor for calculating the weighted combination, wherein the first, second, third, and fourth at least one processor or computing device are located in different / separate (geographical) locations and / or different medical facilities.

[0141] According to a second embodiment, a computer program product, which when executed on a computer, causes the computer to carry out a method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype of the plurality of breast cancer subtypes; and applying a second ML model to histopathological image data associated with the subject to generate second output logits each corresponding to a plurality of breast cancer subtypes, the second ML model being trained to classify the CNV data to correspond to a subtype of the plurality of breast cancer subtypes. applying a 3ML model to medical data related to the subject including information regarding a diagnosis of breast cancer in the subject to generate third output logits, each corresponding to a plurality of breast cancer subtypes, wherein the 3ML model has been trained to classify the medical data including information regarding a diagnosis of breast cancer in the subject to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; and classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination.

[0142] According to a third embodiment, an information processing device includes a memory and a processor coupled to the memory, wherein the processor applies a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes, and applies a second ML model to pathological tissue image data associated with the subject to generate second output logits each corresponding to the plurality of breast cancer subtypes, the second ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes. An information processing device is disclosed that is configured to: apply a 3ML model, the 3ML model being trained to classify pathological tissue data, to medical data related to the subject that includes information regarding a diagnosis of breast cancer in the subject to generate third output logits, each corresponding to a plurality of breast cancer subtypes; the 3ML model being trained to classify the medical data including information regarding the diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculate a weighted combination of the first output logit, the second output logit, and the third output logit; and classify the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination.

[0143] According to a fourth embodiment, a system includes a plurality of computing devices, the plurality of computing devices applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes, and applying a second ML model to histopathological image data associated with the subject to generate second output logits each corresponding to the plurality of breast cancer subtypes, the second ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes. a third ML model trained to classify histopathological data into a first output logit, a second output logit, and a third output logit; and applying the third ML model to medical data related to the subject including information regarding a diagnosis of breast cancer in the subject to generate third output logits each corresponding to a plurality of breast cancer subtypes, the third ML model trained to classify medical data including information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; and classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination.

[0144] The plurality of computing devices can include a first at least one processor or computing device configured to apply a second ML model, the first at least one processor or computing device having larger and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device configured to do any of the following: apply the first ML model, apply a third ML model, calculate a weighted combination, or classify breast cancer in the subject.

[0145] The plurality of computing devices can include using a first at least one processor or computing device configured to apply the second ML model and a second at least one processor or computing device configured to calculate the weighted combination and classify breast cancer in the subject, wherein the first at least one processor or computing device has larger and / or higher processing power than the second at least one processor or computing device.

[0146] The multiple computing devices may include multiple processors and / or computing devices located in different / separate (geographical) locations and / or different medical facilities.

[0147] The plurality of computing devices may include a first at least one processor or computing device configured to apply the second ML model, a second at least one processor or computing device configured to apply the first ML model, a third at least one processor or computing device configured to apply the third ML model, and a fourth at least one processor configured to calculate the weighted combination, wherein the first, second, third, and fourth at least one processor or computing device are located in different / separate (geographical) locations and / or different medical facilities.

[0148] Example of operation Here, we describe an example of how multimodal prediction of PAM50 cancer subtypes works for a subject with sample ID “tcga-a2-a0ev-01” from the dataset TCGA-BRCA.

[0149] The CNV input contains approximately 23k genes. This raw input is z-scored using the mean and standard deviation of the training dataset to obtain the corresponding normalized gene values. A sample of eight CNV values ​​and their normalized CNV values ​​are shown below.

number

[0150] The 1ML model (after training) is applied to the subject's preprocessed CNV data to generate output logits for this modality (1st output logits). These 1st output logits are shown in Figure 11. For interpretability, the 1ML model uses integrated gradients to generate attributes for each input gene. The absolute values ​​of these attribute weights are taken to rank genes according to their contribution to the final output. The top 20 contributing genes for the subject are shown in Figure 12.

[0151] The output logits are serialized and used for multimodal prediction.

[0152] The raw input WSI for subject “tcga-a2-a0ev-01” is the WSI image shown in Figure 13 (in black and white, although the image may actually contain color).

[0153] Since large regions of the WSI are usually the background, a segmentation algorithm is applied to separate the tissue from the background. In line with the previous description, the mask is generated by: Extract 256x256 patches from WSI. Convert the patch from RGB color space to HSV color space. Apply a median blur to the saturation channel using a kernel size of 7. Threshold the saturation channel using a binarization threshold of 20 to generate a binarization mask.

[0154] This produces a binarized mask that distinguishes foreground from background, as shown in Figure 14. Using this mask, the tissue content is calculated for each patch, and patches that do not contain at least 5% tissue are filtered out. This process is visually illustrated in Figure 15, where the solid lines indicate accepted patches that contain sufficient tissue, and the dashed / dotted lines indicate rejected patches that are mostly background.

[0155] Accepted patches are downsampled by 20x, resized to 342x342 using bilinear interpolation, and center-cropped to a 299x299 square. The pixel values ​​are then rescaled to the range [0.0, 1.0] and normalized using mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] from the training dataset. The accepted patches are then passed to the second ML model (after training). An output is generated for each patch. These outputs are aggregated using the mean operator to form the entire set of subject output logits (second output logits). The subject second output logits are shown in Figure 16.

[0156] For explainability, the 2ML model uses the integrated gradient to calculate the attributes for each pixel in the patch accepted by the WSI. The sum of the attributes across image channels is taken and the absolute value of the attribute is calculated. This produces a heatmap of the top 25% contributing pixels overlaid on a grayscale version of the WSI, as shown in Figure 17 (shown here in black and white, but color could also be used to illustrate the heatmap).

[0157] The second output logits are serialized and used for multimodal prediction.

[0158] The raw EHR data for subject "tcga-a2-a0ev-01" is as follows:

number

[0159] This data is first preprocessed as follows, where the preprocessing includes: ordinal and categorical values ​​are converted to numeric values; missing values ​​are imputed using k-NN; categorical values ​​are one-hot coded; and the data is z-scored based on the statistics of the training dataset. Through one-hot coding, the subject's input data is expanded to 271 features. A sample of the transformed preprocessed input data for the subject is as follows:

number

[0160] The resulting numeric vector is then used in a 3ML model (after training) to generate the 3 output logit, which is shown in Figure 18. For explainability, the 3ML model uses integrated gradients. An attribute for each dimension of the input is calculated. Because one-hot coding allows a single raw feature to contribute to multiple input values, the contribution of each feature is summed to obtain the overall feature contribution. The absolute value of these contributions is taken, and the features are ranked based on their importance for explainability. The top contributing features for the subject are shown in Figure 19.

[0161] The third output logits are serialized and sent to the integration module to make multimodal predictions.

[0162] The first, second, and third output logits for subject "tcga-a2-a0ev-01" are shown in Figure 20. For each modality and each class, the weights learned during the training process are stored. The learned weights are shown in Figure 21.

[0163] The outputs of the single modalities are fused according to the equations given above:

number

[0164] result The methodology disclosed herein was trained and tested as described above, and the results are described below. Specifically, WSI, CNV, and EHR data from the TCGA-BRCA dataset were used for training and testing. As described above, instances with missing modality or no PAM50 classification label were filtered out, resulting in 977 cases. Furthermore, as substantially described above, the subtype imbalance present in the training data was mitigated using one of three strategies: oversampling, stratified sampling, and loss weighting. In oversampling, each class is resampled to have the same number of samples as the majority class. In stratified sampling, data is sampled from each class with a probability inversely proportional to its size. In loss weighting, the loss for each class is rescaled by a factor inversely proportional to its size in the dataset. Because the dataset was imbalanced, both accuracy and the area under the receiver operating characteristic curve (AUROC) of the sensitivity-specificity relationship curve were used to evaluate the model. For AUROC, AUC per class was calculated and then averaged across classes. Model evaluation used 10-fold cross-validation, and the average performance across rounds was reported.

[0165] The optimal models after hyperparameter search were as follows: The first ML model used an SNN backbone with 8192, 4096, and 2048 channel layers and a classification head with 512 and 128 channel hidden layers. Oversampling balancing was used for the CNV data, and the inputs were z-scored based on the training subset. The second ML model used an Inceptionv3 backbone with 512 and 128 channel hidden layer classification heads with ReLU activation, instance normalization, 0.5 dropout, weighted loss balancing, and majority voting pooling. The third ML model used a vanilla MLP with 128 and 64 channel hidden layers, ReLU activation, and instance normalization. The Adam optimizer was used with default parameters for all single-modality experiments in breast cancer subtyping.

[0166] Evaluation of the specific implementation mentioned above revealed a performance of 76.88% accuracy and 0.8976 AUROC.

[0167] Here we disclose the following: · RNA-independent PAM50 breast cancer subtyping system for multimodal prediction, improving performance by integrating CNV, EHR, and WSI. A decentralized, distributed implementation of the PAM50 breast cancer subtyping system that can operate on locally distributed data and uses minimal communication. Distributed modular implementation that can efficiently utilize heterogeneous CPU / GPU resources based on modality complexity. An efficient multimodal late fusion strategy that does not require centralized collection of data and can operate in a decentralized manner using only the outputs of other modules. A WSI module capable of operating at low magnification WSI while offering competitive single-modality and multimodal performance. Furthermore, it can be used as a standalone single-modality module or as part of a multimodal system. The module can provide a heat map illustrating the input regions that primarily contributed to its decisions. The CNV, EHR, and WSI modules have implementations that can provide explainability for each modality, helping users understand the system's predictions. A CNV module with a simplified pipeline that can operate as a standalone module or as part of a larger medical application system. Input attributes can be provided to indicate which genes contribute most to the decision. An EHR module that can use various types of clinical data, including numeric, ordinal, and categorical features. It can operate as a standalone single-modality PAM50 classification system using the EHR, or as a component of a larger healthcare system. The module can provide attributes as input and rank features based on their contribution to the decision. PAM50 WSI segmentation pipeline that can reduce computation by segmenting tissue from background and filtering out WSI patches that are mostly background. Two balancing strategies, oversampling and loss weighting, to improve training performance for unbalanced PAM50 classes. A decentralized implementation that can be deployed on heterogeneous CPU / GPU hardware and operates on locally distributed modalities without the need for centralized data transfer. Support for heterogeneous hardware allows for efficient use of resources: computationally expensive modules such as digital pathology processing can benefit from GPU acceleration, while less demanding modules can run on the CPU. It is an efficient late fusion strategy that relies solely on model output, making it suitable for decentralized multi-omics integration. Furthermore, it has competitive PAM50 multi-omics performance without relying on RNA data.

[0168] An advantage of the disclosed methodology is that multi-omics data can reside in different systems and therefore be governed by different laws and privacy provisions that make it impossible to transfer them to a central system. Furthermore, because different data modalities have different computational constraints, to achieve efficient utilization of computational resources, the disclosed methodology can operate in a heterogeneous hardware environment that can utilize GPUs for computationally expensive modalities and CPUs for less demanding workloads.

[0169] Another advantage of the disclosed methodology is the transparency of the classification system in a clinical setting: an explainable system that can provide the rationale for its decisions is important for clinicians to trust and have confidence in the system's operation.

[0170] Single-modality modules / method steps are self-contained and can be used as standalone PAM50 systems based on a single modality, or integrated as part of a healthcare application system. A late fusion strategy can be used for other cross-domain use cases where downstream applications can benefit from the fusion of multimodal data.

[0171] Healthcare applications are generally multimodal problems that benefit from the integration of diverse data sources. Due to the exponential growth in the amount of data generated by various healthcare-related modalities, there is a dire need for multimodal machine learning methods that can organize and discover useful patterns from these data in a scalable and flexible manner. This application discloses a scalable, loosely coupled multimodal framework that integrates data from diverse modalities, including genomics, clinical records, and histopathology, to perform breast cancer subtyping. The disclosed methodology allows for extension to additional modalities (e.g., RNA-seq) and scaling up and down with minimal overhead without the need to retrain existing modalities.

[0172] Classifying breast cancer into subtypes is a multifaceted exercise that relies on diverse sources of information, including molecular data such as copy number variation (CNV) and ribonucleic acid sequencing (RNA-seq), histopathological images, and electronic health records (EHRs). This has become increasingly challenging for physicians and researchers, especially with the explosion in the amount of information provided by different relevant modalities. This necessitates machine learning (ML) models that can help organize this flow of information, detect patterns, and draw useful conclusions from the data.

[0173] A challenge facing multimodal prediction is that in clinical practice, the number and type of modalities available to a multimodal method are partially available and variable. Different datasets and different downstream tasks may have different numbers and types of modalities available at any given time. Therefore, a scalable and flexible multimodal method is needed that can extend to a large number of modalities and adapt efficiently and with minimal overhead to changes in the number and type of modalities. The scalability and flexibility of multimodal methods are critical in clinical practice. Changes in medical data and availability of modalities may change over time. The disclosed methodology can efficiently scale to the addition or removal of modalities with minimal overhead.

[0174] In the above embodiment, the PAM50 classification is used as the breast cancer subtype, specifically 4 is used and "normal-like" is omitted, but "normal-like" may be included, thus 5 subtypes may be considered. Other classifications / subtypes may also be used.

[0175] 23 is a block diagram of an information processing appliance 900 or computing device 900, such as a data storage server, that may be used to embody the present invention, to perform some or all of the operations of the methods embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiments. The computing device 900 may be used to implement any of the method steps described above, such as any of steps S10-S90 and / or S12-S60, and / or the processing of any of modules 10, 20, 30, and 60.

[0176] Computing device 900 includes a processor 993 and memory 994. Optionally, the computing device also includes a network interface 997 for communicating with other such computing devices, such as other computing devices of embodiments of the present invention. Optionally, the computing device also includes one or more input mechanisms 996, such as a keyboard and mouse, and a display unit 995, such as one or more monitors. These elements may facilitate user interaction. The components may be connected to each other via a bus 992.

[0177] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause it to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing any of the steps disclosed herein above, such as any of steps S10-S90 and / or S12-S60, and / or the processing of any of modules 10, 20, 30, and 60. The term "computer-readable storage medium" may include any medium capable of storing, encoding, or retaining a set of instructions for execution by a machine and causing the machine to perform any one or more of the method steps of the present disclosure; therefore, the term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media. For example, but not limited to, the term "computer-readable storage medium" may include solid-state memory, optical media, and magnetic media. By way of example, and without limitation, such computer-readable media may include non-transitory computer-readable media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), and the like.

[0178] The processor 993 is configured to control a computing device and perform processing operations, such as executing computer program code stored in the memory 994, to perform, for example, any of steps S10-S90 and / or S12-S60 and / or any of the processes of modules 10, 20, 30, and 60 described above. The memory 994 stores data read and written by the processor 993, and may store CNV data, histopathological image data, medical data, WSI data, EHR data, ML model weights, learnable weights, output logits, training data, test data, output classifications, input data, and / or other data described above, and / or programs for performing any of the method steps or processes described above. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. The processor may include a processor implementing a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or other instruction set or processor implementing a combination of instruction sets. The processor may also include one or more special-purpose processing devices, such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), a network processor, or the like. In one or more embodiments, the processor is configured to execute instructions to perform the method steps and / or operations described herein, e.g., any of steps S10-S90 and / or S12-S60, and / or the processing of any of the modules 10, 20, 30, and 60 described above. The processor 993 may be considered to include any of the modules 10, 20, 30, and 60 described above. Any operations described as being performed by a module may be implemented as a method by a computer, e.g., the processor 993. The processor 993 may be considered to include multiple processors / computing devices.

[0179] The display unit 995 may display representations of data stored by the computing device, such as a representation of the first ML model and / or a representation of the second ML model and / or a representation of the third ML model and / or a representation of a breast cancer classification and / or input data and / or GUI windows and / or interactive representations that allow a user to interact with the device 900, for example by drag-and-drop or selection interactions, and / or any other output described above, and may display cursors and dialog boxes and screens that allow interaction between a user and programs and data stored on the computing device. The input mechanism 996 may enable a user to input data and instructions into the computing device, such as allowing a user to enter any of the user inputs described above.

[0180] The network interface (network I / F) 997 can be connected to a network such as the Internet and can be connected to other such computing devices via the network. The network I / F 997 can control data input and output from other devices via the network. For example, computing devices configured similarly to device 900 can be configured to operate remotely together to implement first, second, and third ML models, generate a weighted combination, and ultimately generate a breast cancer classification. Other peripheral devices, such as a microphone, speakers, printer, power supply unit, fan, case, scanner, tracker ball, etc., may also be included in the computing device.

[0181] A method embodying the present invention may be performed on a computing device / appliance 900 such as that shown in FIG. 23. Such a computing device need not have all of the components shown in FIG. 23, but may consist of a subset of these components. For example, the device 900 may include a processor 993 and a memory 994 coupled to the processor 993. Alternatively, the device 900 may include a processor 993, a memory 994 coupled to the processor 993, and a display 995. A method embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may also be a data storage device that stores at least a portion of the data.

[0182] Methods embodying the present invention may be performed by multiple computing devices operating in cooperation with one another. One or more of the multiple computing devices may be data storage servers that store at least a portion of the data. Device 900 may be a computing device among a plurality of similarly configured computing devices, the computing devices together configured to perform any of the method steps described herein, e.g., any of steps S10-S90 and / or S12-S60, and / or any of modules 10, 20, 30, and 60 described above. For example, at least one computing device may train and / or implement a first ML model, at least one computing device may train and / or implement a second ML model, at least one computing device may train and / or implement a third ML model, and at least one computing device may train and / or implement weights for computing weighted combinations and classifying breast cancer. The processors and / or computing devices may be heterogeneous, e.g., in terms of processing power.

[0183] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can also be implemented as a computer program or computer program product, i.e., a computer program embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0184] A computer program may be in the form of a stand-alone program, a computer program portion, or multiple computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0185] The method steps of the present invention may be performed by one or more programmable processors executing computer programs to perform the functions of the present invention by operating on input data and generating output. The apparatus of the present invention may be implemented as programmed hardware or as dedicated logic circuitry including Field Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs), etc.

[0186] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memories that store instructions and data.

[0187] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments.

[0188] The present disclosure extends to the following statements:

[0189] (S1) 1. A computer-implemented method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the (training) CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify (train) the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data relating to the subjects, the third ML model comprising information regarding a diagnosis of breast cancer in the subjects, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify (training) medical data comprising information regarding a diagnosis of breast cancer (associated with the training subjects) to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; A method comprising:

[0190] (S2) The computer-implemented method described in statement S1, wherein the first output logit indicates a probability of CNV data corresponding to each of the plurality of breast cancer subtypes, the second output logit indicates a probability of pathological tissue image data corresponding to each of the plurality of breast cancer subtypes, and the third output logit indicates a probability of medical data corresponding to each of the plurality of breast cancer subtypes.

[0191] (S3) The computer-implemented method of S1, wherein the first ML model comprises a neural network.

[0192] (S4) The computer-implemented method of any of S1-S3, wherein the first ML model comprises a feedforward propagation network.

[0193] (S5) 5. The computer-implemented method of any of S1-S4, wherein the first ML model includes a feedforward network that includes a scaled exponential linear unit (SELU) activation function and / or uses alpha-dropout. (S6) The computer-implemented method of any of S1-S5, wherein the first ML model comprises a self-normalizing network (SNN).

[0194] (S7) The computer-implemented method of any of S1 to S6, wherein applying the first ML model to CNV data associated with the subject comprises normalizing the CNV data using z-score normalization (based on training CNV data).

[0195] (S8) The computer-implemented method of any of S1 to S7, wherein the histopathological image data related to the subject includes a whole scale image (WSI).

[0196] (S9) The computer-implemented method of S8, wherein applying the second ML model to the pathological tissue image data associated with the subject comprises dividing the WSI into a plurality of patches, filtering a saturation channel of the WSI using a central blur, applying a threshold to the WSI according to the filtered saturation channel to generate a mask, and using the mask to select patches for input to the second ML model based on tissue content.

[0197] (S10) The computer-implemented method of S9, wherein selecting patches for input to the second ML model based on tissue content using the mask includes selecting patches having tissue content (percent) above a tissue content threshold according to the mask.

[0198] (S11) The computer-implemented method of S9 or S10, wherein applying the second ML model to the tissue pathology image data associated with the subject comprises downsampling the selected patches and / or normalizing pixel values ​​of the selected patches.

[0199] (S12) The computer-implemented method of any of S1-S11, wherein the second ML model comprises a neural network.

[0200] (S13) The computer-implemented method of any of S1-S12, wherein the second ML model comprises a fully connected neural network.

[0201] (S14) The computer-implemented method of any of S1-S13, wherein the second ML model comprises a convolutional neural network.

[0202] (S15) The medical data about the subject may include any of the following: the subject's age when the subject was first diagnosed with breast cancer; the time between the subject's first diagnosis with breast cancer and tissue sample collection; the disease-free time between the subject's previous diagnosis with breast cancer and the current diagnosis; Fraction of Genome Altered (FGA) measurement; the number of positive lymph nodes by Hematoxylin and Eosin Staining Microscopy; the number of positive lymph nodes by Keratin Immunohistochemistry Staining Method; the number of lymph nodes examined; Mutation Count; tissue specimen weight; nonsynonymous Tumor Mutational Burden (TMB); estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) Metastasis Stage Code Code); AJCC Neoplasm Disease Stage Code; AJCC Tumor Stage Code; AJCC Publication Version Type; Breast Cancer Type; Subject's Disease-Free Status; Breast Cancer Estrogen Receptor Status; Breast Cancer Progesterone Receptor Status; Subject's Ethnicity; HER2 FISH Test Results; Neoplasm Histology Type; Whether the Subject Has Received Neoadjuvant Therapy; Whether the Subject Has Previously Been Diagnosed with Cancer; ICD-10 Classification; ICD-O-3 Tissue Code; ICD-O-3 Site Code; Whether Lymph Node Assessment Was Performed; Menopausal Status; Metastatic Tumor Index; Initial Pathology Biopsy Method Type; Micrometastasis Index; Oncotree Code; Whether an Optimal Cutting Temperature (OCT) Process Was Used on the Tissue Specimen; Surgical Margin Status; Primary Tumor Site; Subject's Ethnicity; Breast Cancer Surgical Procedure;Tissue Source Site; Neoplastic Status; Subject Survival Time from Breast Cancer Diagnosis, a computer-implemented method according to any one of S1 to S14;

[0203] (S16) The computer-implemented method of S15, wherein the type of breast cancer includes an indication of the type of breast cancer from a plurality of types including invasive ductal carcinoma of the breast, invasive lobular carcinoma of the breast, mixed ductal and lobular carcinoma of the breast, invasive mixed mucinous carcinoma of the breast, and metaplastic breast carcinoma.

[0204] (S17) The computer-implemented method of S15 or S16, wherein the neoplasm histological type name includes an indication of a type name from among a plurality of tumor histological type names including invasive ductal carcinoma, invasive lobular carcinoma, and mucinous carcinoma.

[0205] (S18) The computer-implemented method of any of S15 to S17, wherein the initial pathology diagnostic biopsy specimen acquisition method type includes an indication of a method to be used to diagnose breast cancer from among a plurality of methods including core needle biopsy, tumor resection, fine needle aspiration biopsy, and excision biopsy.

[0206] (S19) The medical data for the subject may include the following: the subject's age when the subject was first diagnosed with breast cancer; a measurement of the Fraction of Genome Altered (FGA); the number of positive lymph nodes based on hematoxylin and eosin microscopy; the number of positive lymph nodes based on keratin immunohistochemistry staining; nonsynonymous tumor mutational burden (TMB); estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) metastasis stage code; AJCC neoplastic disease lymph node stage code; AJCC neoplastic disease stage code; AJCC tumor stage code; type of breast cancer; the subject's disease-free status; estrogen receptor status of breast cancer; progesterone receptor status of breast cancer; the subject's ethnicity; and HER2. FISH test results; neoplasm histology type; whether the subject has received neoadjuvant therapy; whether the subject has previously been diagnosed with cancer; ICD-10 classification; ICD-O-3 tissue code; ICD-O-3 site code; menopausal status; Oncotree code; primary tumor site; and subject's racial information, a computer-implemented method described in any of S1 to S14.

[0207] (S20) The medical data relating to the subject includes any of the following: Subject's age at first breast cancer diagnosis; estrogen receptor positivity rate; progesterone receptor positivity rate; HER2 IHC test results; American Joint Committee on Cancer (AJCC) metastatic stage code; AJCC Neoplastic Disease Lymph Node Stage Code;AJCC Neoplastic Disease Stage Code;AJCC Tumor Stage Code; type of breast cancer; ethnicity of the subject; HER2 FISH test results; neoplasm histology name; ICD-10 classification; ICD-O-3 histology code; ICD-O-3 site code; menopausal status; Oncotree code; primary tumor site; and racial information of the subject, a computer-implemented method described in any of S1 to S14.

[0208] (S21) The computer-implemented method of any of S1 to S20, wherein the medical data associated with the subject includes electronic health record (EHR) data.

[0209] (S22) The computer-implemented method of any of S1-S21, wherein applying the third ML model to medical data relating to the subject includes converting ordinal and categorical values ​​in the medical data to numeric values ​​(prior to input into the third ML model).

[0210] (S23) The computer-implemented method of any of S1-S22, wherein applying the 3ML model to the medical data about the subject includes (prior to inputting to the 3ML model) inputting missing values ​​of the medical data about the subject using a nearest neighbor algorithm based on medical data about a plurality of other subjects.

[0211] (S24) The computer-implemented method of any of S1-S23, wherein applying the 3ML model to medical data relating to the subject comprises one-hot encoding values ​​in the medical data (prior to inputting them into the 3ML model) and normalizing the encoded values ​​using z-score normalization.

[0212] (S25) The computer-implemented method of any of S1-S24, wherein the third ML model comprises a neural network.

[0213] (S26) The computer-implemented method of any of S1 to S25, comprising obtaining CNV data related to the subject and / or pathological tissue image data related to the subject and / or medical data related to the subject.

[0214] (S27) The first ML model is trained according to a first ML model training process, which includes: applying the first ML model to training CNV data related to a plurality of training subjects to predict a subtype from among the plurality of breast cancer subtypes for each training subject; comparing the predicted subtype with ground truth data indicating ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the first ML model based on the comparison.

[0215] (S28) The computer-implemented method of S27, wherein the first ML model training process includes iteratively applying the first ML model to training CNV data to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the first ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0216] (S29) The computer-implemented method of S27 or S28, comprising, before applying the first ML model to CNV data associated with the subject, performing the first ML model training process to train the first ML model to classify CNV data as corresponding to a subtype among the plurality of breast cancer subtypes.

[0217] (S30) The second ML model is trained according to a second ML model training process, which includes: applying the second ML model to training histopathology image data associated with a plurality of training subjects to predict a subtype from among the plurality of breast cancer subtypes for each training subject; comparing the predicted subtype with ground truth data indicating ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the second ML model based on the comparison.

[0218] (S31) The computer-implemented method of S30, wherein the second ML model training process includes iteratively applying the second ML model to training histopathology image data to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the second ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0219] (S32) The computer-implemented method of S30 or S31, wherein the second ML model training process includes training the second ML model on a patch-by-patch basis.

[0220] (S33) 3. The computer-implemented method of claim 2, wherein predicting a subtype from among the plurality of breast cancer subtypes for each training subject comprises predicting a subtype for each patch for each training subject, and comparing the predicted subtype to the ground truth data comprises performing a comparison for each patch.

[0221] (S34) The computer-implemented method of any of S30 to S33, comprising, prior to applying the second ML model to pathological tissue image data associated with the subject, performing the second ML model training process to train the second ML model to classify the pathological tissue image data as corresponding to a subtype among the plurality of breast cancer subtypes.

[0222] (S35) The 3ML model is trained according to a 3ML model training process, which includes: applying the 3ML model to training medical data associated with a plurality of training subjects, the training medical data including information related to a diagnosis of breast cancer, to predict a subtype from among the plurality of breast cancer subtypes for each training subject; comparing the predicted subtype with ground truth data indicating ground truth subtypes each corresponding to the training subjects; and adjusting at least one weight of the 3ML model based on the comparison.

[0223] (S36) The computer-implemented method of S35, wherein the 3ML model training process includes iteratively applying the 3ML model to training medical data containing information regarding breast cancer diagnoses to predict subtypes for test subjects, comparing the predicted subtypes to ground truth subtypes, and adjusting at least one weight of the 3ML model based on the comparison until a threshold number of iterations or a loss threshold is met.

[0224] (S37) The computer-implemented method of S35 or S36, comprising, prior to applying the 3ML model to training medical data related to the subject, performing the 3ML model training process to train the 3ML model to classify training medical data including information regarding a diagnosis of breast cancer as corresponding to a subtype among the plurality of breast cancer subtypes.

[0225] (S38) The computer-implemented method of any of S27 to S37, wherein in any of the first to third ML model training processes, comparing the predicted subtype to the ground truth data includes calculating a loss based on the comparison, and adjusting at least one weight of the ML model includes adjusting at least one weight based on the calculated loss.

[0226] (S39) The computer-implemented method of any of S1 to S38, wherein the step of classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes comprises generating a probability distribution using a softmax function based on the weighted combination, and classifying the subject's breast cancer according to the probability distribution.

[0227] (S40) The computer-implemented method of any of S1 to S39, wherein calculating a weighted combination of the first output logit, the second output logit, and the third output logit comprises using first weights, second weights, and third weights corresponding to the first output logit, the second output logit, and the third output logit, respectively, wherein the first weights, the second weights, and the third weights each comprise weights corresponding to the plurality of breast cancer subtypes, respectively.

[0228] (S41) The computer-implemented method of S40, wherein the first, second and third weights have been trained according to a (fourth) training process.

[0229] (S42) The (fourth) training process: calculating a training weighted combination of first, second, and third training output logits, where the first training output logits are results of applying the first ML model to training CNV data associated with training subjects, the second training output logits are results of applying the second ML model to training histopathology image data associated with the training subjects, and the third training output logits are results of applying the third ML model to training medical data associated with the training subjects, the training medical data including information regarding a diagnosis of breast cancer in the training subjects; classifying the breast cancer of the training subjects as a subtype among a plurality of breast cancer subtypes based on the training weighted combinations; comparing the classification to ground truth data indicative of a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison; The computer-implemented method of S41, comprising:

[0230] (S43) The computer-implemented method of S41 or S42, wherein the first weight, second weight, and third weight are trained by repeating the (fourth) training process for a threshold number of iterations for a plurality of training subjects or until a loss threshold is met.

[0231] (S44) The computer-implemented method of any of S41 to S43, wherein the (fourth) training process includes iteratively computing a training weighted combination of the first, second, and third training output logits for a plurality of training subjects to classify the breast cancer of the training subjects as a subtype, comparing the classification to a ground truth subtype, and adjusting at least one of the first weight, second weight, and third weight based on the comparison.

[0232] (S45) The computer-implemented method of any of S41 to S43, comprising training the first weight, second weight, and third weight according to the (fourth) training process before calculating the weighted combination.

[0233] (S46) The computer-implemented method of S44, comprising performing the first, second, and third ML model training processes to train the first, second, and third ML models before performing the (fourth) training process.

[0234] (S47) The subject is a training subject, and the method includes comparing the classification with ground truth data indicating a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison.

[0235] (S48) The computer-implemented method of any of S1 to S46, wherein the subject is a target patient, and the computer-implemented method includes a step of outputting a diagnosis of the target patient, the diagnosis including a classification of the subject's breast cancer as a subtype among the plurality of breast cancer subtypes.

[0236] (S49) The computer-implemented method of any of S1 to S48, wherein the plurality of breast cancer subtypes comprises a plurality of PAM50 breast cancer subtypes.

[0237] (S50) The computer-implemented method of any of S1 to S49, wherein the plurality of breast cancer subtypes include Luminal A, Luminal B, Her2-enriched, and Basal-like.

[0238] (S51) The computer-implemented method of any of S1 to S50, comprising using a first at least one processor or computing device to apply the second ML model, the first at least one processor or computing device having larger and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device being used for any of the following: applying the first ML model, applying the third ML model, calculating the weighted combination, or classifying breast cancer in the subject.

[0239] (S52) 5. The computer-implemented method of any of S1-S50, comprising using a first at least one processor or computing device for applying the second ML model and a second at least one processor or computing device for calculating the weighted combination and classifying the subject's breast cancer, wherein the first at least one processor or computing device has larger and / or higher processing power than the second at least one processor or computing device.

[0240] (S53) The computer-implemented method of any of S1-S52, comprising using multiple processors and / or computing devices located in different / separate (geographical) locations and / or different medical facilities.

[0241] (S54) 10. The computer-implemented method of any of S1-S50, comprising using a first at least one processor or computing device for applying the second ML model, using a second at least one processor or computing device for applying the first ML model, using a third at least one processor or computing device for applying the third ML model, and using a fourth at least one processor for calculating the weighted combination, wherein the first, second, third, and fourth at least one processors or computing devices are located in different / separate (geographical) locations and / or different medical facilities.

[0242] (S55) A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; a computer program comprising:

[0243] (S56) An information processing device comprising: a memory; and a processor coupled to the memory, the processor comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. An information processing device configured as follows.

[0244] (S57) 1. A system including a plurality of computing devices, the plurality of computing devices comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. The system is configured as follows:

[0245] (S58) The plurality of computing devices includes a first at least one processor or computing device configured to apply the second ML model, the first at least one processor or computing device having greater and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device being configured to: applying the first ML model; applying the third ML model; calculating said weighted combination; classifying the subject's breast cancer; The system of S57, configured to perform any of the following:

[0246] (S59) 10. The system of claim 9, wherein the plurality of computing devices includes a first at least one processor or computing device configured to apply the second ML model and a second at least one processor or computing device configured to calculate the weighted combination and classify breast cancer in the subject, the first at least one processor or computing device having greater and / or higher processing power than the second at least one processor or computing device.

[0247] (S60) The system of any of S57 to S59, wherein the plurality of computing devices comprises a plurality of processors and / or computing devices located in different / separate (geographical) locations and / or different medical facilities.

[0248] (S61) The system of S57, wherein the plurality of computing devices includes a first at least one processor or computing device configured to apply the second ML model, a second at least one processor or computing device configured to apply the first ML model, a third at least one processor or computing device configured to apply the third ML model, and a fourth at least one processor configured to calculate the weighted combination, wherein the first, second, third, and fourth at least one processor or computing device are located in different / separate (geographical) locations and / or different medical facilities.

[0249] In addition to the above-described embodiments, the following supplementary notes are also disclosed. (Supplementary Note 1) A computer-implemented method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; 20. A computer-implemented method comprising: (Appendix 2) The computer-implemented method of Appendix 1, wherein the step of classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes includes generating a probability distribution using a softmax function based on the weighted combination, and classifying the subject's breast cancer according to the probability distribution. (Supplementary Note 3) The computer-implemented method of Supplementary Note 1 or 2, wherein the step of calculating a weighted combination of the first output logit, the second output logit, and the third output logit includes using first weights, second weights, and third weights corresponding to the first output logit, the second output logit, and the third output logit, respectively, and the first weights, the second weights, and the third weights each include weights corresponding to the plurality of breast cancer subtypes, respectively. (Appendix 4) The computer-implemented method of Appendix 3, wherein the first, second, and third weights have been trained according to a training process. (Appendix 5) The training process includes: calculating a training weighted combination of first, second, and third training output logits, where the first training output logits are results of applying the first ML model to training CNV data associated with training subjects, the second training output logits are results of applying the second ML model to training histopathology image data associated with the training subjects, and the third training output logits are results of applying the third ML model to training medical data associated with the training subjects, the training medical data including information regarding a diagnosis of breast cancer in the training subjects; classifying the breast cancer of the training subjects as a subtype among the plurality of breast cancer subtypes based on the training weighted combinations; comparing the classification to ground truth data indicative of a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison; 5. The computer-implemented method of claim 4, comprising: (Supplementary Note 6) The computer-implemented method of Supplementary Note 3, wherein the subject is a training subject, the method comprising: comparing the classification with ground truth data indicating a ground truth subtype corresponding to the training subject; and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison. (Appendix 7) The computer-implemented method of Appendix 1 or 2, wherein the subject is a target patient, and the computer-implemented method includes a step of outputting a diagnosis of the target patient, the diagnosis including a classification of the subject's breast cancer as a subtype among the plurality of breast cancer subtypes. (Appendix 8) The computer-implemented method of Appendices 1 or 2, wherein the plurality of breast cancer subtypes comprises a plurality of PAM50 breast cancer subtypes. (Supplementary Note 9) Using a first at least one processor or computing device to apply the second ML model, the first at least one processor or computing device having greater and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device being: applying the first ML model; applying the third ML model; calculating the weighted combination; classifying the subject's breast cancer; 3. The computer-implemented method of claim 1, further comprising the steps of: (Supplementary Note 10) The computer-implemented method of any one of Supplementary Notes 1 to 2, including using multiple processors and / or computing devices located in different geographic locations and / or different medical facilities. (Supplementary Note 11) A computer program that, when executed on a computer, causes the computer to perform a method, the method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; a computer program comprising: (Supplementary Note 12) An information processing device, comprising: a memory; and a processor coupled to the memory, the processor: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. An information processing device configured as follows. (Supplementary Note 13) A system including a plurality of computing devices, wherein the plurality of computing devices: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. The system is configured as follows: (Supplementary Note 14) The plurality of computing devices includes a first at least one processor or computing device configured to apply the second ML model, the first at least one processor or computing device having greater and / or higher processing power than a second at least one processor or computing device, and the second at least one processor or computing device being configured to: applying the first ML model; applying the third ML model; calculating the weighted combination; classifying the subject's breast cancer; 14. The system of claim 13, configured to: (Appendix 15) The system of Appendix 13 or 14, wherein the plurality of computing devices includes a plurality of processors and / or computing devices located in different geographic locations and / or different medical facilities. [Explanation of symbols]

[0250] 10 CNV modules 20 WSI modules 30 EHR modules 60 Integrated Modules

Claims

1. 1. A computer-implemented method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; 20. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes comprises generating a probability distribution using a softmax function based on the weighted combination, and classifying the subject's breast cancer according to the probability distribution.

3. 3. The computer-implemented method of claim 1, wherein calculating a weighted combination of the first output logit, the second output logit, and the third output logit comprises using first weights, second weights, and third weights corresponding to the first output logit, the second output logit, and the third output logit, respectively, wherein the first weights, the second weights, and the third weights each comprise weights corresponding to the plurality of breast cancer subtypes, respectively.

4. The computer-implemented method of claim 3 , wherein the first, second, and third weights are trained according to a training process.

5. The training process includes: calculating a training weighted combination of first, second, and third training output logits, wherein the first training output logits are results of applying the first ML model to training CNV data associated with training subjects, the second training output logits are results of applying the second ML model to training histopathology image data associated with the training subjects, and the third training output logits are results of applying the third ML model to training medical data associated with the training subjects, the training medical data including information regarding a diagnosis of breast cancer in the training subjects; classifying the breast cancer of the training subjects as a subtype among the plurality of breast cancer subtypes based on the training weighted combinations; comparing the classification to ground truth data indicative of a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison; The computer-implemented method of claim 4 , comprising:

6. 4. The computer-implemented method of claim 3, wherein the subject is a training subject, and the method comprises comparing the classification to ground truth data indicative of a ground truth subtype corresponding to the training subject, and adjusting at least one of the first weight, the second weight, and the third weight based on the comparison.

7. 3. The computer-implemented method of claim 1, wherein the subject is a target patient, and the computer-implemented method comprises outputting a diagnosis of the target patient, the diagnosis comprising a classification of the subject's breast cancer as a subtype among the plurality of breast cancer subtypes.

8. 3. The computer-implemented method of claim 1, wherein the plurality of breast cancer subtypes comprises a plurality of PAM50 breast cancer subtypes.

9. using a first at least one processor or computing device to apply the second ML model, the first at least one processor or computing device having greater and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device being configured to: applying the first ML model; applying the third ML model; calculating the weighted combination; classifying the subject's breast cancer; 3. The computer-implemented method of claim 1, further comprising the steps of:

10. 3. The computer-implemented method of claim 1 or 2, comprising using multiple processors and / or computing devices located in different geographic locations and / or different medical facilities.

11. A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with a subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the third ML model comprising information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to one of the plurality of breast cancer subtypes, the third ML model being trained to classify medical data comprising information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; computing a weighted combination of the first output logits, the second output logits, and the third output logits; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination; a computer program comprising:

12. An information processing device comprising: a memory; and a processor coupled to the memory, the processor comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the medical data including information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to the plurality of breast cancer subtypes, the third ML model being trained to classify medical data including information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. An information processing device configured as follows.

13. 1. A system including a plurality of computing devices, the plurality of computing devices comprising: applying a first machine learning (ML) model to copy number variation (CNV) data associated with the subject to generate first output logits each corresponding to a plurality of breast cancer subtypes, the first ML model being trained to classify the CNV data to correspond to a subtype among the plurality of breast cancer subtypes; applying a second ML model to the histopathology image data associated with the subject to generate second output logits each corresponding to one of the plurality of breast cancer subtypes, the second ML model being trained to classify the histopathology data to correspond to a subtype among the plurality of breast cancer subtypes; applying a third ML model to medical data related to the subject, the medical data including information regarding a diagnosis of breast cancer in the subject, to generate third output logits each corresponding to the plurality of breast cancer subtypes, the third ML model being trained to classify medical data including information regarding a diagnosis of breast cancer to correspond to a subtype among the plurality of breast cancer subtypes; calculating a weighted combination of the first output logit, the second output logit, and the third output logit; classifying the subject's breast cancer as a subtype among the plurality of breast cancer subtypes based on the weighted combination. The system is configured as follows:

14. The plurality of computing devices includes a first at least one processor or computing device configured to apply the second ML model, the first at least one processor or computing device having greater and / or higher processing power than a second at least one processor or computing device, the second at least one processor or computing device being configured to: applying the first ML model; applying the third ML model; calculating the weighted combination; classifying the subject's breast cancer; The system of claim 13 , configured to do any of the following:

15. 15. The system of claim 13 or 14, wherein the plurality of computing devices comprises a plurality of processors and / or computing devices located in different geographic locations and / or different medical facilities.

Images