Systems and methods for determining breast cancer prognosis and associated characteristics - Patents.com

Abstract

Computer-implemented methods and machine learning models are provided for predicting the likelihood that a subject with breast cancer will recur after treatment, predicting tumors in whole slide images, and / or predicting biomarker status in breast cancer tissue.

Description

[Technical field]

[0001] (cross reference) This application claims the benefit of European Patent Application No. EP21306284.7, filed September 16, 2021, which is incorporated by reference in its entirety.

[0002] (Technical field) The present invention relates generally to machine learning and computer vision, and more particularly to image pre-processing and classification. [Background technology]

[0003] Histopathological imaging (HIA) is an important component of diagnosis in many fields of medicine, especially in oncology. The long-term prognosis of breast cancer is good, with an average 5-year survival rate estimated at 87%. Nevertheless, approximately 10% of patients relapse after initial treatment, either locally or with distant metastases. Early detection and management of recurrence can provide therapeutic benefits and improve the prognosis and quality of life (QOL) of these patients. Thus, there is a need for diagnostic tools to better assess the risk of recurrence and associated characteristics in breast cancer patients. Summary of the Invention

[0004] A method and apparatus for a device for classifying images is described.

[0005] In one aspect, disclosed herein is a computer-implemented method for predicting the likelihood that a subject with breast cancer will experience recurrence after treatment. In an exemplary embodiment, the method obtains a digital image of a histological section of a breast cancer sample obtained from the subject and one or more subject attributes obtained from the subject. The method further calculates an artificial intelligence (AI) risk score using a machine learning model, the machine learning model being trained by processing a plurality of training images to predict recurrence risk. Furthermore, the method calculates a clinical risk score using a clinical model and one or more subject attributes, the clinical model being trained using one or more subject training attributes from different subjects. Furthermore, the method calculates a final risk score for the subject from the AI ​​risk score and the clinical risk score, the final risk score representing the likelihood that the subject will experience recurrence after treatment.

[0006] In a further aspect, disclosed herein is a machine-readable medium having executable instructions for causing one or more processing units to execute a method for predicting the likelihood that a subject with breast cancer will experience a recurrence after treatment. In an exemplary embodiment, the machine-readable medium method obtains a digital image of a histological section of a breast cancer sample obtained from the subject and one or more subject attributes obtained from the subject. The machine-readable medium method further calculates an AI risk score using a machine learning model, the machine learning model being trained by processing a plurality of training images to predict the risk of recurrence. Furthermore, the machine-readable medium method calculates a clinical risk score using a clinical model and one or more subject attributes, the clinical model being trained using one or more subject training attributes from different subjects. Furthermore, the machine-readable medium method calculates a final risk score for the subject from the AI ​​risk score and the clinical risk score, the final risk score representing the likelihood that the subject will experience a recurrence after treatment.

[0007] The present invention is illustrated by way of example, and not by limitation, in the figures of the accompanying drawings in which like references indicate similar elements and in which: [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an exemplary flow diagram of a process for determining breast cancer prognosis using machine learning and clinical models, according to an embodiment of the present disclosure.

[0009] [Figure 2A-2B] FIG. 1 shows an exemplary flow diagram for a process of training and using a machine learning model to determine a breast cancer risk score.

[0010] [Figure 3A-3B] 3A-3B are an exemplary flow diagram for the process of training and using a clinical model to determine a breast cancer risk score.

[0011] [Figure 4A] FIG. 1 depicts metastasis-free interval (MFI) curves of the study cohort calculated including censored subjects of the baseline population, in accordance with the present disclosure. [Figure 4B] FIG. 1 depicts survival curves with censoring at the MFI endpoint.

[0012] [Figure 5A-5B] FIG. 5B illustrates an exemplary comparison of tumors annotated by a pathologist (FIG. 5A) and tumor predictions generated based on WSI using multi-layered percepts with MoCo features according to an embodiment of the present disclosure (FIG. 5B).

[0013] [Figures 6A-6D]The figure shows a comparison of breast cancer recurrence prediction (MFI) at 5 years after diagnosis in a specific population by a Cox model based only on baseline clinical variables (BV) ("Cox demo"), a Cox model based on extended clinical variables (EV) ("Cox all"), a deep learning (DL) model based only on WSI ("AI"), a combination of BV and DL models ("Cox demo+AI"), and a combination of EV and DL models ("Cox all+AI"). The time-dependent AUC of Uno at 5 years was used as an index to quantify the discriminatory ability of the models. [Figure 6A] Figure 1 depicts the results of cross-validation using DeepMIL with MoCo features of the DL model in the entire study cohort (n=1800, "all", left), subjects with no lymph node metastases at diagnosis ("n0", center), and subjects with lymph node metastases at diagnosis ("n+", right). [Figure 6B] FIG. 13 shows the results of cross-validation using DeepMIL with ImageNet features of DL models across the study cohort. [Figure 6C] Figure 1 shows the results of cross-validation using DeepMIL with MoCo features for DL ​​models in a population of ER+ / HER2- (also referred to as "HR+ / HER2-" in this disclosure) subjects (n=1437) ("all", left), subjects who had no lymph node metastases at diagnosis ("n0", center), and subjects who had lymph node metastases at diagnosis ("n+", right). [Figure 6D] FIG. 13 shows the results of cross-validation using DeepMIL with ImageNet features for DL ​​models of ER+ / HER2- subjects.

[0014] [Figure 7A-7B] Figure 1 shows the results of a multivariate analysis of the influence of all clinical variables used in the Cox model prediction of prognosis. The Shapley value represents the influence of the clinical variable on the model prediction. The clinical variables are listed in order of the magnitude of their influence on the model prediction.

[0015] [Figure 8A-8B] Figure 1 shows the results of multivariate analysis of the influence of clinical variables, excluding pN and pT, on the Cox model prediction of prognosis. Shapley values ​​represent the influence of the clinical variables on the model prediction. Clinical variables are listed in order of the magnitude of their influence on the model prediction.

[0016] [Figure 9A-9F] 9A-9E are graphs showing stratification of study cohort subjects into low and high risk of recurrence based on AI risk scores generated using DeepMIL trained with MoCo features according to an embodiment of the present disclosure. The following clinical features MFI (FIG. 9A), age (FIG. 9B), tumor grade (FIG. 9C), pN (FIG. 9D), pT (FIG. 9E), and Ki67 (FIG. 9F) of the high and low risk groups are depicted.

[0017] [Figure 10A] FIG. 1 illustrates example tiles associated with high risk of recurrence generated from whole slide images (WSIs) using DeepMIL with MoCo features according to an embodiment of the present disclosure.

[0018] [Figure 10B] FIG. 13 illustrates example tiles associated with low risk of recurrence generated from WSI using DeepMIL with MoCo features according to an embodiment of the present disclosure.

[0019] [Figure 10C] FIG. 13 illustrates example locations of predictive tiles and surrounding regions of high risk of recurrence within a WSI identified using DeepMIL with MoCo features according to one embodiment of the present disclosure.

[0020] [Figure 10D] FIG. 13 illustrates an example heatmap for prediction of tumors in a WSI generated using trained multi-layered perception with MoCo features according to one embodiment of the present disclosure.

[0021] [Figure 11A] FIG. 13 illustrates an example tile predicting high risk of recurrence generated from WSI using DeepMIL with MoCo features according to an embodiment of the present disclosure.

[0022] [Figure 11B] FIG. 13 illustrates an example tile that is a prediction of low risk of recurrence generated from WSI using DeepMIL with MoCo features according to an embodiment of the present disclosure.

[0023] [Figure 11C] FIG. 1 shows survival curves of subjects stratified into high, medium, and low risk groups based on the AI ​​risk score according to the present disclosure.

[0024] [Figure 12] 1 is an exemplary heatmap illustrating regions within a WSI having characteristics associated with high and low risk of recurrence, according to an embodiment of the present disclosure.

[0025] [Figure 13] FIG. 1 illustrates an exemplary tile predicting high risk and surrounding areas of recurrence within a WSI, according to an embodiment of the present disclosure.

[0026] [Figure 14] FIG. 1 illustrates exemplary tiles predicting tumor ER+ status by DeepMIL using ImageNet features and evaluation of prediction accuracy, according to an embodiment of the present disclosure.

[0027] [Figure 15] FIG. 1 illustrates exemplary tiles predicting tumor ER- status by DeepMIL with ImageNet features and evaluation of prediction accuracy, according to one embodiment of the present disclosure.

[0028] [Figure 16]FIG. 13 is a graph of sensitivity and specificity of DeepMIL prediction with ImageNet features of tumor ER status in analyzed tumor tissues, according to one embodiment of the present disclosure.

[0029] [Figure 17] 1 is a graph of sensitivity and specificity of DeepMIL prediction with ImageNet features of tumor PR status in analyzed tumor tissue, according to one embodiment of the present disclosure.

[0030] [Figure 18] 1 is a graph of sensitivity and specificity of DeepMIL prediction with ImageNet features of tumor Ki67 status in analyzed tumor tissues according to one embodiment of the present disclosure.

[0031] [Figure 19] 13 is a graph of sensitivity and specificity of DeepMIL prediction of tumor HER2 status in analyzed tumor tissue using ImageNet features, according to one embodiment of the present disclosure.

[0032] [Figure 20] FIG. 1 illustrates an example of a computer system that can be used in connection with the embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Computer-implemented methods, associated systems, devices, and computer-readable media for determining breast cancer diagnosis and / or prognosis are described. In some embodiments, provided herein are diagnostic tools that apply deep learning (DL) to digital images of tissue sections, e.g., whole slide images (WSI), and / or clinical data to aid in therapeutic decisions, to identify subjects with increased likelihood of recurrence after initial treatment for breast cancer, and / or to determine biomarker status in a subject's breast cancer tissue.

[0034] In the following description, numerous specific details are set forth in order to provide a thorough description of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention can be practiced without these specific details. In other instances, well-known components, structures and techniques are not shown in detail in order not to obscure the understanding of this specification.

[0035] References in this specification to "one embodiment" or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the present invention. The appearance of the phrase "in one embodiment" in various places in this specification does not necessarily refer to the same embodiment. The term "exemplary" is used in this specification to mean "example" rather than "ideal". It should be understood from this disclosure that the present invention is not limited to the examples described herein.

[0036] For any method described herein, unless otherwise specified or required by context, the order of steps presented, either in the text or in the accompanying flow diagrams, should not be interpreted as necessarily implying that these steps must be performed in the order presented. Rather, the order of steps is illustrative of one embodiment of the method provided, and in general, such steps may alternatively be performed in different orders or simultaneously. The processes depicted in the following figures may be performed by processing logic comprising hardware (e.g., circuits, dedicated logic, etc.), software (such as those executed on a general-purpose computer system or dedicated machine), or a combination of both. Although the process is described below in terms of some sequential operations, it should be understood that some of the described operations may be performed in different orders. Furthermore, some operations may be performed in parallel rather than sequentially.

[0037] Computing methods used to implement the methods provided herein may include, for example, machine learning, artificial intelligence (AI), deep learning (DL), neural networks, classification and / or clustering algorithms, and regression algorithms.

[0038] The terms "server," "client," and "device" are intended to refer generally to data processing systems, rather than to any particular form factor of a server, client, and / or device specifically.

[0039] The articles "a" and "an" are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, "an element" means one element or more than one element, e.g., a plurality of elements.

[0040] As used herein, the term "comprising" is used to mean, and is used synonymously with, the phrase "including but not limited to." The term "comprising" does not necessarily imply that there must be additional elements beyond the listed elements.

[0041] The term "about" or "approximately" when referring to a numerical value or numerical range means that the numerical value or numerical range referred to is an approximation within experimental variability (or within statistical experimental error), and thus the numerical value or numerical range may vary, for example, between 1% and 20% of the stated numerical value or numerical range. In some embodiments, "about" refers to a value within 20% of the stated value. In more preferred embodiments, "about" refers to a value within 10% of the stated value. In even more preferred embodiments, "about" refers to a value within 1% of the stated value.

[0042] Unless otherwise indicated, all numerical values ​​expressing the amounts of ingredients, properties such as molecular weight, reaction conditions, and the like used in the specification and claims are understood to be modified in all instances by the term "about". Thus, unless otherwise indicated, the numerical properties set forth in the following specification and claims are approximations that may vary depending on the desired properties sought to be obtained in the embodiments of the invention. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the invention are approximations, the numerical values ​​set forth in the specific examples are reported as precisely as possible. However, any numerical value inherently contains certain errors necessarily resulting from error found in their respective measurements.

[0043] The term "at least" before a numerical value or range of numerical values ​​is understood to include the number adjacent to the term "at least" and all subsequent numerical values ​​or integers that may be logically included, as is clear from the context. When "at least" is before a series of numerical values ​​or a range, it is understood that "at least" can modify each numerical value in the series or range.

[0044] As used herein, "not exceeding" or "less than" is to be understood as the logically smaller value or integer, up to zero (where negative values ​​are not possible), logically given the value adjacent to the expression and the context. When "not exceeding" is present before a series of numerical values ​​or ranges, it is understood that "not exceeding" can modify each of the numerical values ​​in the series or ranges.

[0045] As used herein, "up to ten," as in "up to ten," is understood in the context of non-negative integers to include up to and including ten, i.e., 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

[0046] When a range of values ​​is provided, it is understood that each intervening value between the upper and lower limit of that range (e.g., to one tenth of the unit of the lower limit unless the context clearly dictates otherwise), as well as any other stated or intervening value within that stated range, is encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also encompassed within the invention, subject to any specifically excluded limit in the stated range. When the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention.

[0047] "Patient" refers to a subject who exhibits symptoms and / or complications of a disease or condition (e.g., breast cancer), is under the care of a clinician (e.g., an oncologist), has been diagnosed with a disease or condition, and / or is at risk for developing a disease or condition. The term "patient" includes human and animal subjects. All references to a subject in this disclosure should be understood to include the possibility that the subject is a "patient" unless the context clearly dictates otherwise.

[0048] As used herein, in the context of this disclosure, "predict" or "predicting" refers to determining the likelihood of past, present, or future occurrence or non-occurrence of a disease, condition, or event (e.g., breast cancer recurrence). In some embodiments, the model (e.g., DeepMIL with MoCo features, DeepMIL with ImageNet features) can predict the likelihood of breast cancer recurrence, tumor area, or biomarker (e.g., PR, ER, HER2, Ki67) status by one or more of the following measures of test accuracy:

[0049] an odds ratio of greater than 1, preferably greater than about 2 or less than about 0.5, greater than about 3 or less than about 0.33, greater than about 4 or less than about 0.25, greater than about 5 or less than about 0.2, or greater than about 10 or less than about 0.1;

[0050] a specificity that is greater than 0.5, preferably at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95, with a corresponding sensitivity that is greater than 0.2, preferably at least about 0.3, at least about 0.4, at least about 0.5, at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95;

[0051] a sensitivity of at least 0.5, preferably at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95, with a corresponding sensitivity of at least 0.2, preferably at least about 0.3, at least about 0.4, at least about 0.5, at least about 0.6, at least about 0.7, at least about 0.8, at least about 0.9, or at least about 0.95;

[0052] A sensitivity of at least about 75% combined with a specificity of at least about 75%;

[0053] A positive likelihood ratio [calculated as sensitivity / (1-specificity)] of greater than 1, preferably at least about 2, at least about 3, at least about 4, at least about 5, at least about 10; or

[0054] A negative likelihood ratio [calculated as (1-sensitivity) / specificity] of less than 1, preferably about 0.5 or less, about 0.33 or less, about 0.25 or less, or about 0.1 or less.

[0055] As used herein, "risk score" refers to the likelihood that an event, e.g., disease, recurrence, will occur in the future. In some embodiments, the risk score represents the likelihood that a patient will experience recurrence after treatment. In some embodiments, the risk score is expressed as a category. In other embodiments, the risk score is expressed as a continuous range. In one embodiment, the risk score represents the likelihood that a patient will experience recurrence within 5 years from the date that a breast cancer sample is obtained from the patient.

[0056] As used herein, a "subject" is an animal, such as a mammal, including primates (such as humans, monkeys, and chimpanzees) or non-primates (such as cows, pigs, and horses), that would benefit from a method according to the present disclosure. In some embodiments of the present invention, the subject is a human, such as a human diagnosed with breast cancer. The subject may be a female human. The subject may be a male human. In some embodiments, the subject is an adult subject.

[0057] As used herein, "tumor" refers to the abnormal proliferation of cells or tissues and / or the resulting mass. In some embodiments, the tumor tissue or tumor cells are malignant (e.g., cancerous). As used herein, "cancer" refers to a tumor of epithelial origin in which abnormal cells divide uncontrollably and have the ability to invade or metastasize to adjacent or distant tissues or organs.

[0058] How to predict the likelihood of breast cancer recurrence In some aspects, provided herein are computer-implemented methods for predicting the likelihood that a patient with breast cancer will experience a recurrence after treatment.

[0059] Histology is the study of the microscopic characteristics of biological specimens. Histopathology refers to the microscopic examination of a specimen, e.g., tissue, obtained or otherwise obtained from a subject, e.g., a patient, to assess a disease state. A histopathology specimen is generally obtained by processing a specimen, e.g., tissue, in such a way that the specimen, or a portion thereof, is attached to a microscope slide. For example, a microtome or other suitable device can be used to obtain thin sections of the tissue specimen, which can be attached to a slide. The specimen can optionally be further processed, e.g., by applying stains, to aid in visualization of the specimen. Many stains have been developed to visualize cells and tissues. These include, but are not limited to, hematoxylin and eosin (H&E), methylene blue, Masson's trichrome, Congo red, Oil Red O, and safranin. H&E is routinely used by pathologists to aid in visualization of cells within tissue specimens. Hematoxylin stains the nuclei of cells blue, and eosin stains the cytoplasm and extracellular matrix pink. A pathologist visually inspecting an H&E stained slide can use this information to evaluate the morphological characteristics of the tissue. However, H&E stained slides typically contain insufficient information to visually evaluate the presence or absence of a particular biomarker. Visualization of a particular biomarker (e.g., protein or RNA biomarker) can be achieved with additional staining techniques, such as immunofluorescence, immunohistochemistry, in situ hybridization, etc., that rely on the use of labeled detection reagents that specifically bind to the marker of interest. While such techniques are useful for determining the expression of individual genes or proteins, they are impractical for evaluating complex expression patterns that include many biomarkers. Global expression profiling can be achieved by genomic and proteomic methods using separate samples obtained from the same tissue source as the specimen used for histopathological analysis. However, such methods are costly and time-consuming, require the use of specialized equipment and reagents, and do not provide information that correlates the expression of a biomarker with specific regions within a tissue specimen, e.g., specific regions within an H&E stained image.

[0060] As used herein, the term "digital image" refers to an electronic image represented by a collection of pixels that can be viewed, processed, and / or analyzed by a computer. In some aspects of the present disclosure, a digital image of a histology slide, e.g., an H&E stained slide, allows for computerized evaluation of the tissue specimen in addition to or in lieu of visual inspection by a pathologist. In some embodiments, the digital image can be acquired by a digital camera or other optical device capable of capturing a digital image from a slide or a portion thereof. In other embodiments, the digital image can be acquired by means of scanning a non-electronic image of a slide or a portion thereof. In some embodiments, the digital image used in the applications provided herein is a whole slide image. As used herein, the term "whole slide image (WSI)" refers to an image that includes all or nearly all of a tissue section, e.g., a tissue section present on a tissue slide. In some embodiments, a WSI includes an image of a whole slide. In other embodiments, the digital image used in the applications provided herein is a selected portion of a tissue section, e.g., a tissue section present on a tissue slide. In some embodiments, the digital image is acquired after the tissue section has been treated with a stain, e.g., H&E.

[0061] In some aspects, provided herein is a computer-implemented method for predicting the likelihood that a patient having breast cancer will experience a recurrence after treatment, the method comprising: obtaining a digital image of a histological section of a breast cancer sample obtained from a patient; obtaining clinical attributes from the subject; Dividing a digital image into a set of tiles; extracting a plurality of feature vectors from the set of tiles or a subset thereof; calculating an AI risk score using a machine learning model, the machine learning model being trained by processing a plurality of training images to predict risk of recurrence; calculating a clinical risk score using a clinical model, the clinical model being trained using one or more subject attributes; and calculating a final risk score for the subject from the AI ​​risk score and the clinical risk score, the final risk score representing the likelihood that the patient will experience a relapse after treatment.

[0062] In some embodiments, the digital images of the present methods are whole slide images (WSIs).

[0063] In some embodiments, the histological sections of the breast cancer samples are stained with a dye, such as H&E.

[0064] In some embodiments, the breast cancer sample is obtained from a patient prior to treatment for said breast cancer.

[0065] In some embodiments, the machine learning model is a Deep Multiple Instance Learning (DeepMIL) model. In some embodiments, the machine learning model is a Weldon model. In some embodiments, the machine learning model is a multi-layer perception model.

[0066] In some embodiments, the machine learning algorithm is a self-supervised learning algorithm. In some embodiments, the self-supervised learning algorithm is Momentum Contrast (MoCo) or Momentum Contrast v2. In some embodiments, the self-supervised learning algorithm is a DeepMIL model. In some embodiments, the DeepMIL model is trained on tissue images using MoCo or MoCo v2. In some embodiments, the DeepMIL model is trained on histological images using ImageNet.

[0067] In some embodiments, a machine learning algorithm, e.g., DeepMIL MoCo v2 or DeepMIL ImageNet, extracts a plurality of feature vectors from the digital image, and the extraction of the plurality of feature vectors is performed using a first convolutional neural network, e.g., a ResNet50 neural network.

[0068] In some embodiments, the computer-implemented method further includes removing background segments from the image. In some embodiments, removing the background segments from the image is performed using a second convolutional neural network. In some embodiments, the second convolutional neural network is a semantic segmentation deep learning network.

[0069] In some embodiments, the final risk score is calculated as a weighted average of the AI ​​risk score and the clinical risk score, where the weights can be the same or different.

[0070] In some embodiments, the computer-implemented method further comprises selecting a subset of the tiles for application to the machine learning model, hi some embodiments, the subset of tiles is selected by random sampling.

[0071] In some embodiments, the machine learning model is trained using a plurality of training images, and the plurality of training images includes digital images of histological sections of breast cancer samples obtained from a plurality of control subjects with breast cancer. In some embodiments, the plurality of training images includes images lacking local annotations. In some further embodiments, the plurality of training images includes images associated with one or more global labels indicative of one or more disease characteristics of the control patients from whom the samples were obtained. In some embodiments, the one or more disease characteristics are one or more of the following: time to breast cancer recurrence, age of the subject at the time of surgery, menopausal status, tumor stage, tumor size, number of positive lymph nodes (N+), number of nodes, type of surgery, type of treatment, estrogen receptor (ER) status, progesterone receptor (PR) status, HER2 status, tumor grade, Ki67 expression, histology, and / or the presence or absence of one or more mutations in BRCA genes or TP53 genes, or a combination thereof.

[0072] In some embodiments, the one or more indicative disease characteristics are time to breast cancer recurrence, survival rate, patient age at time of surgery, tumor stage, tumor size, tumor location (e.g., unifocal, multifocal), number of positive lymph nodes, and / or type of surgery, one or more biomarker status, tumor grade, and / or histology, or a combination thereof.

[0073] In some embodiments, time to breast cancer recurrence and / or prognosis is assessed by, for example, overall survival, invasive disease-free survival (iDFS), distant disease-free survival (dDFS), or metastasis-free interval (MFI).

[0074] As used herein, "overall survival" refers to the time from diagnosis of a particular disease, e.g., breast cancer, to death from any cause. "Overall survival" refers to the proportion of all subjects diagnosed with a particular disease, e.g., breast cancer, who are alive at some point after diagnosis.

[0075] As used herein, "invasive disease-free survival (iDFS)" in the context of breast cancer refers to the time from diagnosis of breast cancer to the occurrence of either ipsilateral invasive breast cancer recurrence, locally invasive breast cancer recurrence, distant recurrence, death from any cause, contralateral invasive breast cancer, or a second non-invasive breast cancer.

[0076] As used herein, "distant disease-free survival (dDFS)" in the context of breast cancer refers to the time from diagnosis to recurrence at a distant site or death from any cause.

[0077] As used herein, "metastasis-free interval (MFI)" in the context of breast cancer refers to the period from diagnosis of primary non-metastatic breast cancer to the date of the first distant metastasis.

[0078] In some embodiments, the one or more biomarkers are one or more of estrogen receptor (ER) status, progesterone receptor (PR) status, HER2 status, and Ki67 expression. As used herein, estrogen receptor (ER) and progesterone receptor (PR) are well-known receptors for the hormones estrogen and progesterone, respectively.

[0079] As used herein, "hormone receptor (HR)" status in the context of this disclosure refers to estrogen receptor (ER) status. HR positive breast cancer cells or tissues are ER positive.

[0080] ER(HR)-positive and / or PR-positive breast cancer can be treated with hormone therapy drugs that lower ligand (e.g., estrogen or progesterone) levels or block receptors. ER(HR)- and / or PR-positive cancers tend to grow slower than ER(HR)- and / or PR-negative cancers. Without wishing to be bound by theory, subjects with ER(HR)- and / or PR-positive breast cancer tend to have a better prognosis in the short term compared to subjects with ER(HR)- and / or PR-negative breast cancer, but are at risk of recurrence in the long term.

[0081] As used herein, human epidermal growth factor receptor 2 (HER2) refers to a growth-promoting receptor present on the membrane of all breast cells. Breast cancer cells that have higher than normal levels of HER2 are called HER2 positive. These cancers tend to grow and spread faster than other breast cancers, but are more likely to respond to treatment with drugs that target the HER2 protein.

[0082] In some embodiments, the computer-implemented method of the present disclosure includes obtaining one or more disease features of a patient, and applying a machine learning model to both the extracted features and the patient's disease features, where one or more of the patient's disease features are the same as one or more disease features represented in a global label associated with a training image.

[0083] In some embodiments, the risk score represents the likelihood that the patient will experience a recurrence within five years from the date the breast cancer sample was obtained from the patient.

[0084] As described above, in one embodiment, the risk score is calculated based on at least the risk score of the machine learning model and the risk score of the clinical model. In this embodiment, the accuracy of the resulting model is improved by using a combination of the machine learning model and the clinical model to determine the final risk score. For example, in one embodiment (see FIG. 6D left and Example 1), for a given validation training set of whole breast slide images of a known subject group, the machine learning model risk score gives an Uno time-dependent area under the curve (AUC) metric of 0.77 (e.g., the DeepMIL machine learning model described below), with a higher AUC representing a higher discriminatory ability of the model. For the same subject group, the risk score based on the clinical model (e.g., the Cox model) gives an AUC of 0.77. However, in this embodiment, by using the average of the machine learning model risk score and the clinical model risk score, the final risk score increases to an AUC of 0.81. This is an unexpected increase in AUC using the hybrid risk score (e.g., the average of the machine learning model risk score and the clinical model risk score), since the resulting risk score shows a better AUC than either the AI ​​risk score or the clinical model risk score.

[0085] In some embodiments, the machine learning model provides a higher accuracy of predicting breast cancer recurrence compared to the clinical model, hi some embodiments, the machine learning model has a higher AUC of predicting breast cancer recurrence of 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.25, 0.30, 0.35, 0.40 or more compared to the clinical model.

[0086] In some embodiments, combining machine learning and clinical models to generate a hybrid risk score provides a higher accuracy of predicting breast cancer recurrence compared to machine learning models alone or clinical models alone. In some embodiments, combining machine learning and clinical models provides a larger AUC for predicting breast cancer recurrence than machine learning models alone, such as 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.25, 0.30, 0.35, 0.40 or more. In some embodiments, the combination of the machine learning model and the clinical model results in a larger AUC for predicting breast cancer recurrence of 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.25, 0.30, 0.35, 0.40 or more compared to the clinical model alone.

[0087] 1 shows an exemplary flow diagram of a process 100 for determining breast cancer prognosis using machine learning and clinical models, according to one embodiment of the present disclosure. In block 101, the process 100 trains a machine learning model that can be used to calculate a risk score for breast cancer recurrence. In one embodiment, the process 100 trains the machine learning model using a training set of digital images, where each digital image is a histological section of a breast cancer sample obtained from a patient. In some embodiments, the digital images can include a digital WSI.

[0088] In some embodiments, the histological sections of the breast cancer samples are stained with a dye to visualize the underlying tissue structures, for example with hematoxylin and eosin (H&E). Other common stains that can be used to visualize tissue structures in the input images include, for example, Masson's Trichome stain, Periodic Acid Schiff stain, Prussian Blue stain, Gomori's Trichome stain, Alcian Blue stain, or Ziel-Neelsen stain. In some embodiments, the breast cancer samples are obtained from patients with pre-treatment of breast cancer.

[0089] In this embodiment, the machine learning model can be a self-supervised machine learning model that uses histology data (such as whole slide images) by extracting features from tiles of whole slide images. In one embodiment, the feature extractor is trained on in-domain histology tiles without annotations. In one embodiment, to apply the self-supervised framework to histology data, tiles are concatenated from all WSIs extracted in the tiling step to form a training dataset. A feature extractor is then trained on this set of unlabeled tile images using MoCo v2. First, the set of tiles can be split into two batches of tiles. The first batch of tiles and the second batch of tiles can be modified, for example, by adding a 90° rotation and vertical flip, and further performing color enhancement. Rotation is a good enhancement to perform, since histology tiles contain the same information regardless of their orientation. Since histology tiles are images containing cells or tissues and are orientation independent, such tiles can be viewed properly regardless of rotation, horizontal flip, etc. Thus, rotating the image can provide valuable enhancements without losing important features of the image. By applying a batch of tiles to each feature extractor, a tile embedding can be generated.

[0090] In one embodiment, the tile embedding is the output of the feature extractor and serves as a signature for each tile that contains semantic information for that tile. In other words, the tile embedding is a representation of the tile that contains semantic information about that tile.

[0091] In some embodiments, the self-supervised learning algorithm uses a contrastive loss to form a tile embedding such that different augmented views of the same image are close or have similar tile embeddings. In other words, the contrastive loss can compare two tile embeddings, and based on the comparison, the first feature extractor can be adjusted so that its tile embedding is similar to the tile embedding of the second feature extractor. The gradient is back-propagated through the first feature extractor. In some embodiments, the weights of the second feature extractor are updated with an exponential moving average (EMA) of the weights of the first extractor. By using EMA, some embodiments can avoid overfitting. Thus, the output of the system is a trained feature extractor, which has been trained with in-domain histology tiles such that the tile embeddings of different augmentations of the same image are similar. This kind of specially trained feature extractor can provide significant improvements in downstream performance, as described below. In some embodiments, the trained feature extractor can be achieved after a certain number of epochs of training. In some embodiments, training is performed until the accuracy is at or near 1 (or 100%), the AUC is at or near 1 (or 100%), or the loss is near zero. In some embodiments, during training of a feature extractor with contrastive loss, useful metrics may not be available in abundance. Therefore, one of the available metrics of the downstream task, such as AUC, can be monitored to see how the feature extractor is performing. As an example, a feature extractor trained at a particular epoch can be used to train a downstream weakly supervised task and evaluate the performance. Additional training may be deemed necessary if such training could improve downstream performance.

[0092] In some embodiments, the second feature extractor may be optional, and a single feature extractor may be used to generate tile embeddings from the two batches of tiles. In such embodiments, one feature extractor is used to generate tile embeddings from the two batches of tiles, and a contrastive loss, as described above, is used to compare the two tile embeddings, and tune the first feature extractor so that the tile embeddings are similar to the tile embeddings.

[0093] At block 103, the process 100 trains a clinical model. In one embodiment, the process 100 trains a proportional hazards model using a set of clinical attributes associated with the subject. In this embodiment, the clinical attributes can be age, treatment characteristics, biomarker characteristics (e.g., ER, PR, HER2, Ki67 status), histological characteristics (grade, histological subtype), tumor characteristics (tumor size, tumor stage, tumor location), lymph node characteristics (number of involved lymph nodes). Furthermore, the clinical model can be a proportional hazards model, which is a statistical model used to associate the time that has passed before some event (e.g., recurrence in breast cancer) occurs. For example, in one embodiment, the proportional hazards model can be a Cox model that uses a fitting algorithm to fit the clinical data of the training set (e.g., known recurrence times and the clinical attributes outlined above). Training of the clinical model is further described in FIG. 3A below.

[0094] With the trained machine learning model and clinical model, the process 100 can receive the WSI and clinical attributes to determine a risk score for the subject. The process 100 receives the WSI and one or more clinical attributes of the subject at block 105. In one embodiment, the WSI is a digital image associated with the subject and the one or more clinical attributes are attributes used as inputs for the clinical model. At block 107, the process 100 calculates an AI risk score using the trained machine learning model. The calculation of the AI ​​risk score is further described in FIG. 2B below. At block 111, the process 100 calculates a clinical risk score using the trained clinical model. The step of calculating the clinical risk score is further described in FIG. 3B below.

[0095] Using the machine learning and clinical risk scores calculated for the subject, process 100 can calculate a final risk score from the machine learning and clinical risk scores. In one embodiment, the final risk score is the average of the machine learning and clinical risk scores. For example, in one embodiment, JPEG2024537681000002.jpg9150, where Rf is the final risk score, Rm is the AI ​​risk score, and Rc is the machine clinical risk score. Alternatively, the final risk score can be a weighted average of the machine learning and clinical risk scores. For example, in one embodiment: JPEG2024537681000003.jpg6150, where am and am are the weights of Rm and Rc, respectively. In further embodiments, the process 100 can calculate the final risk score in a different manner (e.g., as the square root of the sum of the squares or another function of the two inputs).

[0096] As described above, process 100 uses two trained models to arrive at a final risk score. Figures 2A-2B are an example flow diagram of a process for training and using a machine learning model to determine a breast cancer risk score. Figure 2 is a flow diagram of an embodiment of a process 200 for using self-supervised learning on histological images to train a feature extractor, according to an embodiment of the present disclosure. In Figure 2, process 200 begins with step 201 receiving a training set of histological images. In some embodiments, each image in the training set of images is an unannotated whole slide image.

[0097] At block 203, the process 200 proceeds to tiling and augmenting the training set of images into a set of tiles. In one embodiment, the digital images can be divided into a set of tiles. Tiling the images can include dividing the original image into smaller, more manageable images called tiles. In one embodiment, the tiling operation is performed by applying a fixed grid to the whole slide images using a segmentation mask generated by the segmentation method to select tiles that contain tissue or other regions of interest. To further reduce the number of tiles to process, in one embodiment, additional or alternative selection methods can be used, such as random subsampling, which leaves only a given number of slides.

[0098] In one embodiment, augmentation may be applied to each of the set of tiles. The process 200 proceeds to generate a set of processed tiles by performing the following operations for each batch of tiles selected from the set of tiles: In block 205, a first set of features is extracted from the first batch of augmented tiles. In block 207, a second set of features is extracted from the second batch of augmented tiles. In some embodiments, the augmented tiles include zoomed-in or rotated views, or views with color augmentation. For example, in tissue slides, orientation is not important, so the slides can be rotated at various angles. The slides can also be enlarged or zoomed in. In block 209, the process 200 uses contrast loss between pairs of the first and second sets of extracted features to bring matching tile pairs closer together and dissimilar tile pairs further apart. The contrast loss is applied to pay attention to positive pairs taken from the first and second sets of features, rather than negative pairs.

[0099] At block 211, process 200 proceeds to train a feature extractor using the processed set of tiles generated via operations 205-209. In some embodiments, the trained feature extractor disclosed herein can be used to improve classification of histological images. At block 213, process 200 proceeds to output a trained feature extractor that has been trained using a self-supervised ML algorithm. In some embodiments, the feature extractor can be trained for a particular number of epochs (e.g., 200 epochs) such that each training image is seen a particular number of times.

[0100] A risk score can be calculated using the trained machine learning model. Figure 2B is a flow diagram of an embodiment of a process 250 for identifying a region of interest in a histological image, according to an embodiment of the present disclosure. In Figure 2B, the process 250 begins with receiving an input tissue image at block 251. In some embodiments, the input histological image is a WSI and may be obtained from a patient tissue sample. In some embodiments, the patient tissue sample is known or suspected to contain a tumor.

[0101] In some embodiments, the process 250 includes removing background segments from the input image. In some embodiments, material detection can be used to sample only tiles from tissue regions of the input image. In some embodiments, the input image can be converted to Hue-Saturation-Value (HSV) color space, after which the background can be removed using Otsu's method applied to the hue and saturation channels.

[0102] At block 253, the process 250 proceeds to tile the tissue image into a set of tiles. In one embodiment, the process 250 uses tiling to enhance the ability to pre-process the image. For example, in one embodiment, using a tiling method is useful in histopathological analysis due to the large size of whole slide images. More broadly, when dealing with specialized images such as histopathological slides, or satellite images, or other types of large images, the resolution of the image sensors used in these fields can grow as quickly as the capacity of the random access memory associated with the sensor. This large image size makes it difficult to store a batch of images, or even a single image in the random access memory of a computer. This difficulty is further amplified when trying to store these large images in the dedicated memory of a graphics processing unit (GPU). This situation makes it computationally difficult to process slide images, or other similarly sized images, in their entirety.

[0103] In one embodiment, a step of tiling the image (or the image minus the background) addresses this challenge by dividing the original image (or the image minus the background) into smaller, more manageable images called tiles. In one embodiment, the tiling operation is performed by applying a fixed grid to the whole slide image using a segmentation mask generated by the segmentation method, and selecting tiles that contain tissues or other types of regions of interest for subsequent classification processing. As used herein, a "region of interest" of an image can be any region that is semantically related to the task to be performed, and in particular in the context of histopathology, regions that correspond to tissues, organs, bones, cells, body fluids, etc. To further reduce the number of tiles to be processed, additional or alternative selection methods can be used, such as random subsampling that leaves only a predetermined number of slides.

[0104] For example, in one embodiment, process 250 divides the image (or the image minus the background) into tiles of a fixed size (e.g., each tile has a size of 224x224 pixels). Alternatively, the tile size can be smaller or larger. In this example, the number of tiles generated depends on the size of the detected material and can vary from a few hundred tiles to 50,000 tiles or more. In one embodiment, the number of tiles is limited to a configurable fixed number based at least on computation time and memory requirements (e.g., 10,000 tiles).

[0105] For each tile, the process 250 proceeds to extract one or more features of that tile at block 255. In one embodiment, each of the features is extracted by applying a trained feature extractor trained with a contrast loss ML algorithm using a training set of images. In one embodiment, the training set of images is a set of images without annotations. In one embodiment, the input images and the training set of images are of the same domain, meaning images of the same category or type. For example, the input images and the training set of images are both tissue images. This is in contrast to an embodiment where the training set of images includes images outside the domain, or images that are not histological images, or images that are not of the same category or type as the images being analyzed. In one embodiment, the contrast loss ML algorithm is Momentum Contrast, or Momentum Contrast v2 (MoCo v2). In some embodiments, the trained feature extractor is an ImageNet type feature extractor. In one embodiment, the trained machine learning model is the machine learning model trained in FIG. 2A above.

[0106] In some embodiments, the machine learning model is a deep multiple instance learning model. In some embodiments, the machine learning model is a Weldon model. In some embodiments, the machine learning model is applied to the entire group of tiles. In some embodiments, the machine learning model is applied to a subset of tiles. The training images can include digital images of histological sections of breast cancer samples obtained from many control subjects. In some cases, the training images are devoid of local annotations. The training images can include images associated with one or more global labels indicative of one or more disease characteristics of the control patients from whom the samples were obtained. The disease characteristics, in some embodiments, can include a time period to breast cancer recurrence. In other embodiments, the one or more disease characteristics can include one or more of patient age at time of surgery, tumor stage, tumor size, number of positive lymph nodes (N+), number of nodes, and / or type of surgery, or a combination thereof. In other embodiments, the disease characteristics can include one or more of estrogen receptor (ER) status, progesterone receptor (PR) status, HER2 status, tumor grade, Ki67 expression, and / or histology, or a combination thereof.

[0107] In some embodiments, one or more disease features of the patient can be obtained, and the machine learning model is applied to both the extracted features and the disease features of the patient. The examples of the disease features of the patient can be the same as the disease features represented in the global labels associated with the training images.

[0108] In block 259, the process 250 uses the machine learning model to calculate a risk score for the subject. The risk score represents the likelihood that the patient will experience a recurrence after treatment and can be expressed as a categorical or continuous range. In one embodiment, the risk score represents the likelihood that the patient will experience a recurrence within five years from the date the breast cancer sample was obtained from the patient.

[0109] In addition to the AI ​​risk score calculated using the trained machine learning model, a clinical model is trained and used to calculate a clinical risk score. FIGS. 3A-3B are exemplary flow diagrams of a process for training and using a clinical model to determine a risk score for breast cancer. In FIG. 3A, process 300 begins by receiving training clinical attributes and risk score results for a training set of subjects at block 301. In one embodiment, the training clinical attributes can be one or more of age at surgery, tumor stage (pT), tumor size, number of cancer positive lymph nodes (N+), node count, type of surgery, hormone receptor (e.g., ER, PR, HER2) status, Ki67 status, tumor grade, histology, and genetic abnormalities. At block 303, process 300 trains a clinical model using the training set of clinical attributes and risk scores.

[0110] The trained model can be used to calculate a clinical risk score. Figure 3B shows an example flow diagram of a process 350 for determining a calculated risk score using a clinical model, according to one embodiment of the present disclosure. In Figure 3B, the process 350 begins with receiving input clinical attributes of a subject in block 351. In block 353, the process 350 calculates a risk score using the trained clinical model. As described above, the clinical risk score can be combined with the AI ​​risk score to obtain a final risk score.

[0111] Methods for predicting biomarker status in breast cancer In some embodiments, provided herein are computer-implemented methods for predicting biomarker status in breast cancer tissue. In some embodiments, the same models for predicting risk of breast cancer recurrence can be used to predict biomarker status in breast cancer tissue. In some embodiments, the methods for predicting biomarkers include: obtaining a digital image of a histological section of a breast cancer sample obtained from the subject; Dividing the digital image into a set of tiles; Extracting a number of feature vectors from the set of tiles or a subset thereof; calculating the biomarker state using a machine learning model trained by processing a plurality of training images to predict the biomarker state; Includes.

[0112] In some embodiments, the biomarker is estrogen receptor (ER), progesterone receptor (PR), HER2, or Ki67. In some embodiments, the predicted marker status is positive or negative. In some embodiments, the predicted marker status is expressed as a value within a continuous range of values: the lower and upper ends of the range represent, respectively, the lower and upper limits of detectable levels of the marker in the tissue; or The lower and upper ends of the range represent the highest and lowest limits of detection of the marker in the tissue.

[0113] In some embodiments, the digital images used in accordance with the methods of the present disclosure are whole slide images.

[0114] In some embodiments, the tissue section of the breast cancer sample is stained with a dye, hi some embodiments, the dye is hematoxylin and eosin (H&E).

[0115] In some embodiments, the machine learning model is a Deep Multiple Instance Learning (DeepMIL) model. In some embodiments, the machine learning model is a Weldon model. In some embodiments, the machine learning model is a multi-layer perception model.

[0116] In some embodiments, the machine learning algorithm is a self-supervised learning algorithm. In some embodiments, the self-supervised learning algorithm is Momentum Contrast (MoCo) or Momentum Contrast v2. In some embodiments, the self-supervised learning algorithm is DeepMIL. In some embodiments, the DeepMIL model is trained on histological images using MoCo or MoCo v2. In some embodiments, the DeepMIL model is trained on histological images using ImageNet. The computer-implemented method for predicting biomarker status in breast cancer tissue can incorporate any features or algorithms described elsewhere in this disclosure.

[0117] In some embodiments, the extraction of the plurality of feature vectors is performed using a first convolutional neural network. In some embodiments, the first convolutional neural network is a ResNet50 neural network. In some embodiments, a software platform for training and using machine learning models can be used to analyze digitized pathology slides and predict biomarker status in tissue.

[0118] In some embodiments, the method further comprises removing background segments from the image. In some embodiments, removing the background segments from the image is performed using a second convolutional neural network. In some embodiments, the second convolutional neural network is a semantic segmentation deep learning network.

[0119] In some embodiments, the methods of the present disclosure further comprise selecting a subset of tiles for application to the machine learning model, hi some embodiments, the subset of tiles is selected by random sampling.

[0120] In some embodiments, the machine learning model is trained using a plurality of training images, the plurality of training images comprising digital images of histological sections of breast cancer samples obtained from a plurality of control subjects. In further embodiments, the plurality of training images comprises images lacking local annotations. In some embodiments, the plurality of training images comprises images associated with one or more global labels indicative of one or more disease features of the control subject from which the samples were obtained.

[0121] In some embodiments, the one or more indicative disease features include one or more biomarker status or a combination thereof, for example, ER, PR, HER2, Ki67.

[0122] In some embodiments, the method includes obtaining one or more disease features of a subject and applying a machine learning model to both the extracted features and the disease features of the subject, wherein the one or more disease features of the subject are the same as one or more disease features represented in the global labels associated with the training images.

[0123] In some embodiments, the machine learning models of the present disclosure can identify novel biomarkers for the diagnosis and prognosis of breast cancer based on the provided histological information, e.g., WSI.

[0124] Computer System and Machine-Readable Medium As shown in FIG. 20, a computer system 2000, which is one form of data processing system, includes a microprocessor 2005 and a bus 2003 coupled to a ROM (Read Only Memory) 2007, a volatile RAM 2009, and a non-volatile memory 2013. The microprocessor 2005 can include one or more CPUs, GPUs, special purpose processors, and / or combinations thereof. The microprocessor 2005 can communicate with a cache 2004 and can retrieve and execute instructions from the memories 2007, 2009, and 2013 to perform the operations described above. The bus 2003 interconnects these various components and also interconnects these components 2005, 2007, 2009, and 2013 to peripheral devices such as a display controller and display device 2015, and input / output (I / O) devices 2011, which can be a mouse, keyboard, modem, network interface, printer, and other devices known in the art. Typically, the input / output devices 2011 are coupled to the system through an input / output controller 2017. Volatile RAM (random access memory) 2009 is typically implemented as dynamic RAM (DRAM) and requires continuous power to refresh or maintain the data in the memory.

[0125] The non-volatile memory 2013 may be, for example, a magnetic hard drive or a magnetic optical drive or an optical drive or a DVD RAM or a flash memory or other type of memory system that retains data (e.g., large amounts of data) even after power is removed from the system. Typically, the non-volatile memory 2013 is also a random access memory, although this is not required. While FIG. 20 illustrates that the non-volatile memory 2013 is a local device directly coupled to the remaining components in the data processing system, it will be understood that the present invention may utilize non-volatile memory that is remote from the system, such as a network storage device coupled to the data processing system via a network interface, such as a modem, an Ethernet interface, or a wireless network. The bus 2003 may include one or more buses connected together through various bridges, controllers, and / or adapters, as is well known in the art.

[0126] Some of the above may be implemented using logic circuitry, such as dedicated logic circuitry, or using a microcontroller or other type of processing core that executes program code instructions. Thus, the processes taught by the above discussion may be implemented using program code, such as machine-executable instructions that cause a machine that executes those instructions to perform a particular function. In this context, a "machine" may be a machine that converts intermediate-form (or "abstract") instructions into processor-specific instructions (e.g., abstract execution environments such as a "virtual machine" (e.g., Java Virtual Machine), an interpreter, a common language runtime, a high-level language virtual machine, etc.), and / or electronic circuitry (e.g., "logic circuitry" implemented with transistors) located on a semiconductor chip that is designed to execute instructions, such as a general-purpose processor and / or a special-purpose processor. The processes taught by the above discussion may also be implemented by (as a replacement for or in combination with a machine) electronic circuitry that is designed to execute the process (or a portion thereof) without the execution of program code.

[0127] The present invention also relates to an apparatus for performing the operations described herein, which may be specially constructed for the required purposes or may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such computer programs may be stored on a computer readable storage medium such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read only memory (ROM), RAM, EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0128] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, and the like.

[0129] The article of manufacture can be used to store program code. The article of manufacture storing the program code can be embodied as, but is not limited to, one or more memories (e.g., one or more flash memories, random access memories (static, dynamic or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards, or other types of machine-readable media suitable for storing electronic instructions. The program code can also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by a data signal embodied in a propagation medium (e.g., via a communications link (e.g., a network connection)).

[0130] The preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0131] It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As is apparent from the above discussion, unless otherwise specifically stated, throughout this specification, discussions using terms such as "segmentation," "tiling," "receiving," "calculating," "extracting," "processing," "applying," "augmenting," "normalizing," "pre-training," "screening," "selecting," "aggregating," "sorting," and the like are understood to refer to the operations and processes of a computer system or similar electronic computing device that operates and transforms data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system, or other such information storage, transmission, or display device.

[0132] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the described operations. The required structure for a variety of these systems will appear from the description herein. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0133] product In some embodiments, provided herein is a product capable of identifying subjects at high risk of recurrence of breast cancer, identifying areas of a tissue section of a tumor at high or low risk of recurrence, identifying tumor or malignant areas within a WSI, and / or identifying the status of biomarkers associated with breast cancer within a tumor of a subject. In some embodiments, the product is attached to a scanner. In some embodiments, the scanner can scan pathology slides, such as H&E slides. In some embodiments, the product is particularly useful for medical facilities, clinics, or providers that do not have expertise in breast cancer pathology, including making breast cancer diagnoses or prognoses, or access to molecular testing. In some embodiments, the products are useful for identifying personalized or targeted therapy options for breast cancer in a subject. EXAMPLES

[0134] Example 1: Predicting breast cancer recurrence using H&E whole slide images

[0135] Study cohort A cohort of 1813 individuals diagnosed with early stage breast cancer (Grand TMA) at Gustave Roussy between 2005 and 2013 was included in the study. Of this total population, 1437 individuals were diagnosed with ER+ HER2- breast cancer between 2005 and 2013. As described elsewhere in this disclosure, some analyses of the study focused on this subpopulation. All subjects in the cohort underwent surgical resection and all follow-up and hematoxylin and eosin (H&E) stained slides were available.

[0136] Clinical and pathological data collected and reviewed included, but were not limited to, age, tumor size, grade, histology, pT, pN, mutations (e.g., BRCA or TP53 genes), molecular subtype, HER2, Ki67, RE, and RP. Treatment data considered type of surgery, previous radiation therapy, previous hormonal therapy, and type of chemotherapy.

[0137] Assessment of survival In monitoring the subjects enrolled in this study, four different survival endpoints were considered, including overall survival from the time of diagnosis (t=0), invasive disease-free survival (iDFS), distant disease-free survival (dDFS) and metastasis-free interval (MFI). In Figures 4A and 4B, "time at risk" refers to subjects who were initially diagnosed with breast cancer but did not develop metastasis. "Censored" refers to subjects who dropped out of the study or for whom data was unavailable. "Event" refers to subjects with breast cancer who developed metastasis. As shown in Figure 4A, the MFI survival curve (event / event + time at risk + censoring) showed a gradual and continuous decline over 120 months (10 years). As shown in Figure 4B, the censored MFI curve showed a gradual decline from 0 months to 60 months (5 years), followed by a steep decline from 60 months to 120 months (5-10 years). The sharp decline at 5 years and beyond may be due to a large number of censored subjects, including those lost to institutional follow-up after 5 years of disease-free survival.

[0138] Deep Learning The AI ​​risk score was calculated based on the WSI of breast tissue as follows: H&E stained slides, available from all subjects enrolled in the study, were digitized, pre-processed, and cut into small patches. These tiles were fed into a deep learning ("DL") network, i.e., DeepMIL MoCo v2 or DeepMIL ImageNet. First, a machine learning model was trained using Momentum Contrast v2 (MoCo v2) or ImageNet by receiving and processing training tiles as shown in FIG. 2A in accordance with the present disclosure. The trained ML model then received tiles generated from the WSI, and an AI risk score, representing the risk of recurrence, was calculated based on a weighted average of the tile features as shown in FIG. 2B in accordance with the present disclosure.

[0139] FIG. 5A shows an example tumor in a WSI annotated by a pathologist in a software platform for training and using machine learning models for diagnosis and prognosis. FIG. 5B shows tumor prediction in a WSI using Multi-Layer Perception with MoCo features in the platform. As shown in FIG. 5B, given a tile or WSI, a machine learning model can predict and identify the tumor location within the tile or WSI.

[0140] In addition to the DL model applied to WSI, a clinical risk score was determined using clinical variables as follows: First, a Cox model was trained by receiving the training clinical attributes of a training set of subjects as shown in FIG. 3A and the risk score results according to the present disclosure. The clinical attributes included baseline clinical variables (BV) (age at surgery, tumor stage (pT), tumor size, tumor node number, infiltrated lymph node number, surgery type), and extended clinical variables (EV) [combination of BV with ER / PR / HER2 / Ki67 status, tumor grade, histology, and gene mutation status (e.g., BRCA, TP53)]. The trained Cox model then received the clinical attributes, and a clinical risk score representing the risk of recurrence was calculated based on the weighted average of the clinical features, as shown in FIG. 3B according to the present disclosure.

[0141] In some models, a final risk score was determined for the subject based on the AI ​​risk score based on the WSI and the clinical risk score based on clinical attributes. The final risk score was calculated as the average of the machine learning risk score and the clinical risk score.

[0142] For each model, performance was evaluated using cross-validation. Metastasis-free interval (MFI) was selected as the primary survival endpoint. The time-dependent AUC of Uno was used as an index to quantify the discriminatory ability of the model.

[0143] Results – Prediction of recurrence risk based on deep learning FIG. 6A shows the cross-validation results of recurrence risk in the entire study cohort using DeepMIL MoCo v2 and / or clinical risk scores. As shown in the “all” group (left panel) of FIG. 6A, in the entire population (n=1800), 5-year survival prediction (MFI) based on baseline clinical variables (BV) alone resulted in an AUC of 0.76 (“Cox demo”), and the model based on extended clinical variables (EV) resulted in an AUC of 0.82 (“Cox all”). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.74 (“AI”). When BV and DeepMIL MoCo v2 were combined, the AUC improved to 0.79 (“Cox demo + AI”). When EV and DeepMIL MoCo v2 were combined, the AUC was 0.82 (“Cox all + AI”).

[0144] As shown in the "n0" group in Figure 6A (middle panel), in the subpopulation of subjects without lymph node metastases at diagnosis (N0; n = 1180), prediction of 5-year survival (MFI) based on BV alone resulted in an AUC of 0.72 ("Cox demo"), whereas the model based on EV yield resulted in an AUC of 0.82 ("Cox all"). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.73 ("AI"). Combining BV with DeepMIL MoCo v2 improved the AUC to 0.76 ("Cox demo + AI"). Combining AEV with DeepMIL MoCo v2 resulted in an AUC of 0.82 ("Cox all + AI").

[0145] As shown in the "n+" group in Figure 6A (right panel), in the subpopulation of subjects with lymph node metastases at diagnosis (N+; n = 615), prediction of 5-year survival (MFI) based on BV alone resulted in an AUC of 0.71 ("Cox demo"), while the model based on EV resulted in an AUC of 0.75 ("Cox all"). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.70 ("AI"). Combining BV and DeepMIL MoCo v2 improved the AUC to 0.75 ("Cox demo + AI"). Combining EV and DeepMIL MoCo v2 improved the AUC to 0.76 ("Cox all + AI").

[0146] Figure 6B shows the cross-validation results of recurrence risk in the entire study cohort using DeepMIL ImageNet and / or clinical risk scores. As shown in the "all" group in Figure 6B (left panel), in the entire population (n=1800), prediction of 5-year survival rate (MFI) based on BV alone resulted in an AUC of 0.76 ("Cox demo"), and the model based on EV resulted in an AUC of 0.82 ("Cox all") (also shown in Figure 6A (left panel)). The AUC of DeepMIL ImageNet based on WSI alone was 0.77 ("AI"). Combining BV with DeepMIL ImageNet improved the AUC to 0.81 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.83 ("Cox all + AI").

[0147] In the subpopulation of subjects who did not have lymph node metastasis at diagnosis (N0; n=1180), as shown in the "n0" group in Figure 6B (middle panel), prediction of 5-year survival rate (MFI) based on BV alone resulted in an AUC of 0.72 ("Cox demo"), while the model based on EV resulted in an AUC of 0.82 ("Cox all"), also shown in Figure 6A (middle panel). Meanwhile, the AUC of DeepMIL ImageNet based on WSI alone was 0.76 ("AI"). Combining BV with DeepMIL ImageNet improved the AUC to 0.78 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.83 ("Cox all + AI").

[0148] In the subpopulation of subjects with lymph node metastasis at diagnosis (N+; n=615), prediction of 5-year survival (MFI) based on BV alone resulted in an AUC of 0.71 ("Cox demo"), as shown in the "n+" group in Figure 6B (right panel), and the model based on EV resulted in an AUC of 0.75 ("Cox all"), as also shown in Figure 6A (right panel). Meanwhile, the AUC of DeepMIL ImageNet based on WSI alone was 0.74 ("AI"). Combining BV with DeepMIL ImageNet improved the AUC to 0.77 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.79 ("Cox all + AI").

[0149] FIG. 6C shows cross-validation results of recurrence risk in the ER+ / HER2- subpopulation using DeepMIL MoCo v2 and / or clinical risk scores. As shown in the "all" group in FIG. 6C (left panel), in the ER+ / HER2- subject population (n=1437), prediction of 5-year survival rate (MFI) based on BV alone had an AUC of 0.77 ("Cox demo"), while the model based on EV had an AUC of 0.80 ("Cox all"). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.77 ("AI"). When BV and DeepMIL MoCo v2 were combined, the AUC improved to 0.81 ("Cox demo + AI"). When EV and DeepMIL MoCo v2 were combined, the AUC improved to 0.83 ("Cox all + AI").

[0150] As shown in the "all" group in Figure 6C (middle panel), in the subpopulation of ER+ / HER2- / N0 subjects (n=957), prediction of 5-year survival (MFI) based on BV alone yielded an AUC of 0.72 ("Cox demo"), whereas the model based on EV yielded an AUC of 0.81 ("Cox all"). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.77 ("AI"). Combining BV and DeepMIL MoCo v2 resulted in an AUC of 0.77 ("Cox demo + AI"). Combining EV and DeepMIL MoCo v2 improved the AUC to 0.82 ("Cox all + AI").

[0151] As shown in the "all" group in Figure 6C (right panel), in the subpopulation of ER+ / HER2- / N+ subjects (n=480), prediction of 5-year survival (MFI) based on BV alone yielded an AUC of 0.76 ("Cox demo"), whereas the model based on EV yielded an AUC of 0.76 ("Cox all"). The AUC of DeepMIL MoCo v2 based on WSI alone was 0.73 ("AI"). When BV was combined with DeepMIL MoCo v2, the AUC improved to 0.80 ("Cox demo + AI"). When EV was combined with DeepMIL MoCo v2, the AUC improved to 0.79 ("Cox all + AI").

[0152] Figure 6D shows the cross-validation results of recurrence risk in the ER+ / HER2- subpopulation using DeepMIL ImageNet and / or clinical risk scores. In the ER+ / HER2- population (n=1437), prediction of 5-year survival rate (MFI) based on BV alone resulted in an AUC of 0.77 ("Cox demo"), as shown in the "all" group in Figure 6D (left panel), and the model based on EV resulted in an AUC of 0.80 ("Cox all"), as also shown in Figure 6C (left panel). The AUC of DeepMIL ImageNet based on WSI alone was 0.76 ("AI"). Combining BV with DeepMIL ImageNet improved the AUC to 0.82 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.83 ("Cox all + AI").

[0153] As shown in the "n0" group in Figure 6D (middle panel), in the subpopulation of subjects who did not have lymph node metastases at diagnosis (N0; n = 957), prediction of 5-year survival (MFI) based on BV alone yielded an AUC of 0.72 ("Cox demo"), whereas the model based on EV yielded an AUC of 0.81 ("Cox all"), also shown in Figure 6C (middle panel). The AUC of DeepMIL ImageNet based on WSI alone was 0.77 ("AI"). Combining BV with DeepMIL ImageNet resulted in an AUC of 0.77 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.82 ("Cox all + AI").

[0154] As shown in the "n+" group in Figure 6D (right panel), in the subpopulation of subjects with lymph node metastasis at diagnosis (N+; n=480), prediction of 5-year survival (MFI) based on BV alone yielded an AUC of 0.76 ("Cox demo"), while the model based on EV yielded an AUC of 0.76 ("Cox all"), also shown in Figure 6B (right panel). The AUC of DeepMIL ImageNet based on WSI alone was 0.72 ("AI"). Combining BV with DeepMIL ImageNet improved the AUC to 0.80 ("Cox demo + AI"). Combining EV with DeepMIL ImageNet improved the AUC to 0.79 ("Cox all + AI").

[0155] In these results, the increase in AUC when combining machine learning and clinical models compared to the machine learning or clinical models alone (e.g., a higher AUC for "Cox Demo + AI" compared to "Cox Demo" or "AI", or a higher AUC for "Cox All + AI" compared to "Cox All" or "AI") indicates that using a combination of machine learning and clinical models to determine the final risk score improves the accuracy of the resulting model. This is an unexpected increase in AUC using a hybrid risk score (e.g., the average of the machine learning risk score and the clinical model risk score), and the resulting risk score shows a better AUC than both the AI ​​risk score and the clinical model risk score.

[0156] Multivariate analysis of the influence of clinical variables on the Cox model prediction of prognosis was performed (Figures 7-8). See Figures 7-8. 7A and 7B show the analysis of all clinical variables. 7A and 7B show the analysis of all clinical variables. 8A and 8B show the analysis of clinical variables excluding pN and pT. Shapley values ​​represent the influence of clinical variables on model prediction. Clinical variables are listed in order of their influence on model prediction.

[0157] The results show that the machine learning model applied to WSI can predict the risk of recurrence in subjects with early stage breast cancer. The results further demonstrate that by combining with a clinical model (e.g., a Cox model based on baseline clinical variables or extended clinical variables), the machine learning model can predict the risk of breast cancer recurrence with higher accuracy compared to the machine learning model or clinical model alone. Thus, the machine learning model combined with clinical variables is a promising tool for treatment decision-making for diseases or conditions, such as breast cancer, at low cost.

[0158] Further validation of the machine learning and clinical models, as well as the combination of risk scores derived therefrom, will be performed in large independent cohorts from different sources, including the UNICANCER cohort, e.g. PACS (n=3400), UNIRAD (n=4000) and CANTO (n=4000).

[0159] In some embodiments, the most predictive tiles are identified and analyzed in a machine learning model, such as DeepMIL with MoCo features, DeepMIL with ImageNet features, or Multi-Layer Perception with MoCo features. In some embodiments, the machine learning model and the most predictive tiles are used to discover new biomarkers and develop new models for recurrence prediction as a cost-effective, rapid, or easy alternative to techniques such as immunohistochemistry or molecular testing.

[0160] Example 2: Risk stratification of ER+ / HER2- subjects based on artificial intelligence (AI) risk scores

[0161] Subjects with ER+ / HER2- tumor status in the cohort of Example 1 were stratified into low-risk and high-risk groups using risk scores generated using DeepMIL trained with MoCo features based on WSI. The following clinical features were compared between high-risk and low-risk groups: MFI (Figure 9A), age (Figure 9B), tumor grade (Figure 9C), pN (Figure 9D), pT (Figure 9E), and Ki67 (Figure 9F). The machine learning model based on WSI stratified high and low recurrence risk groups with a hazard ratio (HR) of 5.25 (CI: 3.20-8.61, p<0.001) (Figure 9A). This result shows the ability of the machine learning model to accurately predict breast cancer recurrence risk based on WSI.

[0162] 10A-C and 11-13 show example tiles generated from WSI of H&E stained breast cancer tissue analyzed using DeepMIL with MoCo features in the analysis platform according to the present disclosure and association with risk of recurrence. FIG. 10A shows example tiles associated with high risk of recurrence. FIG. 10B shows example tiles associated with low risk of recurrence. FIG. 10C shows example locations of tiles and surrounding areas in WSI where high risk of recurrence is predicted. FIG. 10D shows example heatmap for prediction of tumor in WSI generated using trained multi-layer perception with MoCo features.

[0163] Figure 11A shows an exemplary tile predicting high risk of recurrence. Figure 11B shows an exemplary tile predicting low risk of recurrence. Figure 11C shows survival curves and hazard ratio assessments for subjects stratified into high, medium and low risk groups based on the AI ​​risk score.

[0164] FIG. 12 is an exemplary heat map showing regions within the WSI having characteristics associated with high and low risk of recurrence. Red dots correspond to exemplary regions predicting high risk of recurrence. Blue dots correspond to exemplary regions predicted to have low risk of recurrence. FIG. 13 shows exemplary tiles and surrounding regions within the WSI predicting high risk of recurrence.

[0165] Areas identified by the machine learning model as predictive of high or low risk of recurrence are further analyzed histologically to improve the model's ability to predict areas of recurrence based on histology and / or WSI.

[0166] Example 3: Prediction of molecular biomarkers for breast cancer using H&E whole slide images

[0167] The ability of the machine learning model to predict the status of molecular biomarkers ER, PR, Ki67, and HER2 in tumor tissue was analyzed in subjects from the study cohort of Example 1. H&E stained glass slides with breast tissue from subjects were digitized, pre-processed, and cut into small patches (tiles), and the tiles were fed into a deep learning ("DL") network (DeepMIL ImageNet) with biomarker information described in Example 1. A weighted average of the tile features was calculated to predict the positivity or negativity of each biomarker. Figure 14 shows an exemplary tile predicted as ER positive by DeepMIL ImageNet. The machine learning model predicted ER+ status with 88% AUC and 90% accuracy. Figure 15 shows an exemplary tile predicted as ER negative by DeepMIL ImageNet. The machine learning model predicted ER+ status with 88% AUC and 91% accuracy.

[0168] As shown in Figure 16, DeepMIL ImageNet predicted ER status in subject tumor tissue with an AUC of 0.90. For example, by adjusting parameter thresholds, it predicted ER status with 99% sensitivity and 36% specificity (i.e., it can identify 99% of ER+ tumors and exclude 36% of ER- tumors), 98% sensitivity and 52% specificity, or 95% sensitivity and 68% specificity.

[0169] As shown in Figure 17, DeepMIL ImageNet predicted PR status in subject tumor tissue with an AUC of 0.76. For example, by adjusting parameter thresholds, the model predicted PR status with 99% sensitivity and 20% specificity (i.e., it can identify 99% of PR+ tumors and exclude 20% of PR- tumors), 98% sensitivity and 24% specificity, or 95% sensitivity and 35% specificity.

[0170] As shown in Figure 18, DeepMIL ImageNet predicted Ki67 status in a subject's tumor tissue with an AUC of 0.85. For example, by adjusting parameter thresholds, the model predicted Ki67 status with 99% sensitivity and 9% specificity (i.e., it can identify 99% of Ki67+ tumors and exclude 9% of Ki67- tumors), 98% sensitivity and 26% specificity, or 95% sensitivity and 43% specificity.

[0171] As shown in Figure 19, DeepMIL ImageNet predicted HER2 status in a subject's tumor tissue with an AUC of 0.89. For example, by adjusting parameter thresholds, the model predicted HER2 status with 99% sensitivity and 11% specificity (i.e., it can identify 99% of HER2+ tumors and exclude 11% of HER2- tumors), 98% sensitivity and 29% specificity, or 95% sensitivity and 50% specificity.

[0172] The machine learning models of the present disclosure are further evaluated for their ability to predict other breast cancer biomarker status based on histological and morphological information available in histological slides, e.g., WSI. Based on the results of biomarker prediction by the machine learning models, novel biomarkers for breast cancer diagnosis and prognosis are identified.

[0173] Example 4: Computer System and Machine-Readable Medium

[0174] As shown in FIG. 20, an exemplary computer system 2000, which is one form of data processing system, includes a microprocessor 2005 and a bus 2003 coupled to a read only memory (ROM) 2007, a volatile RAM 2009, and a non-volatile memory 2013. The microprocessor 2005 may include one or more CPUs, GPUs, special purpose processors, and / or combinations thereof. The microprocessor 2005 may communicate with a cache 2004 and may retrieve and execute instructions from the memories 2007, 2009, and 2013 to perform the operations described above. The bus 2003 interconnects these various components and also interconnects these components 2005, 2007, 2009, and 2013 to peripheral devices such as a display controller and display device 2015, and input / output (I / O) devices 2011, which may be a mouse, keyboard, modem, network interface, printer, and other devices known in the art. Typically, input / output devices 2011 are coupled to the system through an input / output controller 2017. Volatile RAM (random access memory) 2009 is typically implemented as dynamic RAM (DRAM) and requires continuous power to refresh or maintain the data in the memory.

[0175] The non-volatile memory 2013 may be, for example, a magnetic hard drive or a magnetic optical drive or an optical drive or a DVD RAM or a flash memory or other type of memory system that retains data (e.g., large amounts of data) even after power is removed from the system. Typically, the non-volatile memory 2013 is also a random access memory, although this is not required. While FIG. 20 illustrates that the non-volatile memory 2013 is a local device directly coupled to the remaining components in the data processing system, it will be understood that the present invention may utilize non-volatile memory that is remote from the system, such as a network storage device coupled to the data processing system via a network interface, such as a modem, an Ethernet interface, or a wireless network. The bus 2003 may include one or more buses connected together through various bridges, controllers, and / or adapters, as is well known in the art.

[0176] The above discussion merely describes some exemplary embodiments of the present invention. Those skilled in the art will readily recognize from such discussion, the accompanying drawings, and the claims that various modifications can be made without departing from the spirit and scope of the present invention. Furthermore, where possible, any of the aspects disclosed herein can be combined with each other (e.g., features from one aspect can be added to or substituted for equivalent features of another aspect) or can be combined with features that are well known in the art, unless otherwise indicated by the context.

[0177] All citations to references, including, for example, citations to patents, published patent applications, and articles, are incorporated herein by reference in their entirety.

[0178] The section headings used herein are for organizational purposes only and are not to be construed as limiting in any way the subject matter described. [Explanation of symbols]

[0179] 101 Train a Machine Learning Model 103 Training a Clinical Model Receive 105 WSI and clinical attributes 107 Use trained machine learning models to generate risk scores 109 Generate risk scores using trained clinical models 111 Calculate the weighted average risk score from the machine learning risk score and the clinical risk score

Claims

1. 1. A computer-implemented method for predicting the likelihood that a subject with breast cancer will experience a recurrence after treatment, comprising: obtaining a digital image of a histological section of a breast cancer sample obtained from said subject; obtaining one or more subject attributes obtained from the subject; calculating an artificial intelligence (AI) risk score using a machine learning model, the machine learning model being trained by processing a plurality of training images to predict risk of recurrence; calculating a clinical risk score using a clinical model and one or more subject attributes, wherein the clinical model has been trained using one or more subject training attributes from different subjects; calculating a final risk score for the subject from the AI ​​risk score and the clinical risk score, the final risk score representing the likelihood that the subject will experience a relapse after treatment; 11. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the digital image is a whole slide image.

3. The computer-implemented method of claim 1 , wherein the histological section of the breast cancer sample is stained with a dye.

4. The computer-implemented method of claim 3 , wherein the stain is hematoxylin and eosin (H&E).

5. The computer-implemented method of claim 1 , wherein the breast cancer sample is obtained from the subject prior to treatment for the breast cancer.

6. The computer-implemented method of claim 1 , wherein the machine learning algorithm is a self-supervised learning algorithm.

7. The computer-implemented method of claim 6 , wherein the self-supervised learning algorithm is Momentum Contrast (MoCo) or Momentum Contrast v2.

8. The computer-implemented method of claim 6 , wherein the machine learning model comprises a multi-layer perceptual model.

9. The computer-implemented method of claim 1 , further comprising extracting a plurality of feature vectors from the digital image.

10. 10. The computer-implemented method of claim 9, wherein the extraction of the plurality of features is performed using a first convolutional neural network.

11. 11. The computer-implemented method of claim 10, wherein the first convolutional neural network is a ResNet50 neural network.

12. The computer-implemented method of claim 1 , wherein the method further comprises removing background segments from the image.

13. The computer-implemented method of claim 12 , wherein removing the background segments from the image is performed using a second convolutional neural network.

14. 14. The computer-implemented method of claim 13, wherein the second convolutional neural network is a semantic segmentation deep learning network.

15. The computer-implemented method of claim 1 , wherein the final risk score is calculated as a weighted average of the AI ​​risk score and the clinical risk score.

16. The computer-implemented method of claim 15 , wherein the weights in the weighted average are equal.

17. 10. The computer-implemented method of claim 1, wherein the machine learning model is trained using a plurality of training images, the plurality of training images comprising digital images of histological sections of breast cancer samples obtained from a plurality of control subjects.

18. The computer-implemented method of claim 17 , wherein the plurality of training images includes images lacking local annotations.

19. 20. The computer-implemented method of claim 17, wherein the plurality of training images comprises images associated with one or more global labels indicative of one or more disease characteristics of the control subject from which the sample was obtained.

20. 20. The computer-implemented method of claim 19, wherein the disease characteristic is time to breast cancer recurrence.

21. 20. The computer-implemented method of claim 19, wherein the one or more disease characteristics comprise one or more of the subject's age at the time of surgery, menopausal status, tumor stage, tumor size, number of positive lymph nodes (N+), number of nodes, type of surgery, and / or type of treatment, or a combination thereof.

22. 20. The computer-implemented method of claim 19, wherein the disease features comprise one or more of estrogen receptor (ER) status, progesterone receptor (PR) status, HER2 status, tumor grade, Ki67 expression, histology, and / or the presence or absence of one or more mutations in the BRCA or TP53 genes, or a combination thereof.

23. 20. The computer-implemented method of claim 19, wherein the method comprises obtaining one or more disease features of the subject and applying a machine learning model to both the extracted features and the disease features of the subject, wherein one or more of the disease features of the subject are the same as one or more of the disease features represented in the global labels associated with training images.

24. 10. The computer-implemented method of claim 1, wherein the final risk score represents the likelihood that the subject will experience a recurrence within five years from the date the breast cancer sample was obtained from the subject.

25. The computer-implemented method of claim 1 , wherein the machine learning model is a deep multiple instance learning (DeepMIL) model.

26. The computer-implemented method of claim 1 , wherein the machine learning model is a Weldon model.

27. 1. A machine-readable medium having executable instructions for causing one or more processing units to perform a method for predicting the likelihood that a subject with breast cancer will experience a recurrence after treatment, the method comprising: The method comprises: obtaining a digital image of a histological section of a breast cancer sample obtained from said subject; obtaining one or more subject attributes obtained from the subject; calculating an artificial intelligence (AI) risk score using a machine learning model, the machine learning model being trained by processing a plurality of training images to predict risk of recurrence; calculating a clinical risk score using a clinical model and one or more subject attributes, wherein the clinical model has been trained using one or more subject training attributes from different subjects; calculating a final risk score for the subject from the AI ​​risk score and the clinical risk score, the final risk score representing the likelihood that the subject will experience a relapse after treatment; 1. A machine-readable medium comprising:

Images