System and method for processing electronic images for disease diagnosis and assessment.

Abstract

To provide systems and methods for determining cancer detection results on the basis of digital pathology images.SOLUTION: Systems and methods includes: receiving a digital image corresponding to a target specimen associated with a pathology category, the digital image being an image of tissue specimen; determining a detection machine learning model, the detection machine learning model being generated by processing a plurality of training images to output cancer qualification and further cancer quantification if the cancer qualification is a confirmed cancer qualification; providing the digital image as an input to the detection machine learning model; receiving one of a pathological complete response (pCR) cancer qualification and a confirmed cancer quantification as an output from the detection machine learning model; and outputting the pCR cancer qualification or the confirmed cancer quantification.SELECTED DRAWING: None

Description

Technical Field

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 957,523, filed on January 6, 2020, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] Various embodiments of the present disclosure generally relate to determining the presence or absence of a disease, such as cancer cells. More specifically, certain embodiments of the present disclosure relate to determining at least one of pathologic complete response (pCR) or minimal residual disease (MRD) based on cells within a whole slide image (WSI).

Background Art

[0003] Pathologic complete response (pCR) may refer to the absence of residual invasion and intraepithelial cancer cells on histological microscopic slides of resected tissue samples. pCR can be used as a surrogate endpoint for determining whether a patient is responding to therapy (e.g., therapies related to breast cancer, prostate cancer, bladder cancer, colorectal cancer, etc.). For example, pCR for breast cancer can be defined as the absence of all signs of invasive cancer in the breast tissue and lymph nodes removed during post-treatment surgery.

[0004] Minimal residual disease (MRD) may refer to microscopic diseases such as ultra-microscopic diseases that remain latent within a patient but may ultimately lead to recurrence. In cancer treatment, MRD can provide information regarding whether the treatment has removed the cancer or whether traces remain. Currently, pCR / MRD is determined manually by a pathologist checking tissue samples under a microscope to examine whether cancer cells still remain or whether all cancer cells have been removed. This detection task can be subjective and can be difficult due to various definitions of pCR / MRD and treatment effects that can change the morphology of cancerous and benign tissues due to neoadjuvant therapy. The level of subjectivity and difficulty can increase when treatment damage is present.

[0005] The general descriptions above and the detailed descriptions below are illustrative and explanatory only and do not limit the disclosure. The background information provided herein is generally for the purpose of presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not the prior art of the claims herein and their inclusion in this section does not constitute prior art or an indication of prior art. [Overview of the Initiative] [Means for solving the problem]

[0006] According to certain aspects of this disclosure, a system and method for determining cancer detection results based on digital pathology images are disclosed.

[0007] A method for outputting cancer detection results includes receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample; determining a detection machine learning model, which is generated by processing multiple training images, outputs a cancer qualitative, and if the cancer qualitative is a confirmed cancer qualitative, further outputs a cancer quantification; providing the digital image as input to the detection machine learning model; receiving one of either a pathological complete response (pCR) cancer qualitative or a confirmed cancer quantification as output from the detection machine learning model; and outputting the pCR cancer qualitative or confirmed cancer quantification.

[0008] A system for outputting cancer detection results includes a memory for storing instructions, a processor for executing instructions to carry out a process which includes receiving a digital image corresponding to a target sample associated with a pathology category, the digital image being an image of a tissue sample, determining a detection machine learning model, the detection machine learning model being generated by processing multiple training images, outputting a cancer qualitative, and if the cancer qualitative is a confirmed cancer qualitative, further outputting a cancer quantification, providing the digital image as input to the detection machine learning model, receiving one of either a pathological complete response (pCR) cancer qualitative or a confirmed cancer quantification as output from the detection machine learning model, and outputting the pCR cancer qualitative or confirmed cancer quantification.

[0009] A non-transient computer-readable medium that stores instructions, when executed by a processor, causing the processor to carry out a method for generating a specialized machine learning model, the method comprising: receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample; determining a detection machine learning model, which is generated by processing multiple training images, and outputs a cancer qualitative, and if the cancer qualitative is a confirmed cancer qualitative, further outputs a cancer quantification; providing a digital image as input to the detection machine learning model; receiving one of either a pathological complete response (pCR) cancer qualitative or a confirmed cancer quantification as output from the detection machine learning model; and outputting the pCR cancer qualitative or confirmed cancer quantification. The present invention provides, for example, the following items: (Item 1) A computer implementation method for processing electronic images, wherein the method is Receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample. The method involves determining a detection machine learning model, which is generated by processing multiple training images, outputs a qualitative cancer assessment, and, if the qualitative cancer assessment is confirmed to be a qualitative cancer assessment, further outputs a quantitative cancer assessment. The digital image is provided as input to the aforementioned detection machine learning model, The output from the aforementioned detection machine learning model is to receive either a qualitative assessment of pathological complete response (pCR) cancer or a quantitative assessment of confirmed cancer. Outputting the pCR cancer qualitative analysis or the confirmed cancer quantitative analysis. Computer implementation methods, including those mentioned above. (Item 2) Receiving confirmed cancer quantification includes receiving minimal residual disease (MRD) cancer quantification, as described in item 1. (Item 3) MRD cancer qualitative analysis is performed using the computer implementation method described in item 2, which is specific to the protocol. (Item 4) MRD cancer qualitative analysis is a computer implementation method described in item 2, corresponding to the number of cancer cells below the MRD threshold. (Item 5) The aforementioned pCR cancer qualitative analysis corresponds to the digital image having zero detectable cancer cells. The computer implementation method described in item 1. (Item 6) The computer implementation method according to item 1, wherein the plurality of training images include images that have a therapeutic effect. (Item 7) The computer implementation method described in item 1, comprising the detection machine learning model and the treatment effect machine learning model. (Item 8) The computer implementation method according to item 7, wherein the therapeutic effect machine learning model is initialized by using a trained machine learning model trained on the plurality of training images, and the plurality of training images exclude images that show a therapeutic effect. (Item 9) The computer implementation method described in Item 1, wherein the digital image is from the pathology category, and the pathology category is selected from one or more of the following: histology, cytology, frozen section, immunohistochemistry (IHC), immunofluorescence, hematoxylin and eosin (H&E), hematoxylin only, molecular pathology, or 3D imaging. (Item 10) A system for processing electronic images, wherein the system is At least one memory for storing instructions, At least one processor, the at least one processor executes the instruction, Receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample. The method involves determining a detection machine learning model, which is generated by processing multiple training images, outputs a qualitative cancer assessment, and, if the qualitative cancer assessment is confirmed to be a qualitative cancer assessment, further outputs a quantitative cancer assessment. The digital image is provided as input to the aforementioned detection machine learning model, The output from the aforementioned detection machine learning model is to receive either a pathological complete response (pCR) cancer qualitative assessment or a confirmed cancer qualitative assessment. Outputting the pCR cancer qualitative analysis or the confirmed cancer qualitative analysis, At least one processor and performs operations including A system that includes these features. (Item 11) The system described in item 10 further includes receiving confirmed cancer qualitative assessments and receiving minimal residual disease (MRD) cancer quantifications. (Item 12) MRD cancer qualitative analysis is protocol-specific and follows the system described in item 11. (Item 13) The confirmed cancer qualitative analysis corresponds to the system described in item 10, which detects a threshold number of cancer cells. (Item 14) The system according to item 10, wherein the plurality of training images include images with treatment effects. (Item 15) The system according to item 10, wherein the detection machine learning model includes a treatment effect machine learning model. (Item 16) The system according to item 15, wherein the treatment effect machine learning model is initialized by using a trained machine learning model trained based on the plurality of training images, and the plurality of training images exclude images with treatment effects. (Item 17) The digital image is from a pathology category, and the pathology category is selected from one or more of histology, cytology, frozen section, immunohistochemistry (IHC), immunofluorescence, hematoxylin and eosin (H&E), hematoxylin only, molecular pathology, or 3D imaging. The system according to item 10. (Item 18) A non-transitory computer-readable medium, the non-transitory computer-readable medium stores instructions, and when the instructions are executed by a processor, the processor is caused to perform operations for processing an electronic image, and the operations include: Receiving a digital image corresponding to a target sample associated with a pathology category, the digital image being an image of a tissue sample; Determining a detection machine learning model, the detection machine learning model being generated by processing a plurality of training images, outputting cancer qualification, and when the cancer qualification is a confirmed cancer qualification, further outputting cancer quantification; Providing the digital image as an input to the detection machine learning model; Receiving, as an output from the detection machine learning model, one of a pathologic complete response (pCR) cancer qualification or a confirmed cancer quantification; Outputting the pCR cancer qualification or the confirmed cancer quantification; A non-transitory computer-readable medium including the above. (Item 19) The non-transitory computer-readable medium of item 18, wherein receiving the confirmed cancer quantification includes receiving the minimal residual disease (MRD) cancer quantification. (Item 20) The non-transitory computer-readable medium of item 18, wherein the detection machine learning model comprises a treatment effect machine learning model.

Brief Description of the Drawings

[0010] The accompanying drawings, which are incorporated herein and form a part hereof, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0011] [Figure 1A] FIG. 1A illustrates an exemplary block diagram of a system and network for implementing a detection tool with a digital image according to an exemplary embodiment of the present disclosure.

[0012] [Figure 1B] FIG. 1B illustrates an exemplary block diagram of a machine learning module according to an exemplary embodiment of the present disclosure.

[0013] [Figure 2] FIG. 2 is a flowchart illustrating an exemplary method for using a detection machine learning model according to an exemplary embodiment of the present disclosure.

[0014] [Figure 3] FIG. 3 illustrates an exemplary block diagram of a training module according to an exemplary embodiment of the present disclosure. <{

[0015] [Figure 4] FIG. 4 illustrates a schematic diagram for detecting cancer cells using a detection module according to an exemplary embodiment of the present disclosure.

[0016] [Figure 5] FIG. 5 is a flowchart of an exemplary embodiment of a detection implementation according to an exemplary embodiment of the present disclosure.

[0017] [Figure 6] Figure 6 is a schematic diagram of experimental results using a detection model according to an exemplary embodiment of the present disclosure.

[0018] [Figure 7] Figure 7 illustrates an exemplary system capable of performing the techniques presented herein. [Modes for carrying out the invention]

[0019] Description of the Embodiment Herein, exemplary embodiments of the present disclosure are referenced in detail, and such embodiments are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0020] The systems, devices, and methods disclosed herein are described in detail as examples with reference to the drawings. The examples discussed herein are merely examples and are provided to aid in the description of the apparatus, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be considered essential for any specific implementation of any of these devices, systems, or methods unless specifically designated as essential.

[0021] Furthermore, with respect to any method described, whether or not the method is described in conjunction with a flowchart, unless otherwise specified or required by the context, any explicit or implicit ordering of the steps performed in the execution of the method does not imply that those steps must be performed in the order presented, but rather that they may be performed in a different order or in parallel.

[0022] As used herein, the term “exemplary” is used in the sense of “example” rather than “ideal.” Furthermore, the terms “a” and “an” in this specification do not indicate a limit on quantity, but rather indicate the existence of one or more of the items referenced. In the following discussion, relative terms such as “about,” “substantially,” and “approximately” are used to indicate the stated value, number, or other possible variation of ±10% or less.

[0023] Pathology refers to the study of disease. More specifically, pathology refers to the conduct of tests and analyses used to diagnose disease. For example, a tissue sample may be placed on a slide for visualization under a microscope by a pathologist (e.g., a physician who is an expert in analyzing tissue samples and determining whether any abnormalities are present). That is, a pathological specimen may be cut into multiple sections, stained, and prepared as slides for examination and diagnosis by the pathologist. When uncertainty in diagnosis is found on the slide, the pathologist may order additional cutting levels, staining, or other tests to gather more information from the tissue. The technician may then create a new slide, which may contain additional information for the pathologist to use when making a diagnosis. This process of creating additional slides can be time-consuming, not only because it may involve taking a mass of tissue, cutting it, preparing a new slide, and then staining the slide, but also because it may be combined for multiple instructions. This can significantly delay the final diagnosis given by the pathologist. In addition, even after a delay, there is still no guarantee that the new slides will contain enough information to provide a diagnosis.

[0024] Pathologists may evaluate cancer and other disease pathology slides for cancer detection. This disclosure presents automated methods for identifying cancer cells, performing cancer qualitative analysis, and, where applicable, cancer quantification. In particular, this disclosure describes various exemplary AI tools that can be integrated into workflows to facilitate and improve the work of pathologists.

[0025] For example, a computer may be used to analyze images of a tissue sample to rapidly identify whether the tissue sample contains one or more cancer cells in order to determine the qualitative aspects of cancer (e.g., the presence or absence of cancer) and the quantitative aspects of cancer (e.g., the extent of cancer present). Thus, the process of examining stained slides and tests may be performed automatically before, instead of, or in conjunction with, examination by a pathologist. When paired with automated slide examination and cancer detection, this can provide a fully automated slide preparation and evaluation pipeline.

[0026] Such automation would have the benefit of at least (1) minimizing the amount of time wasted by pathologists determining slide findings by manually detecting cancer cells, (2) minimizing the (average total) time from sample acquisition to diagnosis by avoiding the additional time spent on manual analysis or questionable slides, (3) reducing the amount of repeated tissue evaluations based on missed or difficult-to-detect tissue areas, (4) reducing the cost of repeated biopsies and pathological examinations by considering therapeutic effects, (5) eliminating or mitigating the need for a second or subsequent pathological diagnostic examination, (6) reducing the probability of misdiagnosis, (7) increasing the probability of correct diagnosis, and / or (8) identifying or matching the correct properties (e.g., pCR, MRD, etc.) of digital pathology images.

[0027] The process of using computers to assist pathologists is called computational pathology. Computational methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but are not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. By using computational pathology, lives can be saved by helping pathologists improve the accuracy, reliability, efficiency, and accessibility of their diagnoses. For example, computational pathology may be used to assist in detecting slides suspected of being cancerous, thereby allowing pathologists to check and confirm their initial assessment before giving a final diagnosis.

[0028] Histopathology refers to the study of specimens placed on slides. For example, digital pathology images may consist of digitized images of microscope slides containing specimens (e.g., smears). One method that a pathologist may use to analyze images on slides is to identify nuclei and classify whether they are normal (e.g., benign) or abnormal (e.g., malignant). To assist pathologists in identifying and classifying nuclei, histological staining may be used to visualize cells. Many dye-based staining systems have been developed, including the periodate Schiff reaction, Masson's trichrome, Nissl and methylene blue, and hematoxylin and eosin (H&E). For medical diagnosis, H&E is a widely used dye-based method, where hematoxylin stains cell nuclei blue, eosin stains the cytoplasm and extracellular matrix pink, and other tissue areas exhibit variations of these colors. However, in many cases, histological preparation by H&E staining does not provide sufficient information for pathologists to visually identify biomarkers that may aid in diagnosis or guide treatment. In this situation, techniques such as immunohistochemistry (IHC), immunofluorescence, in-situ hybridization (ISH), or fluorescent in-situ hybridization (FISH) may be used. IHC and immunofluorescence, for example, involve the use of antibodies that enable the visual detection of cells that bind to specific antigens in the tissue and express specific proteins of interest, which may reveal biomarkers that are not reliably identifiable to a trained pathologist based on analysis of H&E stained slides. ISH and FISH may be employed to assess the number of gene copies or the abundance of specific RNA molecules, depending on the type of probe employed (e.g., DNA probes for gene copy number and RNA probes for assessing RNA expression). If these methods also do not provide sufficient information to detect some biomarkers, genetic testing of the tissue may be used to confirm the presence of biomarkers (e.g., overexpression of specific proteins or gene products in tumors, amplification of a given gene in cancer).

[0029] Digitized images may be prepared to show stained microscope slides, which may allow pathologists to manually visually inspect the images on the slides and estimate the number of stained abnormal cells in the images. However, this process can be time-consuming and lead to errors in identifying abnormalities, as some abnormalities are difficult to detect. Computational processes using machine learning models and devices may be used to assist pathologists in detecting abnormalities that may otherwise be difficult to detect. For example, AI may be used to detect cancer cells from prominent areas in digital images of tissue stained using H&E and other dye-based methods (for example, because they may be distinguishable from non-cancerous cells). Images of tissue may be whole-slide images (WSI), images of tissue cores in a microarray, or selected areas of interest in a tissue section. When staining methods such as H&E are used, these cancer cells may be difficult for humans to visually detect or quantify without the assistance of additional testing. Using AI to detect these cancer cells from digital images of tissue has the potential to improve patient care while also being faster and less expensive.

[0030] As described above, the computational pathology processes and devices of this disclosure provide an integrated platform that enables a fully automated process, including data acquisition, processing, and viewing of digital pathology images, via a web browser or other user interface, while integrating with a laboratory information system (LIS). Furthermore, clinical information may be aggregated using cloud-based data analytics of patient data. The data may originate from hospitals, clinics, field researchers, etc., and may be analyzed by machine learning, computer vision, natural language processing, and / or statistical algorithms to enable real-time monitoring and prediction of health patterns at multiple geographical specificity levels.

[0031] Implementations of the disclosed subject matter include systems and methods for using a detection machine learning model to determine the presence or absence of cancer cells in a WSI. The detection machine learning model may also be used to determine a cancer qualitative assessment. The cancer qualitative assessment may include an indication of whether cells represented in a digital image of a tissue sample are cancer cells or whether no cancer cells are identified in the digital image. According to one implementation, the cancer qualitative assessment may also include the type of cancer (e.g., breast, prostate, bladder, colon, etc.). If the cancer qualitative assessment is a confirmed cancer qualitative assessment, a cancer quantification may also be output by the detection machine learning model. The cancer quantification may indicate the number, proportion, or extent of cancer cells identified from the digital image and may be a minimal residual disease (MRD) designation based on established MRD criteria (e.g., one cell per million or less). If the cancer qualitative assessment output by the detection machine learning model does not indicate any cancer cells, a pathological complete response (pCR) cancer qualitative assessment may be output.

[0032] The detection machine learning model may be trained based on supervised, semi-supervised, weakly supervised, or unsupervised training, including, but not limited to, multi-instance learning. The training images may be from the same pathology category as the individual digital images input to the detection machine learning model. According to one implementation, multiple different training images from multiple pathology categories may be used to train the detection machine learning model across pathology categories. According to this implementation, the input to the detection machine learning model may include the pathology category of the digital image. The pathology category may include, but not limited to, histology, cytology, frozen section, immunohistochemistry (IHC), immunofluorescence (IF), hematoxylin and eosin (H&E), hematoxylin only, molecular pathology, 3D imaging, or equivalents. The detection machine learning model may be trained to detect cancer cells based on training images having tagged cancer cells, for example. The detection machine learning model may adjust the weighting in one or more layers to identify regions that are likely to contain cancer cells, based on known or determined cancer types, and may further adjust the weighting in one or more layers based on whether it identifies or does not find cancer cells within those regions.

[0033] According to one implementation, a detection machine learning model may be trained using training digital images depicting tissues exhibiting a therapeutic effect. Therapies that may produce a therapeutic effect include, but are not limited to, hormone therapy (androgen suppression therapy (ADT), nonsteroidal antiandrogen drugs (NSAAs)), radiotherapy, chemotherapy, or neoadjuvant therapies such as equivalents. Such therapies can cause treatment damage, altering the morphology of cancerous and benign cells, and thus making detection-based assessments more difficult than such assessments without a therapeutic effect. A therapeutic effect may be the result of a treatment applied to a patient from whom a tissue sample corresponding to a digital image is taken. Therapies can often alter the morphology of patient tissue, which is commonly known as a “therapeutic effect,” and can often make analyses determining cancer cells different from analyses of tissues that do not exhibit a therapeutic effect. Training digital images depicting tissues exhibiting a therapeutic effect may or may not be tagged as digital images corresponding to tissues with a therapeutic effect. A therapeutic effect machine learning model may be trained on images exhibiting a therapeutic effect and may be part of a detection machine learning model. By utilizing a treatment detection machine learning model, the qualitative and potential quantification of cancer by the detection machine learning model may be indicated by the output of the treatment detection machine learning model, and may provide an indication of the success or failure of a given treatment. The treatment effect machine learning model may be initialized using a base detection machine learning model (i.e., a trained machine learning model trained on multiple training images without treatment effect). Similarly, the pCR and / or MRD detection components of the detection machine learning model may be initialized using a base detection machine learning model.

[0034] Notifications, visual indicators, and / or reports may occur based on the output of a detection machine learning model. Reports may be based on individual digitized images, or on multiple digitized images, either over a given period of time or generally retrospectively.

[0035] The systems disclosed herein may be implemented locally (e.g., on-premises) and / or remotely (e.g., cloud-based). The systems may or may not have user interfaces and workflows directly accessible to pathologists (e.g., downstream oncologists may flag them based on qualitative or quantitative cancer assessment, etc.). Therefore, the implementations disclosed herein may be used as standalone operations or within digital workflows.

[0036] The disclosed subject matter is described as being implemented based on oncological applications, but they may also be used for other forms of cell detection (e.g., infectious cells, cystic fibrosis cells, sickle cell anemia, etc.). In addition to providing cancer detection benefits, the described implementations may also be used to train healthcare professionals (e.g., slide technicians, pathologists, etc.) to practice cell qualitative or quantification and / or diagnostic decisions while reducing the risk of harm to patients.

[0037] Figure 1A illustrates a block diagram of a system and network for determining sample or image property information relating to digital pathology images using machine learning, according to an exemplary embodiment of the present disclosure. As further disclosed herein, the system and network of Figure 1A may include a machine learning module 100 with a detection tool 101 for providing a cancer qualitative output and potentially a cancer quantitative output.

[0038] Specifically, Figure 1A illustrates an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or a physician's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to one implementation, the electronic network 120 may also be connected to a server system 110, which may include a processing device configured to implement a machine learning module 100, as described in an exemplary embodiment of the subject matter disclosed.

[0039] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may create or otherwise acquire images of one or more categories of pathological specimens, including patient cytological specimens, histopathological specimens, slides of cytological specimens, histology, immunohistochemistry, immunofluorescence, digitized images of slides of histopathological specimens, or any combination thereof. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsies, or cytological information. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may transmit digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. The server system 110 may also include processing devices for processing the images and data stored in the storage devices 109. The server system 110 may further include one or more machine learning tools or capabilities via the machine learning module 100. For example, the processing device may include a detection tool 101, as shown as the machine learning module 100 in one embodiment. The detection tool 101 may include one or more other components, such as detection machine learning models and therapeutic effect machine learning models, quantification modules, or equivalents, as disclosed herein. Alternatively, or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0040] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to examine slide images. In a hospital setting, tissue type information may be stored within the laboratory information system 125.

[0041] Figure 1B illustrates an exemplary block diagram of a machine learning module 100 for determining tissue sample properties or image property information related to digital pathology images using machine learning.

[0042] Specifically, Figure 1B depicts the components of a machine learning module 100 according to one embodiment. For example, the machine learning module 100 may include a detection tool 101, a data acquisition tool 102, a slide acquisition tool 103, a slide scanner 104, a slide manager 105, a storage device 106, and a visual application tool 108. For clarity, the machine learning module 100 shown in Figures 1A and 1B is a previously trained and generated machine learning model (e.g., a detection machine learning model which may include a therapeutic effect machine learning model). Additional disclosures are provided herein for training and generating different types of machine learning models which may be used as machine learning module 100.

[0043] The detection tool 101 refers to a process and system for determining cancer qualitative and, if confirmed cancer qualitative exists, determining cancer quantification. Cancer qualitative may be confirmed cancer qualitative, pCR cancer quantification (e.g., no cancer detected), or equivalent. Confirmed cancer qualitative may indicate that one or more cancer cells were detected in the digital image of the tissue sample. Cancer quantification may indicate the number of cancer cells detected, the ratio of cancer cells to non-cancerous cells, or the extent of cancer. A subset of cancer quantification is MRD cancer qualitative, which may indicate whether the number of cancer cells is below the MRD threshold. The MRD threshold may be protocol-specific, cancer-type-specific, site-specific, pathologist-specific, or equivalent. The detection tool 101 may include multiple machine learning models, or may load one machine learning model at a time. For example, the detection tool 101 may include a therapeutic effect machine learning model that is different from or can be trained on additional training datasets than the detection machine learning models disclosed herein.

[0044] The data acquisition tool 102 refers to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices of the machine learning module 100, which are used to characterize and process digital pathology images, according to exemplary embodiments.

[0045] The slide acquisition tool 103 refers to a process and system for scanning pathological images and converting them into a digital format, according to an exemplary embodiment. The slides may be scanned using a slide scanner 104, and the slide manager 105 may process the images on the slides into digitized pathological images and store the digitized images in a storage device 106.

[0046] The Visualization Application Tool 108 refers to a process and system for providing a user (e.g., a pathologist) with characterization or image property information relating to a digital pathology image, according to exemplary embodiments. The information may be provided through various output interfaces (e.g., a screen, monitor, storage device, and / or web browser, etc.). In an example, the Visualization Application Tool 108 may apply an overlay layer over the digital pathology image, the overlay layer may highlight areas of significant consideration. The overlay layer may be the output of the detection tool 101 of the machine learning module 100, or may be based thereon. As further discussed herein, the Visualization Application Tool 108 may be used to indicate specific areas in a digital image corresponding to cancer cells, or areas where cancer cells are more likely to be present.

[0047] Each detection tool 101 and its components may transmit and / or receive digitized slide images and / or patient information via the network 120 to and from the server system 110, physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125. Furthermore, the server system 110 may include a storage device for storing images and data received from at least one of the detection tool 101, data acquisition tool 102, slide acquisition tool 103, slide scanner 104, slide manager 105, and viewing application tool 108. The server system 110 may also include a processing device for processing the images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively, or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be implemented on a local processing device (e.g., a laptop).

[0048] The detection tool 101 may provide the output of the machine learning module 100 (e.g., cancer qualitative analysis, cancer quantification, pCR qualitative analysis, MRD qualitative analysis, etc.). In an example, the slide acquisition tool 103 and the data acquisition tool 102 may receive input to the machine learning module 100, and the detection tool 101 may identify cancer cells in the slide based on the data and output an image highlighting the cancer cells or associated area via the visual application tool 108.

[0049] Any of the above devices, tools, and modules may be located on a device that can be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0050] Figure 2 shows a flowchart 200 for outputting cancer qualitative and quantification based on digital images, according to an exemplary embodiment of the disclosed subject matter. In 202 of Figure 2, a digital image corresponding to a target sample associated with a pathology category may be received. The digital image may be a digital pathology image captured using the slide acquisition tool 103 of Figure 1B. In 204, a detection machine learning model may be determined (for example, in machine learning module 100). The detection machine learning model may be trained by processing multiple training images that are from the same pathology category as the digital image received in 202. The pathology category may include, but is not limited to, histology, cytology, frozen section, or immunohistochemistry. According to one implementation, the detection machine learning model may be trained using training images from multiple different pathology categories. The digital image corresponding to a target sample, such as the one received in 202, may also be received along with its pathology category, which may be provided as input to the detection machine learning model. In 206, the digital image from 202 may be provided to the detection machine learning model as input to the model. One or more other attributes may also be provided as input to the detection machine learning model. One or more other attributes may include, but are not limited to, pathology category, slide type, glass type, tissue type, tissue region, chemical used, amount of staining, time applied, scanning device type, date, or equivalent. In 208, the detection machine learning model may output cancer qualitative (e.g., pCR cancer qualitative) or confirmed cancer quantification (e.g., amount of cancer). According to one implementation, confirmed cancer qualitative (e.g., presence of cancer) may be received in addition to or instead of confirmed cancer quantification. Cancer qualitative may determine whether cancer quantification occurs. For example, if confirmed cancer qualitative indicates the presence of cancer cells, cancer quantification (e.g., number of cancer cells) may be determined. Qualitative indicating the absence of cancer cells may not require cancer quantification unless the cancer qualitative threshold exceeds zero cancer cells.In 210, cancer qualitative analysis (e.g., pCR cancer qualitative analysis, confirmed cancer qualitative analysis, etc.) and, where applicable, cancer quantification may be output as data signals, reports, notifications, alerts, visual outputs (e.g., via a visual application tool 108), or equivalents.

[0051] Conventional techniques for detecting cancer cells, such as those found through manual pathological examination, can be subjective and difficult due to the complexity of the tissue, the various attributes of qualitative and quantitative assessment (e.g., pCR / MRD), the definition of pCR / MRD, and / or the therapeutic effects that can alter the morphology of cancerous and benign tissues, for example, due to neoadjuvant therapy. Techniques and systems based on the process described in flowchart 200 of Figure 2 enable robust, objective, and more accurate calculation and assessment of cancer qualitative or quantitative assessment, including pCR and MRD. MRD and pCR designations may be used as surrogate endpoints for assessment of disease-free survival and overall survival to accelerate clinical trials (e.g., with respect to breast cancer). The automated nature of the process described in Figure 2 can facilitate results and / or treatment approval.

[0052] As shown in Figure 2, a digital image corresponding to a target sample associated with a pathology category may be received at 202. The target sample may be a biopsy or otherwise obtained tissue sample taken from a patient. The target sample may be taken during a surgical procedure in which a portion of the patient's tissue is taken from the patient's body for analysis. The target sample may be a portion or subset of the total amount of tissue extracted from the patient, so that multiple sample slides may arise from tissue extracted from a single procedure.

[0053] The target sample may be associated with at least one pathology category or technique, such as histology, cytology, frozen section, H&E, hematoxylin only, IHC, molecular pathology, 3D imaging, or equivalent, as disclosed herein. Depending on the implementation, the pathology category and other image information about the digital image or target sample may also be received. The image information may include, but is not limited to, slide type, glass type, tissue type, tissue region, chemicals used, and staining amount.

[0054] In 204, a detection machine learning model may be determined. The detection machine learning model may be trained and generated in machine learning module 100, or may be trained and generated externally and received in machine learning module 100. The detection machine learning model may be trained by processing multiple training images, at least some of which are from the same pathology category as the digital images received in 202. The pathology category may include, but is not limited to, histology, cytology, frozen section, H&E, hematoxylin only, IHC, molecular pathology, 3D imaging, or equivalents. The detection machine learning model may be instantiated using one or more of the following: deep learning, including, but not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), fully convolutional networks (FCNs), and recurrent neural networks (RCNs); probabilistic models, including, but not limited to, Bayesian networks and graphical models; and / or discriminative models, including, but not limited to, decision forests and maximum margin methods; or equivalents. These models may be trained using multi-instance learning in an end-to-end supervised, semi-supervised, weakly supervised, or unsupervised manner. The detection machine learning model or a separate machine learning model's quantification component may also use machine learning techniques to generate outputs relating to the quantification of cancer in digital images (e.g., the number of cancer cells in the image). The quantification component may be trained using, but is not limited to, deep learning, CNNs, multi-instance learning, or equivalents, or a combination thereof.

[0055] A detection machine learning model may be trained to output cancer qualitative and quantification, as disclosed herein. The cancer qualitative output may relate to one or more of several different cancer types. The detection machine learning model may be trained using images from one or more of several different cancer types. For example, the training images may include images related to breast cancer, prostate cancer, and lung cancer. Thus, the generated detection machine learning model may receive a digital image in Figure 202 and qualify the image as representing tissue including cancer cells, the number of cancer cells, and / or the type of cancer cells. The detection machine learning model may output cancer qualitative and / or quantification based on weights and / or layers trained during its training process. Based on weights and / or layers, the detection machine learning model may identify regions of the digital image that can be used more strongly as evidence of the presence or absence of cancer and, further, the extent of cancer. The model may then evaluate some or all of those regions and determine the presence, absence, and / or extent of cancer cells based on training with the images that provide them. Feedback (e.g., pathologist review, correction, adjustment, etc.) may further train the detection machine learning model during the model's operation.

[0056] In 204, to generate a detection machine learning model, a training dataset containing numerous digital pathology images of pathology specimens (e.g., histology, cytology, frozen sections, H&E, hematoxylin only, IHC, molecular pathology, 3D imaging, etc.) may be applied. The digital pathology images may be digital images generated based on physical biopsy samples, as disclosed herein, or may be images algorithmically generated to replicate tissue specimens (e.g., human, animal, etc.) by, for example, a rendering system or generative adversarial model. Image or specimen association information (e.g., slide type, glass type, tissue type, tissue region, chemical used, staining amount, time applied, scanning device type, date, etc.) may also be received as part of the training dataset. In addition, as part of training the detection machine learning model, each image may be paired with output information about known or assumed cancer qualitative and, where applicable, cancer quantification. Such output information may include indications of cancer presence or absence, cancer type, and / or cancer severity. A detection machine learning model may be trained from multiple such training images and associated information, such that the detection machine learning model is trained by modifying one or more weights based on the qualitative and quantitative information associated with each training image. Supervised training is provided as an example, but it should be understood that training of a detection machine learning model can be supervised, semi-supervised, weakly supervised, or unsupervised.

[0057] A training dataset containing digital pathology images, image or sample association information, and / or output information may be generated and / or provided by one or more of the following: System 110, Physician Server 121, Hospital Server 122, Clinical Trial Server 123, Research Laboratory Server 124, and / or Laboratory Information System 125. Images used for training may originate from real-world sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various dyes such as H&E, hematoxylin only, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices such as micro-CT.

[0058] The detection machine learning model may optionally arise from applying digital pathology images with associated information that is paired with output information as applied by the machine learning algorithm. The machine learning algorithm may take pathology samples, associated information, and output information (e.g., cancer qualitative, cancer quantification, etc.) as input and implement training using one or more techniques. For example, the detection machine learning model may be trained in one or more deep learning algorithms such as, but is not limited to, DNNs, CNNs, FCNs, RCNs, CNNs with multiple instance learning or multi-label multiple instance learning, recurrent neural networks (RNNs), long short-term memory RNNs (LSTMs), gated recurrent unit RNNs (GRUs), graph convolutional networks, or equivalents, or combinations thereof. Convolutional neural networks can directly learn the image feature representations necessary to discriminate properties, which can work very well when there is a large amount of data to train on for each sample. Other methods can be used in conjunction with conventional computer vision features, such as SURF or SIFT, or learned embeddings (e.g., descriptors) generated by a trained convolutional neural network, which can be advantageous when only a small amount of data to train on is available. The trained detection machine learning model may be configured to provide cancer qualitative and / or cancer quantification outputs based on digital pathology images.

[0059] Figure 3 shows an exemplary training module 300 for training a detection machine learning model. As shown in Figure 3, the training data 302 may include one or more of the following: pathology images 304 (e.g., digital representations of biopsy images), input data 306 (e.g., cancer type, pathology category, etc.), and known outcomes 308 (e.g., quality designation) associated with the pathology images 304. The training data 302 and training algorithm 310 may be provided to a training component 320, which can apply the training data 302 to the training algorithm 310 to generate a detection machine learning model.

[0060] In 206, the detection machine learning model may be provided with inputs including patient-based digital pathology images (e.g., digital images of pathology specimens (e.g., histology, cytology, immunohistochemistry, etc.)) and optionally, associated information. The internal weights and / or layers of the detection machine learning model may be applied to the digital pathology images and associated information to determine cancer qualitative and, where applicable, cancer quantification.

[0061] Cancer qualitative assessment may indicate the presence or absence of cancer. Cancer qualitative assessment may be a binary decision in which the detection of a single cancer cell may correspond to the presence of cancer, and the failure to detect a single cancer cell may correspond to the absence of cancer. pCR cancer qualitative assessment may correspond to the absence of cancer and may be output when no cancer cells are detected.

[0062] According to one implementation, cancer presence and / or pCR cancer qualitative assessment may be protocol-specific so that the protocol can define one or more thresholds for cancer qualitative assessment. For example, the protocol may indicate that a minimum of 5 cancer cells per million cells is required to output cancer presence, and fewer than 5 cancer cells per million cells is sufficient for pCR cancer qualitative assessment.

[0063] In addition, as disclosed herein, the detection machine learning model may be configured to output a cancer type based on the image data received as input. The cancer type output may be an indication of the determined cancer type or the probability of a cancer type. In addition to the tissue sample-based digital image, the cancer type output may be informed by the input and may include tissue characteristics, slide type, glass type, tissue type, tissue region, chemicals used, and / or staining amount.

[0064] Cancer quantification may be the output when the presence of cancer is detected. Cancer quantification may include the number of cancer cells (e.g., number of cancer cells per million), the density of cancer cells, or an indication that the number of cancer cells exceeds a threshold amount (e.g., the MRD threshold). For example, cancer quantification may be or include MRD cancer quantification, which may refer to ultramicroscopic diseases such as diseases that remain latent in the patient but can eventually lead to recurrence. In cancer treatment, MRD may provide information on whether the treatment has eliminated the cancer or whether traces remain.

[0065] The output of the detection machine learning model (i.e., cancer qualitative analysis and, where applicable, cancer quantification) may be provided in 210 to a storage device 109 (e.g., cloud storage device, hard drive, network drive, etc.) in Figure 1A. The output of the detection machine learning model may be cancer qualitative analysis or cancer quantification, or also a notification based thereon. The notification may include information about cancer qualitative analysis, cancer quantification, cancer type, location of cancer cells, or equivalent. The notification may be provided via any applicable technique such as a notification signal, report, message via application, notification via device, or equivalent. The notification may be provided to any applicable device or personnel (e.g., histologist, scanning device operator, pathologist, record keeper, etc.). As an example, the output of the detection machine learning model may be integrated with the history of the corresponding target sample in a laboratory information system 125 that stores records of patients and associated target samples.

[0066] According to one implementation, the output of the detection machine learning model may be a report based on cancer qualitative analysis, cancer quantification, cancer type, cancer cell location, changes in any such factor over time, or equivalents. The report may be in any applicable format, such as PDF, HTML, in-app format, or equivalents.

[0067] According to one implementation, the output of a detection machine learning model may, or may include, a visual indicator in 210. The visual indicator may be provided, for example, via a visual application tool 108 in Figure 1B. The visual indicator may provide a visual representation of a digital image, or it may provide an indication of cells used when determining cancer qualitative or cancer quantification.

[0068] Figure 4 shows an exemplary digital image 400, which is enlarged to provide enlarged view 400A. The digital image 400 in Figure 4 is an H&E slide with therapeutic effects attributable to hormone therapy for prostate cancer. As shown in enlarged view 400A, the area of ​​cells not relied upon for cancer qualitative and / or cancer quantification is shown in area 402. Area 402 may correspond to an area determined by the detection machine learning model as less relevant for detection purposes. Area 402 corresponds to an area determined by the detection machine learning model as benign tissue where no cancer cells are detected. The area of ​​cells relied upon for cancer qualitative and / or cancer quantification is shown in area 404. Compared to the digital image 400 of the tissue sample, the area 404 relied upon for cancer qualitative and / or cancer quantification may be relatively small. Area 404 may correspond to an area determined by the detection machine learning model as more relevant for detection purposes. Area 404 corresponds to the area that the detection machine learning model determines to be a cancerous tissue region in which at least one cancer cell is detected.

[0069] According to one implementation, the detection machine learning algorithm may also be trained on and / or receive clinical information (e.g., patient information, surgical information, diagnostic information, etc.) and laboratory information (e.g., processing time, personnel, tests, etc.) as input. Based on such input, the detection machine learning algorithm may provide cancer qualitative and / or cancer quantification.

[0070] According to one implementation, the detection machine learning model may include a therapeutic effect machine learning model, as disclosed herein. The therapeutic effect may correspond to oncological treatment based on pharmaceuticals, hormone therapy, chemotherapy, etc. Tissue samples from patients treated with a therapy (e.g., cancer therapy) may have different properties from tissue samples from patients not treated with a similar therapy.

[0071] The treatment effect machine learning model may be generated using low-shot or transfer learning methods and may be initialized using a base detection machine learning model (i.e., a trained machine learning model trained on multiple training images that exclude treatment effects). For example, a sample detection machine learning model, such as those disclosed herein, may be trained using digital images from tissue samples from patients who have not received treatment and / or whose tissue samples do not exhibit a treatment effect. The treatment machine learning model may be initialized using the sample detection machine learning model such that the weights and / or one or more layers associated with the sample detection machine learning model are preserved and additional weights, weight modifications, layers, and / or layer modifications are applied when generating the treatment effect machine learning model. Similarly, the pCR and / or MDR detection components of a detection machine learning model may be initialized using a base detection machine learning model.

[0072] Figure 5 shows an exemplary implementation of a detection analysis as disclosed herein, with reference to Figure 2. Figure 5 shows a flowchart relating to a process for predicting pCR and / or MRD in prostate cancer with a therapeutic response. In the prostate cancer embodiment, pCR and MRD may be used as endpoints for clinical trials assessing neoadjuvant therapy. However, a major challenge is the presence of a therapeutic response when assessing cancer.

[0073] In 502, the detection machine learning model may be trained using images to output a cancer qualitative assessment, and, where applicable, a cancer quantification if the cancer qualitative assessment is a confirmed cancer qualitative assessment. Training may include inputting one or more digital images of prostate tissue (e.g., histopathology, H&E, IHC, 3D imaging, etc.) that include indications of the presence or absence of cancer. The detection training model may be trained using one or more machine learning algorithms and / or formats (e.g., deep learning, DNN, CNN, FCN, RCN, probabilistic models, discriminative models, etc.) as disclosed herein. The detection training model may be trained to output a cancer qualitative assessment, a cancer quantification, and assessments of pCR and / or MRD, which may be protocol-specific, as disclosed herein. Training images may include images that may be, or may be tagged as, one of pCR or MRD.

[0074] In 504, the quantification module may be trained based on the digital images applied in 502. To train the quantification module, the quantification of the quantity of prostate cancer (e.g., the number of cells presenting with prostate cancer) may be included along with all or a subset of the digital images used to train the model. It should be understood that the quantification module may be part of the overall detection model trained in 502.

[0075] In 506, the therapeutic effect module may be trained based on the digital images applied in 502. All or a subset of the digital images applied in 502 may depict tissue exhibiting a therapeutic effect. The therapeutic effect may be unique to the image, or it may be tagged so that the tag is used as part of the training.

[0076] In 508, a digital image of a pathology sample may be received as input to one or more of the trained detection machine learning model of 502, the quantification module of 504, and the therapeutic effect module of 506.

[0077] In 510, the detection machine learning model in 502, the quantification module in 504, and / or the treatment effect module in 506 may be used to determine pCR cancer qualitative, confirmed cancer qualitative. If pCR cancer qualitative is determined, it may be output in 512 (e.g., via notification, report, visual indication, etc.). If confirmed cancer qualitative is determined, it may be output in 518. In addition, or alternatively, cancer quantification may be determined. For example, the MRD value may be determined in 514 and output in 516.

[0078] Figure 6 illustrates an experiment based on the process disclosed in flowchart 200 of Figure 2. Figure 6 relates to the pathological evaluation of prostate cancer treated with neoadjuvant hormone therapy in whole-mount sections of radical prostatectomy. Such evaluation conventionally presents significant challenges due to tumor morphological changes, combined with the possibility of very small lesions of residual tumor in a large amount of non-tumorous tissue.

[0079] The detection machine learning model applied to this experiment may utilize a multi-instance learning approach to train a whole-slide image classifier using an SE-ResNet50 convolutional neural network. In this embodiment, the model was trained on 36,644 WSIs (7,514 of which had cancerous lesions) with a reduced embedding size to adapt to a very large number of patch instances in whole-mount slides. The detection machine learning model was then fine-tuned in a fully supervised context on a small annotated set of radical prostatectomy samples taken from prostate cancer patients treated in a neoadjuvant setting to further improve its performance on radical prostatectomy data. This produced an AUC of 0.99 for anti-androgen-treated cases.

[0080] The detection machine learning model showed an AUC of 0.99 in cases with anti-androgen receptor neoadjuvant therapy. After training, this exemplary system was evaluated on 40 WSI images from H&E-stained whole-mount prostatectomy slides from 15 prostatectomy specimens taken from patients after neoadjuvant therapy with anti-androgen drugs. Ground truth was established by pathologist annotations. All 37 malignant WSIs (3 WSIs containing tumors <5 mm and 34 WSIs containing tumors >5 mm) were correctly classified as latent treated cancers by the system. Of the three benign WSIs, the detection machine learning model misclassified one benign lesion as cancer, while the other two were correctly classified as benign tissue.

[0081] Therefore, accurate slide-level classification of H&E-stained slides from radical prostatectomy specimens has the potential to improve the accuracy and efficiency of histopathological evaluation of whole-mount sections from radical prostatectomy specimens from patients who received neoadjuvant therapy prior to surgery. As shown in Figure 6, the digital image 600 includes a visual indication of the cells on which the detection machine learning model is based in area 602. Enlarged figure 600A shows more clearly the cells on which the detection machine learning model is based and also shows specific area 604 within area 602, indicating cancer cells within area 604. Further enlarged figure 600B shows even more clearly the cells on which the detection machine learning model is based and also shows specific area 606 within specific area 604, indicating cancer cells and / or benign cells within area 606.

[0082] The device 700 in Figure 7 may correspond to hardware used by server systems 110, hospital servers 122, research laboratory servers 124, clinical trial servers 123, physician servers 121, laboratory information systems 125, and / or client devices, etc. The device 700 may also include a central processing unit (CPU) 720. The CPU 720 may be any type of processor device, including, for example, any type of dedicated or general-purpose microprocessor device. As will be understood by those skilled in the art, the CPU 720 may also be a single processor in a multicore / multiprocessor system, such a system operating alone or in a cluster of computing devices operating in a cluster or server farm. The CPU 720 may be connected to a data communication infrastructure 710, such as a bus, message queue, network, or multicore message transit scheme.

[0083] Device 700 may also include primary memory 740, for example, random access memory (RAM), and secondary memory 730. The secondary memory 730, for example, read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, flash memory, or the equivalent. In this embodiment, the removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. The removable storage unit may include a floppy disk, magnetic tape, optical disk, etc., which are read from and written to by the removable storage drive. As will be understood by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium having computer software and / or data stored therein.

[0084] In alternative implementations, the secondary memory 730 may include other similar means for enabling computer programs or other instructions to be loaded into device 700. Embodiments of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that enable software and data to be transferred from the removable storage units to device 700.

[0085] Device 700 may also include a communication interface ("COM") 760. The communication interface 760 enables software and data to be transferred between Device 700 and an external device. The communication interface 760 may include a modem, a network interface (such as an Ethernet® card), a communication port, a PCMCIA slot and card, or equivalent. The software and data transferred via the communication interface 760 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 760. These signals may be provided to the communication interface 760 via a communication path of Device 700, which may be implemented using, for example, wires or cables, optical fibers, telephone lines, cell phone links, RF links, or other communication channels.

[0086] The hardware elements, operating systems, and programming languages ​​for such devices are conventional in nature and are assumed to be well-familiar to those skilled in the art. Device 700 also includes input and output ports 750 and may be connected to input and output devices such as keyboards, mice, touchscreens, monitors, and displays. Naturally, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming on a single computer hardware platform.

[0087] Throughout this disclosure, references to components or modules generally refer to items that can be grouped together to perform a function or a related set of functions. Similar reference numbers are generally intended to refer to identical or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware.

[0088] The tools, modules, and functions described above may be implemented by one or more processors. The “storage” type medium may include any or all of the tangible memories of a computer, processor or equivalent, or its associated module, such as various semiconductor memories, tape drives, disk drives, and equivalents, which may, from time to time, provide non-transient storage for software programming.

[0089] The software may be transmitted over the Internet, a cloud service provider, or other telecommunications network. For example, the transmission may allow the software to be loaded from one computer or processor into another. Unless limited to non-transient tangible “storage” media as used herein, the terms computer or machine “readable media,” etc., refer to any medium involved in providing instructions for execution to a processor.

[0090] The general description set forth herein is illustrative and descriptive only and does not limit the present disclosure. Other embodiments of the present invention will be obvious to those skilled in the art from consideration herein and the practice of the present invention disclosed herein. This specification and the examples are intended to be considered illustrative only.

Claims

1. A computer implementation method for processing an electronic image, wherein the method is: Receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample. The method involves determining a detection machine learning model, which is generated by processing multiple training images, outputs a qualitative cancer assessment, and, if the qualitative cancer assessment is confirmed to be a qualitative cancer assessment, further outputs a quantitative cancer assessment. The digital image is provided as input to the aforementioned detection machine learning model, The output of the detection machine learning model is to receive confirmed cancer quantifications including minimal residual disease (MRD), wherein the detection machine learning model comprises a treatment effect machine learning model, and the treatment effect machine learning model is initialized with one of the layers and / or weights from a trained version of the detection machine learning model. Based on the aforementioned treatment effect machine learning model, the MRD cancer qualitative analysis is output. Computer implementation methods, including those mentioned above.

2. The computer implementation method according to claim 1, further comprising receiving one of the pathological complete response (pCR) and cancer qualitative assessment.

3. The computer implementation method according to claim 1, wherein the MRD cancer qualitative analysis is protocol-specific.

4. The computer implementation method according to claim 1, wherein the MRD cancer qualitative assessment corresponds to the number of cancer cells below the MRD threshold.

5. The computer implementation method according to claim 1, wherein the MRD cancer qualitative assessment identifies one or more diseases that remain latent in the patient but may ultimately lead to recurrence.

6. The computer implementation method according to claim 1, wherein the treatment effect machine learning model is trained based on the tagged treatment effects in the plurality of training images.

7. The computer implementation method according to claim 1, wherein receiving the confirmed cancer quantification also includes receiving the cancer type when the output of the detection machine learning model includes the confirmed cancer quantification.

8. The computer implementation method according to claim 7, wherein the type of cancer is determined based on the digital image and one or more of the following: tissue characteristics, slide type, glass type, tissue type, tissue region, chemical substance used, or amount of staining.

9. The computer implementation method according to claim 1, wherein the digital image is from a pathology category, and the pathology category is selected from one or more of histology, cytology, frozen section, immunohistochemistry (IHC), immunofluorescence, hematoxylin and eosin (H&E), hematoxylin only, molecular pathology, and / or 3D imaging.

10. A system for processing an electronic image, wherein the system is: At least one memory for storing instructions, At least one processor, the at least one processor executes the instruction, Receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample. The method involves determining a detection machine learning model, which is generated by processing multiple training images, outputs a qualitative cancer assessment, and, if the qualitative cancer assessment is confirmed to be a qualitative cancer assessment, further outputs a quantitative cancer assessment. The digital image is provided as input to the aforementioned detection machine learning model, The output of the detection machine learning model is to receive confirmed cancer quantifications including minimal residual disease (MRD), wherein the detection machine learning model comprises a treatment effect machine learning model, and the treatment effect machine learning model is initialized with one of the layers and / or weights from a trained version of the detection machine learning model. Based on the aforementioned treatment effect machine learning model, the MRD cancer qualitative analysis is output, At least one processor that performs operations including A system that includes these features.

11. The system according to claim 10, further comprising receiving one of pathological complete response (pCR) and cancer qualitative assessment.

12. The system according to claim 10, wherein the MRD cancer qualitative analysis is protocol-specific.

13. The system according to claim 10, wherein the MRD cancer qualitative analysis corresponds to the number of cancer cells below the MRD threshold.

14. The system according to claim 10, wherein the MRD cancer qualitative assessment identifies one or more diseases that remain latent in the patient but may ultimately lead to recurrence.

15. The system according to claim 10, wherein receiving the confirmed cancer quantification also includes receiving the type of cancer when the output of the detection machine learning model includes the confirmed cancer quantification.

16. The system according to claim 15, wherein the type of cancer is determined based on the digital image and one or more of the following: tissue characteristics, slide type, glass type, tissue type, tissue region, chemical used, or amount of staining.

17. The system according to claim 10, wherein the digital image is from a pathology category, and the pathology category is selected from one or more of histology, cytology, frozen section, immunohistochemistry (IHC), immunofluorescence, hematoxylin and eosin (H&E), hematoxylin only, molecular pathology, and / or 3D imaging.

18. A non-transient computer-readable medium, wherein the non-transient computer-readable medium stores instructions, and when the instructions are executed by a processor, the processor causes the processor to perform an operation for processing an electronic image, and the operation is Receiving a digital image corresponding to a target sample associated with a pathology category, wherein the digital image is an image of a tissue sample. The method involves determining a detection machine learning model, which is generated by processing multiple training images, outputs a qualitative cancer assessment, and, if the qualitative cancer assessment is confirmed to be a qualitative cancer assessment, further outputs a quantitative cancer assessment. The digital image is provided as input to the aforementioned detection machine learning model, The output of the detection machine learning model is to receive confirmed cancer quantifications including minimal residual disease (MRD), wherein the detection machine learning model comprises a treatment effect machine learning model, and the treatment effect machine learning model is initialized with one of the layers and / or weights from a trained version of the detection machine learning model. Based on the aforementioned treatment effect machine learning model, the MRD cancer qualitative analysis is output. Non-transient computer-readable media, including [specific examples of such media].

19. A non-transient computer-readable medium according to claim 18, further comprising receiving one of pathological complete response (pCR) cancer qualitative assessments.

20. The non-transient computer-readable medium according to claim 18, wherein the MRD cancer qualitative analysis is protocol-specific.

Images