Systems and methods for delivering digital biomarker and genomic panels

Abstract

Disclosed is a system for receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information, determining one or more of a prediction, a recommendation, and / or a plurality of data for the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict biomarkers and a plurality of genomic set elements, and determining whether to record an output and at least one visualization area as part of a medical record history within a clinical reporting system based on the prediction, the recommendation, and / or the plurality of data.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 966,659, filed on January 28, 2020, the entire disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] Various embodiments of the present invention are generally directed to developing artificial intelligence (AI) techniques to detect biomarkers, genomic features, treatment resistance, and other relevant features necessary for additional testing of pathology specimens. More specifically, certain embodiments of the present disclosure are directed to systems and methods for predicting, identifying, or detecting biomarkers and genomic features of prepared tissue specimens. The present disclosure further provides systems and methods for creating predictive models that predict labels from never-seen slides. Background Art

[0004] There may be multiple steps, costs, and time required for a pathologist to receive the results of a biomarker or genomics panel. For a biomarker result, (a) the pathologist may note an appropriate or suspicious portion of the patient; (b) the laboratory may receive a request for a slide to be stained; (c) the laboratory cuts the block or finds an appropriate unstained slide; (d) the portion is stained. And (e) the test is electronically documented in the case and given to the pathologist for final review. For a genomic panel, (a) the molecular test request may be given to the pathologist; (b) the pathologist may select a slide from which to sequence; (c) a prompt to re-cut the tissue may be made; (d) a prompt to remove the tumor from a previous biopsy incision may be made based on the pathologist's overview; (e) the genome from the removed tumor tissue may be sequenced; and (f) a genetic report may be created. These processes may be expensive and time consuming.

[0005] Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present disclosure. The background description provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art or a suggestion of prior art by inclusion in this section. Summary of the invention

[0006] According to certain aspects of the present disclosure, systems and methods are disclosed for predicting a biomarker and / or at least one genomic feature in a digital image associated with a tissue specimen.

[0007] A computer-implemented method for processing an electronic image corresponding to a specimen comprises: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information; determining one or more of a prediction, a recommendation, and / or a plurality of data for the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict biomarkers and a plurality of genomic set elements; and determining whether to record an output and at least one visualization region as part of a case history within a clinical reporting system based on the prediction, the recommendation, and / or the plurality of data.

[0008] A system for processing an electronic image corresponding to a specimen, comprising: a memory storing instructions; and at least one processor that executes the instructions to implement a process including receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information; determining one or more of a prediction, a recommendation, and / or a plurality of data for the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict biomarkers and a plurality of genomic set elements; and determining whether to record an output and at least one visualization area as part of a case history within a clinical reporting system based on the prediction, the recommendation, and / or the plurality of data.

[0009] A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to implement a method for processing an electronic image corresponding to a specimen, the method comprising: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information, determining one or more of a prediction, a recommendation, and / or a plurality of data for the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict biomarkers and a plurality of genomic set elements, and determining whether to record an output and at least one visualization region as part of a case history within a clinical reporting system based on the prediction, the recommendation, and / or the plurality of data.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments:

[0012] Figure 1Aillustrates an exemplary block diagram of a system and network for detecting a biomarker and / or at least one genomic feature according to an exemplary embodiment of the present disclosure;

[0013] Figure 1B An exemplary block diagram of a biomarker detection platform for predicting biomarkers and genomic panel features using machine learning according to an embodiment of the present disclosure is illustrated;

[0014] Figure 1C illustrates an exemplary block diagram of a slide analysis tool according to an exemplary embodiment of the present disclosure;

[0015] Figure 2A is a flow chart illustrating an exemplary method for detecting a biomarker and / or at least one genomic feature using a machine learning system according to one or more exemplary embodiments of the present disclosure;

[0016] Figure 2B is a flow chart illustrating an exemplary method for training a machine learning system to detect a biomarker and / or at least one genomic feature according to one or more exemplary embodiments of the present disclosure;

[0017] Figure 3 is a flow chart illustrating an exemplary method for visualizing positive biomarker foci according to one or more exemplary embodiments of the present disclosure;

[0018] Figure 4 is a flow chart illustrating an exemplary method for visualizing a tumor region to guide a molecular pathologist according to one or more exemplary embodiments of the present disclosure.

[0019] Figure 5 is a flow chart illustrating an exemplary method for reporting predicted development of anti-tumor resistance according to one or more exemplary embodiments of the present disclosure;

[0020] Figure 6 Depicting exemplary options for a user to view visualizations and / or reports according to one or more exemplary embodiments of the present disclosure;

[0021] Figure 7 An exemplary system is depicted that may perform the techniques presented herein. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0023] The systems, devices, and methods disclosed herein are described in detail by way of example and with reference to the accompanying drawings. The examples discussed herein are merely examples and are provided to help explain the apparatus, devices, systems, and methods described herein. Unless specifically designated as mandatory, the features or components shown in the drawings or discussed below should not be considered mandatory for any particular implementation of any of these devices, systems, or methods.

[0024] Furthermore, for any method described, whether or not the method is described in conjunction with a flowchart, it should be understood that, unless otherwise specified or required by the context, any explicit or implicit ordering of steps performed in the execution of the method does not imply that the steps must be performed in the order presented, but may instead be performed in a different order or in parallel.

[0025] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal," and furthermore, the terms "a" and "an" herein do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced items.

[0026] Pathology refers to the study of diseases and the causes and effects of diseases. More specifically, pathology refers to the tests and analyses performed to diagnose diseases. For example, tissue samples can be placed on slides to be observed under a microscope by a pathologist (e.g., a physician who is an expert in analyzing tissue samples to determine whether there are any abnormalities). That is, pathology specimens can be cut into multiple slices, stained, and prepared into slides for the pathologist to examine and give a diagnosis. When the diagnosis on the slide is uncertain, the pathologist can arrange additional cutting levels, staining, or other tests to collect more information from the tissue. Then, (one or more) technicians can create new (one or more) slides, which can contain additional information used by the pathologist when making a diagnosis. The process of creating additional slides may be time-consuming, not only because it may involve retrieving a tissue block, cutting it to make a new slide, and then staining the slide, but also because it may be processed in batches for multiple orders. This may significantly delay the final diagnosis given by the pathologist. In addition, even after the delay, it may still not be guaranteed that the new (one or more) slides will have enough information to give a diagnosis.

[0027] Pathologists can individually evaluate cancer and other disease pathology slides. The present disclosure presents the use of AI to detect and predict biomarkers and genomic panel features. Specifically, the present disclosure describes various exemplary user interfaces that can be used in the workflow, as well as AI tools that can be integrated into the workflow to speed up and improve the work of pathologists.

[0028] For example, a computer can be used to analyze images of tissue samples to quickly identify whether additional information is needed about a particular tissue sample, and / or to highlight to a pathologist areas that he or she should look more closely. Thus, the process of obtaining additional stained slides and tests can be automated prior to being reviewed by a pathologist. When paired with an automated slide segmentation and staining machine, this can provide a fully automated slide preparation pipeline. Such automation has at least the following benefits: (1) minimizes the amount of time wasted by a pathologist determining that a slide is insufficient for diagnosis, (2) minimizes the (average total) time from specimen collection to diagnosis by avoiding additional time between scheduling additional tests and generating the additional tests, (3) reduces the amount of time and material wasted per re-cut by allowing re-cutting to be performed while a tissue block (e.g., pathology specimen) is on a cutting station, (4) reduces the amount of tissue material wasted / discarded during slide preparation, (5) reduces the cost of slide preparation by partially or fully automating the procedure, (6) allows automated customized cutting and staining of slides, which will result in more representative / informative slides from the specimen, (7) allows more slides to be generated per tissue block, facilitating a more informed / precise diagnosis by reducing the overhead of requesting additional tests from the pathologist, and / or (8) identifies or verifies the correct attributes of a digital pathology image (e.g., with respect to specimen type), etc.

[0029] The process of using computers to assist pathologists is known as computational pathology. Computational methods used in computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but are not limited to, deep learning, neural networks, classification, clustering, and regression algorithms. Through the use of computational pathology, lives can be saved by helping pathologists improve the accuracy, reliability, efficiency, and accessibility of diagnoses. For example, computational pathology can be used to help detect slides suspected of cancer, allowing pathologists to review and confirm their initial assessment before giving a final diagnosis.

[0030] As described above, the computational pathology process and device of the present disclosure can provide an integrated platform that allows a fully automated process including data acquisition, processing, and viewing of digital pathology images via a web browser or other user interface, while integrating with a laboratory information system (LIS). In addition, cloud-based data analysis of patient data can be used to aggregate clinical information. The data can come from hospitals, clinics, field researchers, etc. And can be analyzed by machine learning, computer vision, natural language processing, and / or statistical algorithms to perform real-time monitoring and prediction of health patterns at multiple geographic specificity levels.

[0031] The present disclosure relates to systems and methods for quickly and correctly identifying and / or verifying the specimen type of a digital pathology image or any information related to the digital pathology image without having to access a LIS or similar information database. One embodiment of the present disclosure may include a system that is trained to identify various characteristics of a digital pathology image based on a data set of previous digital pathology images. The trained system can provide a classification of the specimen shown in the digital pathology image. The classification can help provide (one or more) treatment or diagnostic predictions for a patient associated with the specimen.

[0032] The systems and methods of the present disclosure can use artificial intelligence to detect scanned slides with any features that can be a prerequisite for further testing (e.g., the highest molecules or invasive tumor volume of human epidermal growth factor receptor 2 / estrogen receptor / progesterone receptor (HER2 / ER / PR)). This feature detection can be done at the case, section, or block level of the specimen. The results can be available via any user interface (e.g., through a viewer, report, through a laboratory information system (LIS), etc.). The systems and methods of the present disclosure can also provide instant visualization of predicted immunohistochemistry (IHC) results, genomic panels, information derived using AI (e.g., treatment resistance), etc. based on one or more digital pathology specimen images collected from the patient. This can provide turnaround time and cost benefits for both hospitals and patients. In addition to showing the results of the digital IHC or digital genomic panel, the system can further manage the reimbursement elements of the purchase. This can provide additional efficiency for hospitals and patients.

[0033] The systems and methods of the present disclosure can use artificial intelligence to detect scanned slides with any features that can be a prerequisite for further testing (e.g., the highest molecules or invasive tumor volume of human epidermal growth factor receptor 2 / estrogen receptor / progesterone receptor (HER2 / ER / PR)). This feature detection can be done at the case, section, or block level of the specimen. The results can be available via any user interface (e.g., through a viewer, report, through a laboratory information system (LIS), etc.). The systems and methods of the present disclosure can also provide instant visualization of predicted immunohistochemistry (IHC) results, genomic panels, information derived using AI (e.g., treatment resistance), etc. based on one or more digital pathology specimen images collected from the patient. This can provide turnaround time and cost benefits for both hospitals and patients. In addition to showing the results of the digital IHC or digital genomic panel, the system can further manage the reimbursement elements of the purchase. This can provide additional efficiency for hospitals and patients.

[0034] The present disclosure includes one or more embodiments of a slide analysis tool. The input to the tool may include a digital pathology image and any relevant additional input. The output of the tool may include global and / or local information about the specimen. The specimen may include a biopsy or surgical resection specimen.

[0035] An exemplary global output of the disclosed tool(s) may include information about the entire image, e.g., specimen type, overall quality of the cut of the specimen, overall quality of the glass pathology slide itself, and / or tissue morphological properties. An exemplary local output may indicate information in a particular region of the image, e.g., a particular image region may be classified as having blur or cracks in the slide. The present disclosure includes embodiments for developing and using the disclosed slide analysis tool(s), as described in further detail below.

[0036] Figure 1A A block diagram of a system and network for determining specimen characteristic or image characteristic information related to a digital pathology image using machine learning according to an exemplary embodiment of the present disclosure is illustrated.

[0037] Specifically, Figure 1A An electronic network 120 is illustrated that can be connected to servers of a hospital, laboratory, and / or doctor's office, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a research laboratory server 124, and / or a laboratory information system 125, etc. can each be connected to the electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present disclosure, the electronic network 120 can also be connected to a server system 110, which can include a processing device configured to implement a biomarker detection platform 100, which includes a slide analysis tool for determining specimen characteristics or image characteristics information related to a digital pathology image according to an exemplary embodiment of the present disclosure, and using machine learning to create a genome panel.

[0038] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may create or otherwise obtain images of one or more patient's cytology specimen(s), histopathology specimen(s), slide(s) of (one or more) cytology specimen(s), digitized images of slide(s) of (one or more) histopathology specimen(s), or any combination thereof. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The server system 110 may also include a processing device for processing the images and data stored in the one or more storage devices 109. The server system 110 may further include one or more machine learning tools or capabilities. For example, according to one embodiment, the processing device may include a machine learning tool for the biomarker detection platform 100. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

[0039] The physician server 121, hospital server 122, clinical trial server 123, research laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to examine images of slides. In a hospital environment, tissue type information may be stored in the laboratory information system 125. However, the correct tissue classification information is not always paired with the image content. In addition, even if the LIS is used to access the specimen type of the digital pathology image, the label may be incorrect due to the fact that many components of the LIS may be manually input, leaving a large margin for error. According to an exemplary embodiment of the present disclosure, the specimen type can be identified without accessing the library information system 125, or can be identified to possibly correct the library information system 125. For example, a third party may anonymously access image content for which no corresponding specimen type label is stored in the LIS. In addition, access to LIS content may be restricted due to its sensitive content.

[0040] Figure 1B An exemplary block diagram of a biomarker detection platform for predicting biomarkers and genomic panel features using machine learning according to an embodiment of the present disclosure is illustrated.

[0041] Specifically, Figure 1B Components of a biomarker detection platform 100 according to one embodiment are depicted. For example, the biomarker detection platform 100 may include a slide analysis tool 101, a data acquisition tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, a storage device 106, and a viewing application tool 108.

[0042] As described below, slide analysis tool 101 refers to a process and system for processing digital images associated with tissue specimens and analyzing slides using machine learning according to an exemplary embodiment.

[0043] The data acquisition tool 102 refers to processes and systems for facilitating the delivery of digital pathology images to various tools, modules, components, and devices for classifying and processing digital pathology images according to an exemplary embodiment.

[0044] According to an exemplary embodiment, the slide ingestion tool 103 refers to a process and system for scanning pathology images and converting them into digital form. The slide can be scanned using a slide scanner 104, and the slide manager 105 can process the image on the slide into a digitized pathology image and store the digitized image in a memory 106.

[0045] According to an exemplary embodiment, the viewing application tool 108 refers to a process and system for providing specimen characteristic or image characteristic information related to (one or more) digital pathology images to a user (e.g., a pathologist). The information can be provided through various output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0046] The slide analysis tool 101 and each of its components can transmit and / or receive digitized slide images and / or patient information to the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125 via the electronic network 120. In addition, the server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the slide analysis tool 101, the data acquisition tool 102, the slide capture tool 103, the slide scanner 104, the slide manager 105, and the viewing application tool 108. The server system 110 may also include a processing device for processing the images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or capabilities, for example, due to the processing device. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

[0047] Any of the above devices, tools, and modules may be located on a device that may be connected to an electronic network 120, such as an Internet or cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0048] Figure 1C An exemplary block diagram of a slide analysis tool 101 according to an exemplary embodiment of the present disclosure is illustrated. The slide analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0049] According to one embodiment, the training image platform 131 can create or receive training images for training a machine learning system to effectively analyze and classify digital pathology images. For example, training images can be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The images used for training can come from real sources (e.g., humans, animals, etc.) or can come from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images can include (a) digitized slides stained with various stains, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc.; and / or (b) digitized tissue samples from 3D imaging devices, such as microCT.

[0050] The training image acquisition module 132 can create or receive a data set including one or more training images corresponding to any one or both of an image of human tissue and a graphically rendered image. For example, the training image can be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The data set can be maintained on a digital storage device. The quality score determiner module 133 can identify quality control (QC) issues (e.g., defects) of the training images at a global or local level, which may greatly affect the usability of the digital pathology image. For example, the quality score determiner module can use information about the entire image, such as the specimen type, the overall quality of the specimen cut, the overall quality of the glass pathology slide itself, or the tissue morphological characteristics, and determine the overall quality score of the image. The treatment identification module 134 can analyze the image of the tissue and determine which digital pathology images have a treatment effect (e.g., after treatment) and which images do not have a treatment effect (e.g., before treatment). It is useful to identify whether a digital pathology image has a treatment effect, as the effects of previous treatments in a tissue may affect the morphology of the tissue itself. Most LIS do not explicitly maintain this feature, and therefore it may be desirable to classify the types of specimens that have had a previous treatment effect.

[0051] According to one embodiment, the target image platform 135 may include a target image input module 136, a specimen detection module 137, and an output interface 138. The target image platform 135 may receive a target image and apply a machine learning model to the received target image to determine the characteristics of the target specimen. For example, the target image may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the research laboratory server 124, and / or the laboratory information system 125. The target image acquisition module 136 may receive a target image corresponding to a target specimen. The specimen detection module 137 may apply a machine learning model to the target image to determine the characteristics of the target specimen. For example, the specimen detection module 137 may detect the specimen type of the target specimen. The specimen detection module 137 may also apply a machine learning model to the target image to determine a quality score of the target image. In addition, the specimen detection module 137 may apply a machine learning model to the target specimen to determine whether the target specimen is before or after treatment.

[0052] The output interface 138 may be used to output information about the target image and the target specimen (eg, to a screen, a monitor, a storage device, a web browser, etc.).

[0053] Figure 2A2 is a flowchart illustrating an exemplary method for predicting biomarkers and at least one genomic set element using a machine learning system according to one or more exemplary embodiments of the present disclosure. For example, the exemplary method 200 (i.e., steps 202-212) can be performed by the slide analysis tool 101 automatically or in response to a request from a user.

[0054] According to one embodiment, an exemplary method 200 for predicting biomarkers and at least one genomic set element may include one or more of the following steps. In step 202, the method may include receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information. The tissue specimen may include a histological specimen, and the patient information may include specimen type, case and patient ID, section within the case, a rough description, etc. The plurality of clinical information may include an assigned pathologist, whether the related specimen is available for testing, etc. The digital image may be received in a digital storage device (e.g., a hard drive, a network drive, a cloud storage, a random access memory (RAM), etc.).

[0055] In step 204, the method may include determining a prediction and / or visualization of the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict a biomarker and at least one genomic set element. The machine learning system may additionally output the recommendation and / or data to an electronic storage device.

[0056] In step 206, the method may include generating a notification to the user indicating that the prediction and / or visualization is available. The notification may include a visual display, a pop-up window, or other suitable alert.

[0057] In step 208, the method may include generating options for the user to view the forecast and / or visualization. As described below, the options may include: Figure 6 An exemplary screen display is shown in FIG.

[0058] In step 210, the method may include generating at least one display of at least one recommended treatment path based on the prediction and / or visualization. The at least one recommended treatment path may include a validated treatment path, a new treatment path, a clinical treatment path, etc., or a subsequent step (e.g., a clinical trial, a professional doctor visit, etc.) based on the generated prediction. Visualization of digital immunohistochemistry or genomic panel results may be accomplished using a variety of methods, including but not limited to:

[0059] a. overlaying at least one region of interest on top of the original image;

[0060] b. Side-by-side visualization;

[0061] c. Reports with quantitative metrics; and

[0062] d. Test run using the results summary figures.

[0063] The visualization of the recommendations may include an interactive web interface where a user (e.g., pathologist, oncologist, patient, etc.) can learn more about a particular recommendation (e.g., open clinical trials, hospitals / physicians specializing in treatment, etc.) via direct links to sources (e.g., websites, literature, etc.) of the interface. Alternatively, the visualization may include a report where a user can view a summarized, immutable report that may include, but is not limited to, the following elements:

[0064] a. Patient history

[0065] b. Case summary

[0066] c. Diagnosis summary

[0067] d. Numerical and / or “artificial” test results

[0068] e. Recommended next steps for the patient based on the digital test results.

[0069] The method can group similar patients (e.g., patients with similar morphological patterns, similar biomarker expression, similar genomic profiles, similar treatment pathways, or other similarities) together as a reference for a given case to support the decision-making process for a particular case. The visualization of similar patients can be associated with the recommended treatment pathway for the case, or it can be unrelated. A user (e.g., a pathologist, oncologist, patient, etc.) can learn more about a particular patient and his or her results (e.g., from a clinical trial, a drug, etc.). The results can be visualized through an interactive web interface (e.g., filtering, sharing, saving, etc.) or through a report, as disclosed above.

[0070] The results may be in the form of a comprehensive report that includes the report forecasts and related information (e.g., PDF). An exemplary report may contain one or more of the following elements:

[0071] a. Patient history

[0072] b. Patient Summary

[0073] c. Case summary

[0074] d. Digital testing completed

[0075] e. Digital test results

[0076] f. Synthesis and summary of the results and their implications for patients

[0077] g. Visualization of statistical data based on outcomes of similar patients (e.g., infographics, interactive websites, etc.)

[0078] h. Summary of related and / or recent literature

[0079] i. Recommended next steps (e.g., clinical trials, medications, chemotherapy, etc.), etc.

[0080] In step 212, the method may include determining whether to record the output and at least one visualization region as part of the case history within a clinical reporting system based on the prediction and / or the visualization. The method may also include integrating the recommendation and the visualization into a final diagnostic report for the specimen.

[0081] Figure 2B 2 is a flowchart illustrating an exemplary method for training a machine learning system to predict biomarkers and at least one genomic set element according to one or more exemplary embodiments of the present disclosure. For example, the exemplary method 220 (i.e., steps 221-235) can be performed by the slide analysis tool 101 automatically or in response to a request from a user.

[0082] According to one embodiment, exemplary method 220 is used to train a machine learning system to predict biomarkers and at least one genomic set element. In step 221, the method may include receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information. The tissue specimen may include a histological specimen, and the patient information may include specimen type, case and patient ID, section within the case, a rough description, etc. The plurality of clinical information may include an assigned pathologist, whether the related specimen is available for testing, etc. The digital image may be received in a digital storage device (e.g., a hard drive, a network drive, a cloud storage, a random access memory (RAM), etc.).

[0083] In step 223 , the method may include developing a system to store and archive a plurality of processed images associated with a plurality of patient data.

[0084] In step 225, the method may include storing the plurality of processed images in a digital storage device. The digital storage device may include a hard drive, a network drive, cloud storage, RAM, or the like.

[0085] In step 227, the method may include generating at least one recommendation for a treatment pathway based on the plurality of processed images. The treatment pathway may include a clinical trial, a treatment, etc. The recommendation may be patient specific and may be based on at least one relevant feature of the plurality of stored images and patient data (e.g., patient diagnosis, medical history, demographics). The recommendation for the treatment pathway may include or be based on a clinical practice guideline, which may be customized based on patient demographics, pre-approval drugs or therapies, clinical practice, etc.

[0086] In step 229, the method may include generating a prediction of the biomarker and the at least one genomic set element.

[0087] In step 231, the method may include generating a list of at least one recommended treatment pathway based on the prediction. The list of at least one recommended treatment pathway may include drug treatments, clinical trials, etc., and related information (e.g., success rate, location of treatment, etc.) based on the predicted biomarkers and genomic set elements.

[0088] In step 233, the method may include converting the prediction and at least one recommended treatment pathway into a form that can be visualized and interpreted by a user (e.g., a pathologist, a patient, an oncologist, etc.). The method may additionally include outputting or displaying at least one result in a variety of effective formats (e.g., interactive, structured, templated, static, etc.) depending on the user and use case.

[0089] In step 235, the method may include outputting one or more predicted values ​​and treatment pathway recommendations to a user interface. Depending on the user and use case, the output or display results may be in a variety of effective formats (eg, interactive, structured, templated, static, etc.).

[0090] Figure 3 is a flowchart illustrating an exemplary method for using and training a machine learning system to visualize positive biomarker lesions according to one or more exemplary embodiments of the present disclosure. Visualization of biomarkers (e.g., IHC markers, genomic panels) can help pathologists understand how computational assays are performed. Exemplary methods 300 and 320 can be used to visually display detected positive biomarker lesions. Exemplary methods 300 and 320 (i.e., steps 301-313 and steps 321-333) can be performed by slide analysis tool 101 automatically or in response to a request from a user.

[0091] According to one embodiment, an exemplary method 300 for training a machine learning system to visualize positive biomarker lesions may include one or more of the following steps. In step 301, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available portion, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

[0092] In step 303, the method may include developing a system to store and archive a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, outcomes, and the like.

[0093] In step 305, the method may include storing the plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard drive, a network drive, cloud storage, RAM, or the like.

[0094] In step 307, the method may include generating at least one recommendation for a treatment pathway based on at least one relevant feature of the plurality of digital images. The treatment pathway may include a clinical trial, treatment, etc. determined for the patient based on at least one relevant factor (e.g., patient diagnosis, medical history, demographic data, etc.).

[0095] In step 309, the method may include predicting at least one biomarker and genomic set element.

[0096] In step 311, the method may include generating a list of at least one recommended treatment pathway based on the predicted biomarker and genomic set elements. The recommended treatment pathways (e.g., drugs, clinical trials, etc.) and any related information (e.g., success rates, treatment locations, etc.) may be based on the predicted biomarker and genomic set elements.

[0097] In step 313 , the method may include converting one or more predictions or recommendations into a form that can be visualized or interpreted by a user (eg, a pathologist, a patient, an oncologist, etc.).

[0098] In step 321, the method may include receiving one or more digital images associated with a tissue specimen, a plurality of related case and patient information from a clinical system. The pathology specimen (e.g., histology specimen), related case and patient information (e.g., specimen type, case and patient ID, section within case, gross description, etc.), and information from the clinical system (e.g., assigned pathologist, specimen available for testing, etc.) into a digital storage device (e.g., hard drive, network drive, cloud storage, RAM, etc.).

[0099] In step 323, the method may include generating at least one of a prediction, a recommendation, and / or a plurality of data for the one or more digital images.

[0100] In step 325, the method may include generating a notification indicating that at least one of the prediction, the recommendation, and / or the plurality of data is available. Additionally, visualization for immunohistochemistry or genomic panels may also be available.

[0101] In step 327, the method may include providing the user with an option to select a visualization and / or report to view. The user may be a pathologist.

[0102] In step 329, the method may include generating a visualization of a recommended treatment path based on the prediction, the recommendation, and / or at least one of the plurality of data. The treatment path (e.g., proven, new, clinical, etc.) or subsequent steps (e.g., clinical trial, professional doctor visit, etc.) may be based on the output / generated prediction. The visualization of the digital immunohistochemistry or genomic panel results may include one or more of the following:

[0103] a. Overlay the positive regions of interest on the original image (e.g., outlines, gradients with colors mapped to the algorithm's predictions, etc.)

[0104] b. Side-by-side comparison of images with and without digital IHC or genomic panel predictions displayed

[0105] c. A prioritized list (e.g., a slide show of image crops, an interface that allows the user to jump from one focus to another, etc.) of all positive foci identified as positive regions for the biomarker or mutation of interest

[0106] d. A report that summarizes all tests into one final output (e.g., score, results, recommendations, etc.) or lists the final output for each numerical test.

[0107] In step 331 , the method may include recording the visualization as part of the case history within a clinical reporting system.

[0108] At step 333, the method may include integrating the one or more test results into a final diagnostic report associated with the tissue specimen.

[0109] Figure 44 is a flowchart illustrating an exemplary method for using and training a machine learning system to visualize tumor regions to guide molecular pathologists according to one or more exemplary embodiments of the present disclosure. Visualizing malignant tissue regions on digitized pathology slides can help molecular pathologists evaluate optimal downstream tests. Exemplary embodiments can be used to select optimal regions for downstream testing. Exemplary methods 400 and 420 can be used to visualize tumor regions to guide molecular pathologists. Exemplary methods 400 and 420 (i.e., steps 401-413 and steps 421-433) can be performed by slide analysis tool 101 automatically or in response to a request from a user.

[0110] According to one embodiment, an exemplary method 400 for training a machine learning system to visualize tumor regions to guide molecular pathologists may include one or more of the following steps. In step 401, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available portion, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

[0111] In step 403, the method may include developing a system to store and archive a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, outcomes, and the like.

[0112] In step 405, the method may include storing the plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard drive, a network drive, cloud storage, RAM, or the like.

[0113] In step 407, the method may include generating at least one recommendation for a treatment pathway based on at least one relevant feature of the plurality of digital images. The treatment pathway may include a clinical trial, treatment, etc. for the patient based on at least one relevant factor (e.g., patient diagnosis, medical history, demographic data, etc.).

[0114] In step 409 , the method may include predicting a tumor region on the plurality of digital images.

[0115] In step 411, the method may include generating a list of at least one recommended treatment pathway based on the predicted tumor region. The recommended treatment pathway (e.g., drug, clinical trial, etc.) and any related information (e.g., success rate, treatment location, etc.) may be based on the predicted tumor region.

[0116] In step 413 , the method may include converting one or more predictions or recommendations into a form that can be visualized or interpreted by a user (eg, a pathologist, a patient, an oncologist, etc.).

[0117] In step 421, the method may include receiving one or more digital images associated with a tissue specimen, a plurality of related case and patient information from a clinical system. A pathology specimen (e.g., a histology specimen), related case and patient information (e.g., specimen type, case and patient ID, section within case, gross description, etc.), and information from the clinical system (e.g., assigned pathologist, specimen available for testing, etc.) into a digital storage device (e.g., a hard drive, a network drive, cloud storage, RAM, etc.).

[0118] In step 423, the method may include generating at least one of a prediction, a recommendation, and / or a plurality of data for the one or more digital images.

[0119] In step 425, the method may include generating a notification indicating that at least one of a prediction, a recommendation, and / or a plurality of data is available. Additionally, visualization of immunohistochemistry or genomic panels may also be available.

[0120] In step 427, the method may include providing the user with an option to select a visualization and / or report to view. The user may be a pathologist.

[0121] In step 429, the method may include generating a visualization of a recommended treatment path based on at least one of the prediction, the recommendation, and / or the plurality of data. The treatment path (e.g., proven, new, clinical, etc.) or subsequent steps (e.g., clinical trial, specialist doctor visit, etc.) may be based on the output / generated prediction. The visualization of the digital tumor finder results may include one or more of the following:

[0122] a. Overlay on the original image (e.g., outline, gradient with color mapped to algorithm prediction, etc.) over the positive region of interest. The overlay can be registered to subsequent images to guide the user in scraping the tumor for sequencing

[0123] b. Side-by-side comparison of images with and without predictive display

[0124] c. Prioritized list of top regions (e.g., tumors with highest mutational burden, etc.). The prioritized list can include a report summarizing all sections analyzed for tumor-specific features (e.g., tumor mutational burden) and predictions.

[0125] In step 431 , the method may include recording the visualization as part of the case history within a clinical reporting system.

[0126] At step 433, the method may include integrating the one or more test results into a final diagnostic report associated with the tissue specimen.

[0127] Figure 5 is a flow chart illustrating an exemplary method for using and training a machine learning system to report predicted development of anti-tumor resistance according to one or more exemplary embodiments of the present disclosure. Anti-tumor resistance occurs when cancer cells resist and survive despite anti-cancer treatment. This ability can evolve in cancer during the course of treatment. Predicting therapies that cancer will have the most difficulty acquiring resistance to can improve patient treatment and survival rates. Certain cancers may evolve resistance to multiple drugs during the course of treatment. This information can be provided to identify treatments that are likely to be ineffective. Exemplary embodiments can be used to report predicted development of anti-tumor resistance. Exemplary methods 500 and 520 can be used to predict the development of anti-tumor resistance. Exemplary methods 500 and 520 (i.e., steps 501-511 and steps 521-533) can be performed by the slide analysis tool 101 automatically or in response to a request from a user.

[0128] According to one embodiment, an exemplary method 500 for training a machine learning system to visualize tumor regions to guide molecular pathologists may include one or more of the following steps. In step 501, the method may include receiving one or more digital images associated with a tissue specimen and corresponding information. The one or more digital images may include a histology slide. The corresponding information may include relevant information (e.g., specimen type, available portion, rough description, etc.), clinical information (e.g., diagnosis, biomarker information, etc.), and patient information (e.g., demographic data, gender, etc.).

[0129] In step 503, the method may include developing a system to store and archive a plurality of digital images and corresponding patient data. The corresponding patient data may include images from screening, follow-up, outcomes, and the like.

[0130] In step 505, the method may include storing the plurality of digital images and corresponding patient data in a digital storage device. The digital storage device may include a hard drive, a network drive, cloud storage, RAM, or the like.

[0131] In step 507, the method may include predicting current or future resistance to at least one treatment path or at least one drug. The prediction may use AI, testing, etc. AI may use various inputs to infer this information, including demographic data, digital images of (stained) tissue containing the tumor, patient history, etc.

[0132] In step 509, the method may include generating a list of at least one treatment that is predicted to be unlikely to be effective.

[0133] In step 511, the method may include generating a list of at least one recommended treatment pathway based on the predicted tumor region. The recommended treatment pathway (e.g., medication, clinical trial, etc.) and any related information (e.g., success rate, treatment location, etc.) may be based on the predicted tumor region.

[0134] In step 511 , the method may include converting one or more predictions or recommendations into a form that can be visualized or interpreted by a user (eg, a pathologist, a patient, an oncologist, etc.).

[0135] In step 521, the method may include receiving one or more digital images associated with a tissue specimen, a plurality of related case and patient information from a clinical system. A pathology specimen (e.g., a histology specimen), related case and patient information (e.g., specimen type, case and patient ID, section within case, gross description, etc.), and information from the clinical system (e.g., assigned pathologist, specimen available for testing, etc.) into a digital storage device (e.g., a hard drive, a network drive, cloud storage, RAM, etc.).

[0136] In step 523, the method may include generating at least one therapeutic effect prediction and / or a plurality of data for the one or more digital images.

[0137] In step 525, the method may include generating a notification indicating that a prediction and visualization of at least one treatment that is unlikely to be effective is available.

[0138] In step 527, the method may include providing the user with an option to select a visualization and / or report to view. The user may be a pathologist.

[0139] In step 529, the method may include generating a visualization of at least one treatment that is unlikely to be effective based on the prediction. The visualization of information may be provided via:

[0140] a. An interactive web interface where a user (e.g., pathologist, oncologist, patient, etc.) can learn more about at least one specific recommendation (e.g., open clinical trials, hospitals / physicians specializing in treatment, etc.) via direct links and sources (e.g., websites, literature, etc.) of the interface

[0141] b. Reports, where users can view aggregated, immutable reports that may include, but are not limited to, the following elements:

[0142] i. Patient history

[0143] ii. Case summary

[0144] iii. Diagnosis Summary

[0145] iv. Numerical and / or “artificial” test results

[0146] v. Recommended next steps for the patient based on the digital test results.

[0147] In step 531 , the method may include recording the visualization as part of the case history within a clinical reporting system.

[0148] At step 533, the method may include integrating the one or more test results into a final diagnostic report associated with the tissue specimen.

[0149] Figure 6 Depicted are exemplary options for a user to view visualizations and / or reports according to one or more exemplary embodiments of the present disclosure. In display 60, an example report with a display of slide scoring results is shown. Display 65 shows an exemplary window with an option for a user to order a digital IHC run on a slide.

[0150] like Figure 7 As shown, device 700 may include a central processing unit (CPU) 720. CPU 720 may be any type of processor device, including, for example, any type of dedicated or general purpose microprocessor device. As will be appreciated by those skilled in the relevant art, CPU 720 may also be a single processor in a multi-core / multi-processor system (such a system operates alone), or in a cluster of computing devices operating in a cluster or server farm. CPU 720 may be connected to a data communications infrastructure 710, such as a bus, a message queue, a network, or a multi-core messaging scheme.

[0151] The device 700 may also include a main memory 740, such as a random access memory (RAM), and may also include an auxiliary memory 730. The auxiliary memory 730, such as a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a tape drive, an optical disk drive, or a flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. The removable storage may include a floppy disk, a tape, an optical disk, etc., which are read and written to by the removable storage drive. As will be appreciated by those skilled in the relevant art, such a removable storage unit typically includes a computer-usable storage medium having computer software and / or data stored therein.

[0152] In alternative implementations, secondary memory 730 may include similar components for allowing computer programs or other instructions to be loaded into device 700. Examples of such components may include program cartridges and cartridge interfaces (such as found in video game devices), removable storage chips (such as EPROM or PROM) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from the removable storage unit to device 700.

[0153] The device 700 may also include a communication interface ("COM") 760. The communication interface 760 allows software and data to be transferred between the device 700 and external devices. The communication interface 760 may include a modem, a network interface (such as an Ethernet card), a communication port, or a PCMCIA slot and card, etc. The software and data transferred via the communication interface 760 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by the communication interface 760. These signals may be provided to the communication interface 760 via a communication path of the device 700, which may be implemented using, for example, wires or cables, optical fibers, telephone lines, cellular phone links, RF links, or other communication channels.

[0154] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with this. The device 700 may also include input and output ports 750 to connect to input and output devices such as keyboards, mice, touch screens, monitors, displays, etc. Of course, the various server functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the server can be implemented by appropriate programming of a computer hardware platform.

[0155] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or a group of related functions. Similar reference numerals are generally intended to refer to the same or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware.

[0156] The above tools, modules and functions can be executed by one or more processors. "Storage" type media can include any or all of the tangible memory of a computer or processor, etc. or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-temporary storage for software programming at any time.

[0157] The software may be communicated over the Internet, a cloud service provider, or other telecommunication network. For example, the communication may enable the loading of the software from one computer or processor to another. As used herein, unless restricted to non-transitory, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0158] The foregoing general description is merely exemplary and explanatory, and is not intended to limit the present disclosure. Other embodiments of the present invention will be apparent to those skilled in the art from consideration of the specification and practice of the present invention disclosed herein. The specification and examples are to be considered exemplary only.

Claims

1. A computer-implemented method for processing an electronic image corresponding to a specimen, the method comprising: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information; determining predictions of biomarkers and a plurality of genomic panel elements for the one or more digital images using a machine learning system that has been trained using a plurality of training images to predict biomarkers and the plurality of genomic panel elements; Generate a list of recommended treatment pathways based on predictions of biomarkers and genomic panel elements; determining whether to record the output and at least one visualization region as part of a case history within a clinical reporting system based on the predictions of the biomarker and genomic panel elements; and Based on the predictions of the biomarkers and the genomic suite elements, one or more displays of at least one recommended treatment in the list of recommended treatment pathways are generated by at least one of an overlay of at least one region of interest layered on top of the one or more digital images and a side-by-side visualization of the one or more digital images displaying the predictions and the one or more digital images not displaying the predictions. 2 . The computer-implemented method of claim 1 , wherein the method further comprises generating a notification indicating that a predicted or side-by-side visualization of the one or more digital images is available.

3. The computer-implemented method of claim 1, wherein the method further comprises generating an option for a user to view the forecast or the side-by-side visualization.

4. The computer-implemented method of claim 1, wherein the side-by-side visualization comprises digital immunohistochemistry or genomic panel results, the digital immunohistochemistry or genomic panel results comprising a summary of a digital test run having at least one result.

5. The computer-implemented method of claim 1 , wherein determining a prediction comprises: receiving one or more digitized images of a pathology specimen, related information, clinical information, and patient information; Develop systems to store and archive multiple images and multiple corresponding patient data; determining at least one predicted biomarker and at least one predicted genomic panel element based on the plurality of images and the plurality of corresponding patient data; and The one or more predicted values ​​and the at least one treatment pathway recommendation are converted into a user-readable form.

6. The computer-implemented method of claim 5, further comprising outputting the one or more predicted values ​​and the at least one treatment pathway recommendation to a user interface.

7. The computer-implemented method of claim 5, wherein the at least one treatment pathway recommendation is based on a plurality of clinical practice guidelines.

8. The computer-implemented method of claim 1, wherein at least one treatment pathway recommendation comprises a validated treatment pathway, a new treatment pathway, and / or a clinical treatment pathway based on the prediction.

9. A system for processing an electronic image corresponding to a specimen, the system comprising: at least one memory storing instructions; and At least one processor is configured to execute the instructions to perform operations, the operations comprising: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information; determining predictions of biomarkers and a plurality of genomic encyclopedia elements for the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict the biomarkers and the plurality of genomic encyclopedia elements; Generate a list of recommended treatment pathways based on predictions from biomarkers and genomic panel elements; using the machine learning system and based on the prediction, determining whether to record the output and at least one visualization region as part of a case history within a clinical reporting system; and Based on the predictions of biomarkers and genomic panel elements, a visualization of digital immunohistochemistry or genomic panel results is generated by overlaying at least one region of interest on top of the one or more digital images, and at least one of a side-by-side visualization of the one or more digital images displaying the predictions and the one or more digital images not displaying the predictions.

10. The system of claim 9, the operations further comprising: A notification is generated indicating that a prediction or visualization of the one or more digital images is available.

11. The system of claim 9, the operations further comprising: Generates options for users to view forecasts or side-by-side visualizations.

12. The system of claim 9, the operations further comprising: One or more displays of at least one recommended treatment are generated based on the prediction.

13. The system of claim 9, the operations further comprising: Based on the prediction, one or more displays of at least one recommended treatment are generated by overlaying at least one region of interest on top of the one or more digital images and at least one of displaying the one or more digital images of the prediction and a side-by-side visualization of the one or more digital images displaying the prediction.

14. The system according to claim 9, wherein: The recommendation for at least one treatment pathway was based on multiple clinical practice guidelines.

15. The system of claim 9, wherein determining a prediction comprises: receiving one or more digitized images of a pathology specimen, related information, clinical information, and patient information; Develop systems to store and archive multiple images and multiple corresponding patient data; determining at least one predicted biomarker and at least one predicted genomic panel element based on the plurality of images and the plurality of corresponding patient data; and The one or more predicted values ​​and the at least one treatment pathway recommendation are converted into a user-readable form.

16. The system of claim 15, the operations further comprising outputting the one or more predicted values ​​and the at least one treatment pathway recommendation to a user interface.

17. The system of claim 9, wherein at least one treatment pathway recommendation comprises a clinical trial and / or treatment based on the prediction.

18. A non-transitory computer readable medium storing instructions which, when executed by a processor, cause the processor to perform operations for monitoring the health of a population, the operations comprising: receiving one or more digital images associated with a tissue specimen, a related case, a patient, and / or a plurality of clinical information; determining biomarkers and a plurality of genomic set elements for prediction of the one or more digital images using a machine learning system, the machine learning system having been trained using a plurality of training images to predict the biomarkers and the plurality of genomic set elements; Generate a list of recommended treatment pathways based on predictions from biomarkers and genomic panel elements; determining whether to record the output and at least one visualization region as part of a case history within a clinical reporting system based on the prediction of the biomarker and genomic panel elements; and Based on the predictions of the biomarkers and genomic set elements, one or more displays of at least one recommended treatment are generated by at least one of an overlay of at least one region of interest layered on top of the one or more digital images, and a side-by-side visualization of the one or more digital images displaying the predictions and the one or more digital images not displaying the predictions.

19. The computer-readable medium of claim 18, wherein the operations further comprise generating a notification indicating that a predicted or side-by-side visualization of the one or more digital images is available.

Images