Tool for multiple cancers diagnosis, treatment recommendations and biomarker discovery

Abstract

Systems and methods are directed towards a web application for diagnosing cancer using multimodal artificial intelligence analysis. A web-based artificial intelligence system for multi-modal cancer diagnosis comprises a web application interface that receives uploads of patient data including biopsy images, electronic health records, and omics data; an ensemble of deep learning models including at least a first convolutional neural network and a second convolutional neural network that process the biopsy images at multiple magnification levels to generate cancer grade predictions; a multimodal data integration module that combines the cancer grade predictions with patient demographic information and clinical data from the electronic health records and omics data; and a visualization component that displays the cancer grade predictions as interactive charts with color-coded probability distributions for different cancer severity levels.

Description

TOOL FOR MULTIPLE CANCERS DIAGNOSIS, TREATMENTRECOMMENDATIONS AND BIOMARKER DISCOVERYRELATED APPLICATION(S)

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 672,582, filed July 17, 2024, and U.S. Provisional Application No. 63 / 674,560, filed July 23, 2024, each of which is herein incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates to artificial intelligence-based medical diagnostic systems, and more particularly to a web application for multi-modal cancer diagnosis and analysis using deep learning techniques on diverse medical data types including whole slide biopsy images, electronic health records, and multi-omics data.BACKGROUND OF THE DISCLOSURE

[0003] In the field of cancer diagnostics, healthcare providers face significant challenges in accurately and efficiently analyzing complex medical data from multiple sources. Traditional methods often struggle with integrating diverse data types such as whole slide biopsy images, electronic health records, and multi-omics data, leading to time-consuming analysis and potential diagnostic inconsistencies. Furthermore, the shortage of qualified pathologists worldwide compounds these issues, resulting in delays in diagnosis and treatment planning.SUMMARY

[0004] Here, systems, methods, and computer program products are presented for multimodal cancer diagnosis and analysis.

[0005] Embodiments of the present disclosure include a web application to address these challenges by providing a comprehensive platform for multi-modal cancer diagnosis and analysis. By leveraging advanced artificial intelligence techniques, including deep learning and machine learning algorithms, the application enables rapid processing and integration of diverse medical data types. Embodiments of the present disclosure empower healthcareprofessionals to make more informed diagnostic decisions efficiently, potentially improving patient outcomes and alleviating the burden on pathology resources.

[0006] In some embodiments, a web-based artificial intelligence system for multi-modal cancer diagnosis comprises a web application interface that receives uploads of patient data including biopsy images, electronic health records, and omics data; an ensemble of deep learning models including at least a first convolutional neural network and a second convolutional neural network that process the biopsy images at multiple magnification levels to generate cancer grade predictions; a multimodal data integration module that combines the cancer grade predictions with patient demographic information and clinical data from the omics data; and a visualization component that displays the cancer grade predictions as interactive charts with color-coded probability distributions for different cancer severity levels.

[0007] In some embodiments, the ensemble of deep learning models comprises one of a VGG16 convolutional neural network, a ResNet50 convolutional neural network, a vision transformer, and an efficientnetO-7.

[0008] In some embodiments, ensemble of deep learning models the VGG16 convolutional neural network and the ResNet50 convolutional neural network process biopsy images at magnification levels including 5x, lOx, 20x, 40x, lOOx, 200x, and 400x.

[0009] In some embodiments, the multimodal data integration module processes the omics data including DNA, RNA, protein, metabolite, and phenotype data.

[0010] In some embodiments, the visualization component includes interactive tabs displaying biopsy images, patient demographics, natural language processing summaries, and detailed summary information.

[0011] In some embodiments, the interactive tabs include a biopsy tab that displays wholeslide images with Al-annotated tumor regions marked at pixel and patch levels.

[0012] In some embodiments, the web application interface processes biopsy images by dividing each image into a plurality of patches, each patch having a size selected from the group consisting of 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192 pixels.

[0013] In some embodiments, a computer-implemented method for diagnosing cancer using multimodal artificial intelligence analysis comprises receiving patient data through a web interface , the patient data including whole slide biopsy images, electronic health records, and omics data; processing the whole slide biopsy images using an ensemble ofconvolutional neural networks trained on histopathological data to generate probability scores for multiple cancer grades; integrating the probability scores with patient demographic information and clinical history data extracted from the electronic health records using natural language processing; generating explainable diagnostic results that include spatial markup of tumor regions on the biopsy images and identification of predictive biomarkers from the omics data; and presenting the diagnostic results through an interactive web dashboard with visualization charts that display probability distributions for cancer subtype and severity levels using a color-coded system.

[0014] In some embodiments, generating explainable diagnostic results includes applying attention mechanisms from transformer architectures to identify which genomic features contribute most significantly to a cancer grade prediction.

[0015] In some embodiments, the natural language processing extracts clinical information from physician notes, pathology reports, laboratory results, and treatment history records to create patient summary profiles.

[0016] In some embodiments, the interactive web dashboard includes separate visualization tabs for displaying Al-annotated biopsy images, patient demographics, natural language processing summaries of clinical data, and identified biomarkers with associated confidence scores.

[0017] In some embodiments, a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising receiving multimodal patient data including digitized biopsy images, electronic health record data, and omics data through a web application interface; applying ensemble deep learning models to analyze the digitized biopsy images at multiple scales and magnifications to determine cancer subtype and grade classifications; processing the electronic health record data using natural language processing to extract relevant clinical features; combining outputs from the ensemble deep learning models with the extracted clinical features to generate comprehensive diagnostic predictions; and generating interactive visualizations that display the diagnostic predictions as probability distributions with explainable artificial intelligence features including tumor region annotations and biomarker identification.

[0018] In some embodiments, the biomarker identification applies attention mechanisms from transformer architectures to determine which genomic features contribute most significantly to the cancer grade classifications.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate various exemplary embodiments and together with the description, serve to explain the principles of the disclosed embodiments.

[0020] Fig. 1 depicts a flowchart of a method for processing patient data and generating diagnostic results, according to an embodiment of the present disclosure.

[0021] Fig. 2 depicts a flowchart of a method for processing and analyzing image data, according to an embodiment of the present disclosure.

[0022] Fig. 3 is a block diagram of an exemplary system and web application for processing patient data, according to an embodiment of the present disclosure.

[0023] Fig. 4 is a block diagram of an exemplary system and web application for processing patient data, according to an embodiment of the present disclosure.

[0024] Fig. 5 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure.

[0025] Fig. 6 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure.

[0026] Fig. 7 illustrates exemplary tissue classifications determined by a web application, according to an embodiment of the present disclosure.

[0027] Fig. 8 is a block diagram of an exemplary system architecture for a web application, according to an embodiment of the present disclosure.

[0028] Fig. 9 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure.

[0029] Fig. 10 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure.

[0030] Fig. 11 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure.

[0031] Fig. 12 is an exemplary computing node.DETAILED DESCRIPTION

[0032] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used through-out the drawings to refer to the same or like parts.

[0033] The systems, devices, and methods disclosed herein are described in detail by way of examples and with reference to the figures. The examples discussed herein are examples only and are provided to assist in the explanation of the apparatuses, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be taken as mandatory for any specific implementation of any of these devices, systems, or methods unless specifically designated as man-datory.

[0034] Also, for any methods described, regardless of whether the method is described in conjunction with a flow diagram, it should be understood that unless otherwise specified or required by context, any explicit or implicit ordering of steps performed in the execution of a method does not imply that those steps must be performed in the order presented but instead may be performed in a different order or in parallel.

[0035] As used herein, the term "exemplary" is used in the sense of "example," rather than "ideal." Moreover, the terms "a" and "an" herein do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced items.

[0036] An Al-based framework and tool is provided for multiple cancer diagnosis, treatment recommendations, and biomarker discovery using explainable Al, transfer learning, foundation models and multimodal data including electronic health record data, scanned biopsies (at multiple scales, magnifications, and resolutions, as well as multi-window size, nuclear medicine imaging and multi-omics. Such a tool aims to reduce the time for diagnosis, grading and treatment and results in greatly improved patient care, helping to alleviate the problems caused by staffing shortages in pathologists and the variability between their decisions. In addition, this tool provides automatic diagnosis for rare cancers using techniques developed for transfer learning and foundation models. The tool is explainable, given the technology developed to provide evidence of markup and summary on the decision.

[0037] Embodiments of the tool utilize multi-modal and multi-omics data in the predictive modeling and in providing explainability of the decision using biopsy mark of the tumor and summary about the patients with information that is most unique and helped in the diagnosis and therapy recommendation decision.

[0038] An objective of the web application of the tool is for users to obtain automatic and accurate diagnoses and treatment recommendations for a patient by uploading biopsy images of the tumor and other modalities that are available as nuclear imaging modalities (z.e., CT, X-ray, etc.), electronic medical / health records, and multi-omics data (RNASeq, proteomics, epigenetics or any other omics information). The uploaded information is sent to the backend server where the Al models, including deep learning and machine learning techniques, areserved. These models can generate a representation for each of the different modalities, and then incorporate into one architecture to perform automatic diagnosis and find tumor areas on the slide or imaging. The architecture can find relevant genes that are predictive in nature, and the relevant record in the EMR that helped in the diagnosis to also recommend a personalized therapy for the patient.

[0039] Additionally, embodiments of the web application may incorporate interactive tabs, each providing pertinent information that underpins the decision-making process. A “Biopsy” tab furnishes a visual representation of the whole-slide images (WSIs), enabling users to directly assess the image for cancer grading using their own devices. Should data be available for the given biopsy, the “Demographics” and “NLP Summary” tabs are generated. It is to be noted that NLP on prescriptions was not performed in real-time or using a deep learning algorithm. In the case of omics data, NLP may appear in a new tab.

[0040] The visualization of multi-modal information for patients in the process of automatic cancer diagnosis including providing summaries using natural language techniques, an overview about the patient, probabilities of the decision using various visualization techniques together with the markup of the tumor area in multi-label of pixel level patch level.

[0041] An additional feature of the tool is the capability to diagnose on a multi-scale platform, using different magnifications of the tissue in imaging. This feature allows for deeper analysis of the tissues at multiple levels of magnification and within different patches or pixels of the image. Multi-magnification analysis improves the overall data quality presented to the system user, and can allow for quicker diagnoses, more detailed treatment plans, and a deeper understanding of a patient’s prognosis. Different patches or blocks of tissues may be analyzed

[0042] In some embodiments, the system architecture comprises a web application, a backend server, and cloud storage for processing and analyzing multimodal patient data to generate cancer diagnostic results.

[0043] In some cases, the system may implement a method 100 for processing and analyzing medical data, as illustrated in Fig. 1. The method 100 (e.g., the steps 102-110) may be performed automatically by a processor or in response to a request from a user (e.g., physician, pathologist, etc.). The method 100 may include multiple steps for handling various types of medical information and providing diagnostic capabilities.

[0044] The web application may serve as the user interface, allowing healthcare professionals to input patient data and view results. The web application may initiate themethod 100 at step 102, which involves receiving patient data including whole slide biopsy images, electronic health records, and omic data. The omic data may include DNA sequencing data, RNA expression data, protein analysis data, metabolite profiles, and phenotype information that are processed to identify cancer-specific biomarkers.

[0045] The backend server may act as the system's core, processing incoming data from the web application. At step 104, the method 100 may process the whole slide biopsy images using an ensemble of convolutional neural networks (CNNs) trained on histopathological data to generate probability scores for multiple cancer grades. The backend server may implement algorithms for analyzing multimodal patient data and generating diagnostic results.

[0046] The ensemble of CNNs may include a VGG16 CNN, a ResNet50 CNN, a vision transformer, and an EfficientNet CNN. The EfficientNet CNN may include any suitable model of the EfficientNet network, including any version of EfficientNet-BO through EfficientNet-B7. The combination of these models within the ensemble architecture combines voting and stacking techniques capable of modeling high resolution images at multiple scales given an efficient number of parameters that prevent overfitting to a specific magnifications. The transformer and attentions mechanism within the ensemble spot the most predictive areas, areas with suspicious patterns that may be indicative of cancer, or cancer morphological behavior. This allows the model to distinguish between different cancer types and levels, to improve accuracy and efficiency of the diagnosis workflow. Data is preprocessed using various augmentation techniques, including applying transformation functions to existing images. These functions enhance model robustness and generalization, and may include geometric transformations (rotation, flipping, scaling, translation, shearing, zooming), color space transformations (brightness, contrast, hue adjustments), and more advanced methods like mixing images, random erasing, and adversarial training, to extend the data and generalize the model. Each image is tiled, where a most important tile is selected based on the attention mechanism. The selected tile is provided to the ensemble models, which are stacked parallel to voting at each layer to predict a diagnosis based on a specific cancer outcome. To identify the most important tiles, MedSAM, DeepLabV3, UNET and other segmentation techniques adapted to histology may be applied to find anomaly patterns in the tissue compared to the well structed small cells in benign tissues. Attention mechanisms to determine how much attention should be given to each tile may then be applied to the tiles. Generally, attention mechanisms are used in text-based data, but may be applied to images using the ensemble learning model.

[0047] This ensemble of deep learning models may process input biopsy images at magnification levels including 5x, lOx, 20x, 40x, lOOx, 200x, and 400x. In some embodiments, the system may perform multi-scale diagnosis using different magnifications of the tissue imaging. This capability may be implemented as part of the processing steps in the method 100, allowing for a more comprehensive analysis of the biopsy images at various levels of detail. Processing biopsy images may be accomplished by dividing each image into a plurality of patches. Each patch may be of a size selected from the group consisting of 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192 pixels. Each original image undergoes partitioning into smaller images. This approach effectively reduces the computational burden and memory requirements, which typically ranged from 10 to 20 MB per image. Moreover, employing image-wise labeling facilitated the categorization of patches into distinct classes. The patching algorithm involved dividing the image into two halves horizontally and further segmenting each half into three parts vertically. Subsequently, each patch was resized to a target size of 256 x 256 pixels, for example, and saved individually. This patching strategy not only facilitates efficient processing but also allows for focused analysis on specific regions of interest within the histopathology images, thereby enhancing classification accuracy.

[0048] Cloud storage may provide a secure and scalable solution for managing large volumes of patient data. The cloud storage may store various types of data, including text, images, and sensor readings, ensuring data integrity and accessibility.

[0049] When the web application receives patient data input by the user at step 102, the data may be sent to the backend server after processing at step 104. At step 106, the method 100 may integrate the probability scores with patient demographic information and clinical history data extracted from the electronic health records using natural language processing. The server may process this data, retrieve relevant historical information from cloud storage, and apply diagnostic algorithms. The natural language processing may extract clinical information from physician notes, pathology reports, laboratory results, and treatment history records to create patient summary profiles. As part of the natural language processing step, biomarkers are extracted from clinical notes and medical codesm using named entity recognition, random feature extraction, and attention mechanisms that understand the level of attention to give sequential data in the text and codes in the electronic health record (EHR). Hierarchical clustering is used to segment specific visits and notes, and topic modeling for categorizing various segments of clinical notes to align with a diagnosis prediction. Theinsights and layers from this modeling are integrated with other modalities using various fusion techniques including early fusion, joint fusion and late fusion techniques that will enable understanding different modalities in a similar way to a human interpretation.

[0050] At step 108, the method 100 may generate explainable diagnostic results that include spatial markup of tumor regions on the biopsy images and identification of predictive biomarkers from the genomic data. The system learns to spot these patterns based on learned general morphology and normal behavior of the tissue and hence able to spot anomalies on new unseen cancers. The system may also utilize zero show and few shot learning techniques, as well as registration techniques to match tissues from other modalities, clinical reports, and omics data based on location and match of cancerous and anomaly areas. The resulting diagnostic information may then be sent back to the web application for display to the user, while also being stored in the cloud for future reference. Generating explainable diagnostic results may include applying attention mechanisms from transformer architectures to identify which genomic features contribute most significantly to a cancer grade prediction. Biomarker identification may apply attention mechanisms from transformer architectures to determine which genomic features contribute most significantly to the cancer grade classifications.

[0051] Finally, at step 110, the method 100 may present the diagnostic results through an interactive web dashboard with visualization charts that display probability distributions for cancer severity levels using a color-coded system. The interactive web dashboard may include separate visualization tabs for displaying Al-annotated biopsy images, patient demographics, natural language processing summaries of clinical data, and identified biomarkers with associated confidence scores.

[0052] As illustrated in Fig. 2, the system may implement a method 200 for processing and analyzing data at multiple magnification levels, similar to the method 100 as shown in Fig. 1. The method 200 (e.g., the steps 202-200) may be performed automatically by a processor or in response to a request from a user (e.g., physician, pathologist, etc.). The method 200 may begin at step 202, where data is uploaded to the web application interface; this may be analogous to step 102 in method 100. Data may be uploaded by the system automatically, or in response to a request by a user. For example, a user may upload an image via a screen such as that illustrated in Fig. 6. The image may be an MRI image, an Xray image, or a PET / CT image, as well as a tissue biopsy image.

[0053] Following the upload, the method 200 may proceed to step 204 [step 104], where whole slide image (WSI) data is partitioned into patches using classifiers. Each whole slideimage is systematically partitioned into 256x256 patches, for example, forming the basis for a detailed analysis using CNN classifiers, as in step 206 . The patch sizes may include, but are not limited to, 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192. These classifiers categorize each patch into one of nine distinct classes: scan background (BG), tissue background (T), normal, healthy tissue (N), acquisition artifact (A), or one of the five Gleason grades (R1R5). Exemplary classified tissue samples 702-718 are shown in Fig. 7. Leveraging the versatility of pre-trained ResNet models, specifically ResNet-18, ResNet-34, and ResNet-50, the methodology ensures a comprehensive examination of the dataset at varying levels of complexity. This multi-model approach enhances the robustness of analysis, capturing intricate features within each patch and facilitating a nuanced understanding of prostate cancer pathology across different Gleason grades and tissue types.

[0054] At step 208, the method 200 may repeat the process for each magnification level of the image data, including 40x, 20x, lOx, and 5x. At each magnification, the systematic division and classification of 256 x 256 patches are repeated, allowing for a comprehensive analysis of prostate cancer pathology at varying resolutions. This multi-level evaluation enhances the reliability and generalizability of the system’s findings, acknowledging the significance of adapting to diverse magnification contexts in the field of histopathological analysis.

[0055] The method 200 may conclude at step 210 [step 108], where multiple magnifications are compared, allowing for a comprehensive analysis of the tissue samples at different scales.

[0056] Fig. 3 is a block diagram of an exemplary system and web application for processing patient data, according to an embodiment of the present disclosure. The system may implement various data processing and analysis techniques to handle multimodal input data and generate comprehensive diagnostic results. In some embodiments, the system may include a storage computing platform 300 that stores various types of medical data including a cancer type database. The cancer type database may include, for example, prostate cancer biopsies, lung cancer biopsies, breast cancer biopsies, and corresponding electronic health records (EHR), nuclear imaging, and multi-omics data. The cancer type database may include prostate cancer types, lung cancer, colon and rectal cancer, skin cancer, bladder cancer, breast cancer, cervical cancer, pancreatic cancer, lymphoma, kidney cancer, liver cancer, thyroid cancer, endometrial cancer, mesothelioma, blood cancers, brain cancers, esophageal cancers,gallbladder cancers, anal cancers, intestinal cancer, stomach cancer, adrenal gland cancers, testicular cancers, urethral cancers, uterine cancer, ovarian cancer, laryngeal cancer, nasopharyngeal cancer, and oral cancers, as well as other cancer types. The storage computing platform 300 may connect to cloud storage 302 to maintain and manage large volumes of patient data securely. Cloud storage 302 may store foundation models for imaging modalities, textual modalities, and multi omics, including but not limited to GPT, bioBERT, LLaMA, clinicalBERT, imageBERT, etc. The cloud storage 302 may use transfer learning techniques across cancer types. With transfer learning, the system learns how to identify morphological patterns in the pathology and transfer this knowledge using the learned weights of the deep neural networks of the various components and given a new cancer type that usually has different morphological patters. The system learns to spot these patterns based on its learned general morphology and normal behavior of the tissue and hence able to spot anomalies on new unseen cancers. In some embodiments, zero shot and few shot learning techniques are used. Registration techniques are performed to match the tissue on other modalities as MRI and CT images, clinical reports, and omics data based on locations and match of cancerous and anomaly areas.

[0057] A pathologist interface 304 may connect to the web application 306, allowing medical professionals to interact with the system and review diagnostic information. The system may enable data flow between the storage computing platform 300, cloud storage 302, pathologist interface 304, web application 306, and output module 308 to facilitate medical data analysis and diagnosis capabilities.

[0058] A web application 306 may provide functionality for uploading data, implementing Al-based components with multiple classifiers, and generating visualizations of the analysis results. The web application 306 may process the uploaded medical data and communicate with an output module 308. The output module 308 may generate several types of outputs based on the processed data, including early detection indicators, diagnosis results, treatment recommendations, and tumor segmentation analysis.

[0059] In some embodiments, the system may employ data augmentation techniques to enhance the robustness and generalization of the deep learning models. These techniques may include normalization of input data and random scaled cropping of images. Normalization may help standardize the input data across different sources and modalities, while random scaled cropping may introduce variability in the training data to improve model performance on diverse inputs.

[0060] The system may utilize a 5-fold cross-validation approach for training and testing the deep learning models. This approach may involve dividing the dataset into five subsets, using four subsets for training and one for validation in each iteration. By rotating through all possible combinations, the system may ensure a thorough evaluation of model performance and reduce the risk of overfitting.

[0061] To assess the performance of the diagnostic models, the system may employ the Cohen Kappa Score as an evaluation metric. The Cohen Kappa Score may provide a measure of agreement between the model's predictions and the ground truth labels, taking into account the possibility of agreement occurring by chance. This metric may be particularly useful for evaluating the system's performance in multi-class classification tasks, such as cancer grading.

[0062] The web application 306 may implement various deep learning models to process and analyze the multimodal patient data. These models may include convolutional neural networks for image analysis, natural language processing models for text data. The system may apply these models to extract relevant features from the input data and generate diagnostic predictions. Fusion models may be utilized for integrating information from multiple modalities. The insights and layers from the various modeling techniques of each modality are integrated with other modalities using various fusion techniques including early fusion, joint fusion and late fusion techniques that will enable understanding different modalities in a similar way to a human.

[0063] In some cases, the output module 308 may provide visualizations to present the diagnostic results in an interpretable manner. These visualizations may include heatmaps highlighting regions of interest in biopsy images, probability distributions for different cancer grades, and summaries of relevant patient information extracted from electronic health records.

[0064] The system may continuously update and refine the deep learning models based on new data and feedback from pathologists. This iterative process may help improve the accuracy and reliability of the diagnostic predictions over time.

[0065] The web application may integrate data from the storage computing platform 300 and cloud storage 302 to populate the various interface components. In some cases, the pathologist interface 304 may allow medical professionals to interact with these user interface components, review diagnostic information, and make informed clinical decisions based on the presented data.

[0066] The output module 308 may generate visualizations for display in the various interface components. These visualizations may include probability distributions, tumor segmentation results, and treatment recommendations based on the processed multimodal patient data.

[0067] Fig. 4 is a block diagram of an exemplary system and web application for processing patient data. The system may implement various data processing and analysis techniques to handle multimodal input data and generate comprehensive diagnostic results. In some cases, the system may utilize a web application system 400 as illustrated in Fig. 4 to facilitate data processing and analysis workflows.

[0068] The web application system 400 may include a cloud computing platform 402 that provides storage and computational resources for processing medical data. The cloud computing platform 402 may contain components for storage, computation, and Al-based models with HIPAA / FISMA compliance capabilities.

[0069] In some embodiments, the system may employ data augmentation techniques to enhance the robustness and generalizability of the deep learning models. These techniques may include normalization of input data and random scaled cropping of images.Normalization may help standardize input values across different modalities, while random scaled cropping may introduce variability in the training data to improve model performance on diverse inputs.

[0070] The system may implement a 5-fold cross-validation approach for training and testing models. This approach may involve dividing the dataset into five subsets, using four subsets for training and one for validation in each iteration. The process may be repeated five times, with each subset serving as the validation set once. This technique may help assess the model's performance and generalizability across different data partitions.

[0071] To evaluate the performance of the diagnostic models, the system may utilize the Cohen Kappa Score as a metric. The Cohen Kappa Score may provide a measure of agreement between the model's predictions and the ground truth labels, taking into account the possibility of agreement occurring by chance. This metric may be particularly useful for assessing the reliability of the system's diagnostic outputs.

[0072] The web application system 400 may include a visualization module 404 that enables display and analysis of processed medical data. A web interface 406 may provide access to patient data including vital signs, EHR data, and medical imaging data. The web interface 406 may connect to a pathologist interface 408 that allows medical professionals to review and analyze the diagnostic information generated by the system.

[0073] By integrating these various components and techniques, the system may process and analyze diverse types of medical data to generate comprehensive diagnostic results. The combination of data augmentation, cross-validation, performance metrics, and multi-model processing may contribute to the system's ability to handle complex diagnostic tasks across multiple cancer types.

[0074] Fig. 5 is an exemplary screen of a web application for processing patient data, according to an embodiment of the present disclosure. The web application 306 as shown in Fig. 3 may implement a cancer selection interface 500 for the patient, as illustrated in Fig. 5. The cancer selection interface 500 may display multiple cancer type options in a grid layout. In some cases, each cancer type may be presented in a rectangular box containing a title indicating the cancer type and two interactive buttons - a “Learn More” button and an "Open" button. By selecting the “Learn More” button, the user may be presented with additional information about the cancer type, including prognosis, symptoms, or other related information. The “Open” button allows the user to navigate to a next screen, as illustrated in Fig. 6. The cancer selection interface 500 may provide a structured way to navigate between different cancer types and access detailed information or functionality related to each type.

[0075] Fig. 6 illustrates an upload interface 600 for receiving multi-modal patient data. The upload interface 600 may include multiple upload buttons for different types of medical data, such as biopsy images, pathology reports, and electronic health records. Images from CT scans, MRIs, Xray, and other modalities may be uploaded as well. In some cases, the upload interface 600 may allow users to input various types of medical data and proceed with the data submission process.

[0076] Once the data is uploaded, the system may perform tissue analysis and classification on the received images. Fig. 7 shows a tissue section 700 comprising multiple tissue regions and structures, each representing a different classification by the system. The tissue section 700 may include a stained tissue region 702 displaying an artifact (A). Adjacent to the stained tissue region 702 may be a tissue sample 704 showing background within the stained tissue image.

[0077] In some cases, the system may identify and analyze various tissue structures within the tissue section 700. For example, the system may classify an artifact, as shown in tissue sample 706 and a tissue sample 708 classified as R1 within the section. A tissue sample 710 showing a classification of R2 may be positioned near a tissue sample 712 illustrating a classification of R3. The system may also classify a tissue sample 714 as R4, and a tissue sample 716 as R5. At the bottom portion of the tissue section 700, a tissue specimen 718may be classified as “Tissue (T)”. The R1-R5 classification may correspond to tissue coverage, layers, density, or some other tissue characteristics.

[0078] Once the tissues are classified as shown in Fig. 7, the web application 306 may process whole slide images (WSIs) to perform tumor segmentation and tissue WSIs to perform tumor segmentation and tissue classification. In some embodiments, the system classification employs deep learning models to analyze the tissue structures and identify regions to analyze the tissue structures and identify regions of interest. The system may segment the WSI of interest. The system may segment the WSI of interest. The system may segment the WSI into smaller patches and apply classification algorithms to each patch.

[0079] In some embodiments, the system may employ a system diagram 800 as shown in Fig. 8 for processing and analyzing image data. The system architecture diagram 800 may include an input image 802 that is provided to an input layer 804. The input layer 804 may process the input image and output data with specific dimensions.

[0080] The system diagram 800 may incorporate multiple functional models for parallel processing of the input data. A functional model 806 and a functional model 808 may receive input from the input layer 804 and generate respective outputs. The outputs from the functional models 806 and 808 may be combined in an average layer 810. The average layer 810 may process the inputs to generate a final output, potentially combining predictions from multiple models to produce a consolidated result.

[0081] The input layer 804 may receive the input image 802 and perform initial processing. In some cases, the input layer 804 may apply data augmentation techniques such as normalization and random scaled cropping to enhance the robustness of the analysis.

[0082] The processed data from the input layer 804 may be passed to multiple functional models, such as functional model 806 and functional model 808. These functional models may implement different deep learning architectures to analyze the image data. For example, functional model 806 may be a VGG16 architecture, while functional model 808 may be a ResNet50 architecture. The functional models may comprise any other suitable model for the system, including EfficientNet models.

[0083] For example, the VGG16 architecture is a CNN model comprising multiple convolutional and max-pooling layers, followed by fully connected layers. To expedite implementation, transfer learning is employed by utilizing pre-trained weights from an ImageNet dataset. The VGG16 model is initialized without its top classification layers and the weights of the convolutional layers are frozen to preserve the learned features. Custom fully connected layers were then added on top of the VGG16 architecture for classificationpurposes. During training, these custom layers are fine-tuned while keeping the convolutional layers frozen. This approach facilitated efficient training on our breast cancer histopathology dataset, resulting in accurate classification results.

[0084] In addition to VGG16, in some embodiments the ResNet50 architecture, another prominent CNN model, may be incorporated into the classification framework. ResNet50 is characterized by its deep structure with skip connections, which helped alleviate the vanishing gradient problem and enabled the training of very deep neural networks. Similar to the approach with VGG16, the pre-trained ResNet50 model is loaded without its classification layers and froze the weights of the convolutional layers to retain learned features. Custom fully connected layers were then added for classification purposes. By leveraging ResNet50, embodiments of the present disclosure are able to capture intricate features from the cancer histopathology images, potentially enhancing classification performance compared to shallower architectures.

[0085] The outputs from the functional models 806 and 808 may be combined in the average layer 810. This averaging operation may help improve the overall accuracy and reliability of the predictions.

[0086] In some embodiments, an ensemble learning approach is adopted by combining predictions from both the VGG16 and ResNet50 models to enhance predictive performances. The ensemble model leverages the predicted probabilities generated by the individual models. The implementation involves loading the pre-trained VGG16 and ResNet50 models and creating a new model to combine their outputs using averaging. The ensemble model may then be compiled with appropriate loss and optimization functions. By harnessing the complementary strengths of VGG16 and ResNet50, the ensemble model achieves enhanced classification accuracy and robustness, contributing to more reliable breast cancer histopathology image analysis.

[0087] In some embodiments, the system may employ a 5-fold cross-validation approach for training and testing the models. This technique may help ensure the robustness and generalizability of the diagnostic results.

[0088] The system may use various evaluation metrics to assess the performance of the models. In some embodiments, the Cohen Kappa Score may be used as an evaluation metric. This score may provide a measure of agreement between the model predictions and the ground truth labels, taking into account the possibility of agreement occurring by chance.

[0089] Fig. 9 is an exemplary screen of an embodiment of a web application for processing patient data. The web application may include various user interface components to facilitatedata input, analysis, and result visualization for healthcare professionals. In some cases, the web application may implement a scan interface 900, as illustrated in Fig. 9. The scan interface 900 may display scan completion results and provide a visual representation of diagnostic information.

[0090] The scan interface 900 may include a donut chart 902 visualization region positioned on the left side for displaying scan analysis results. In some cases, the donut chart 902 may present probability distributions for different cancer grades or severity levels. The scan interface 900 may also include a patient demographics section 904 positioned on the right side, in some emnbodiments. This section may contain fields for displaying patient information, including patient name, cancer detection results, and other related scores.

[0091] Fig. 10 illustrates a system interface 1000 for displaying and managing patient diagnostic information. The system interface 1000 may include a demographics module 1002, a summary module 1004, and a biopsy module 1006. In some cases, the demographics module 1002 may present patient demographic information such as age, height, weight, and BMI. The summary module 1004 may display natural language processing (NLP) summaries of patient information and visit details. The biopsy module 1006 may enable visualization and analysis of whole slide biopsy images.

[0092] The demographics module 1002, summary module 1004, and biopsy module 1006 may operate together to provide an integrated view of patient diagnostic data. The biopsy module 1006 may display tissue images while the demographics module 1002 and summary module 1004 may present corresponding patient context and visit information.

[0093] In some cases, the web application may implement a patient interface system 1100, as shown in Fig. 11. The patient interface system 1100 may include multiple modules for presenting comprehensive patient health information. A patient information module 1102 may display basic demographic and medical information such as age, height, weight, and BMI.

[0094] The patient interface system 1100 may also include a cancer assessment module 1104 that shows information related to cancer grading, including an average ISUP grade visualization. A visit summary module 1106 may present details from the patient's last visit, including medications and observed symptoms.

[0095] In some cases, the patient interface system 1100 may incorporate additional health assessment modules. A cerebrovascular assessment module 1108 may indicate cerebrovascular disease status. A hypertension assessment module 1110 may showhypertension status. A diabetes assessment module 1112 may display information about the patient's diabetes condition.

[0096] The modules 1102, 1104, 1106, 1108, 1110, and 1112 may be arranged in a grid layout to present patient health data in an organized manner. The modules may display their respective information through text, charts, and status indicators to provide a comprehensive overview of the patient's health status.

[0097] Referring now to Fig. 12, a schematic of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, computing node 10 is capable of being implemented and / or performing any of the functionality set forth hereinabove.

[0098] In computing node 10 there is a computer system / server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0099] Computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system / server 12 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0100] As shown in Fig. 12, computer system / server 12 in computing node 10 is shown in the form of a general-purpose computing device. The components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0101] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0102] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 12, and it includes both volatile and non-volatile media, removable and nonremovable media.

[0103] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may further include other removable / non-removable, volatile / non- volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0104] Program / utility 40, having a set (at least one) of program modules 42, may be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0105] Computer system / server 12 may also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices thatenable a user to interact with computer system / server 12; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 22. Still yet, computer system / server 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer system / server 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0106] The present disclosure may be embodied as a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0107] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD- ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiberoptic cable), or electrical signals transmitted through a wire.

[0108] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a localarea network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0109] Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PL A) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0110] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0111] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0112] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0113] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0114] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found inthe marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

What is claimed:

1. A web-based artificial intelligence system for multi-modal cancer diagnosis, comprising: a web application interface that receives uploads of patient data including biopsy images, electronic health records, and omics data; an ensemble of deep learning models including at least a first convolutional neural network and a second convolutional neural network that process the biopsy images at multiple magnification levels to generate cancer grade predictions; a multimodal data integration module that combines the cancer grade predictions with patient demographic information and clinical data from the electronic health records and omics data; and a visualization component that displays the cancer grade predictions as interactive charts with color-coded probability distributions for different cancer severity levels.

2. The web-based artificial intelligence system of claim 1, wherein the ensemble of deep learning models comprises one of a VGG16 convolutional neural network, a ResNet50 convolutional neural network, a vision transformer, and an EfficientNetO-7 convolutional neural network.

3. The web-based artificial intelligence system of claim 2, wherein the ensemble of deep learning models process biopsy images at magnification levels including 5x, lOx, 20x, 40x, lOOx, 200x, and 400x.

4. The web-based artificial intelligence system of claim 1, wherein the multimodal data integration module processes the omics data including DNA, RNA, protein, metabolite, and phenotype data.

5. The web-based artificial intelligence system of claim 1, wherein the visualization component includes interactive tabs displaying biopsy images, patient demographics, natural language processing summaries, and detailed summary information.

6. The web-based artificial intelligence system of claim 5, wherein the interactive tabs include a biopsy tab that displays whole-slide images with Al-annotated tumor regions marked at pixel and patch levels.

7. The web-based artificial intelligence system of claim 1, wherein the web application interface processes biopsy images by dividing each image into patches of different sizes including 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192 pixels.

8. A computer- implemented method for diagnosing cancer using multimodal artificial intelligence analysis, comprising: receiving patient data through a web interface, the patient data including whole slide biopsy images, electronic health records, and omics data; processing the whole slide biopsy images using an ensemble of convolutional neural networks trained on histopathological data to generate probability scores for multiple cancer grades;integrating the probability scores with patient demographic information and clinical history data extracted from the electronic health records using natural language processing; generating explainable diagnostic results that include spatial markup of tumor regions on the biopsy images and identification of predictive biomarkers from the omics data; and presenting the diagnostic results through an interactive web dashboard with visualization charts that display probability distributions for cancer subtype and severity levels using a color-coded system.

9. The computer-implemented method of claim 8, wherein the ensemble of convolutional neural networks comprises a VGG16 network and a ResNet50 network that process the whole slide biopsy images at multiple magnification levels including 5x, lOx, 20x, 40x, lOOx, 200x, and 400x.

10. The computer-implemented method of claim 9, wherein processing the whole slide biopsy images includes dividing each image into patches of varying sizes including 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192 pixels.

11. The computer-implemented method of claim 8, wherein the omics data includes DNA sequencing data, RNA expression data, protein analysis data, metabolite profiles, and phenotype information that are processed to identify cancer-specific biomarkers.ll. The computer-implemented method of claim 11, wherein generating explainable diagnostic results includes applying attention mechanisms from transformer architectures to identify which genomic features contribute most significantly to a cancer grade prediction.

13. The computer-implemented method of claim 8, wherein the natural language processing extracts clinical information from physician notes, pathology reports, laboratory results, and treatment history records to create patient summary profiles.

14. The computer-implemented method of claim 13, wherein the interactive web dashboard includes separate visualization tabs for displaying Al-annotated biopsy images, patient demographics, natural language processing summaries of clinical data, and identified biomarkers with associated confidence scores.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving multimodal patient data including digitized biopsy images, electronic health record data, and omics data through a web application interface; applying ensemble deep learning models to analyze the digitized biopsy images at multiple scales and magnifications to determine cancer subtype and grade classifications; processing the electronic health record data using natural language processing to extract relevant clinical features; combining outputs from the ensemble deep learning models with the extracted clinical features to generate comprehensive diagnostic predictions; and- Tigenerating interactive visualizations that display the diagnostic predictions as probability distributions with explainable artificial intelligence features including tumor region annotations and biomarker identification.

16. The non-transitory computer-readable medium of claim 15, wherein the ensemble deep learning models comprise one of VGG16 convolutional neural network, a ResNet50 convolutional neural network, an EfficientNet convolutional neural network and a vision transformer model that processes the digitized biopsy images at magnification levels including 5x, lOx, 20x, 40x, lOOx, 200x, and 400x.

17. The non-transitory computer-readable medium of claim 16, wherein the digitized biopsy images are divided into patches of varying sizes including 2x2, 4x4, 8x8, 16x16, 64x64, 128x128, 224x224, 240x240, 256x256, 260x260, 300x300, 380x380, 456x456, 512x512, 528x528, 600x600, 1024x1024, 2048x2048, 4096x4096, and 8192x8192 pixels for multi-scale analysis.

18. The non-transitory computer-readable medium of claim 15, wherein the omics data includes DNA sequencing data, RNA expression data, protein analysis data, metabolite profiles, and phenotype information that are processed to identify cancer-specific biomarkers.

19. The non-transitory computer-readable medium of claim 18, wherein the bio marker identification applies attention mechanisms from transformer architectures to determine which genomic features contribute most significantly to the cancer grade classifications.

20. The non-transitory computer-readable medium of claim 17, wherein the interactive visualizations include separate display tabs for Al-annotated biopsy images with tumor region markup, patient demographics, natural language processing summaries of clinical data, and identified biomarkers with associated confidence scores.

Images