Systems and methods for digital pathology and mitosis detection using artificial intelligence

Abstract

Systems and methods for digital pathology and mitosis detection in accordance with embodiments of the invention are illustrated. One embodiment includes a method for performing human-AI digital pathology. The method includes receiving at least one whole slide images (WSIs) of tissues, pre-processing the received WSIs, and generating meningioma grades for the received WSIs. The method further includes providing evidence justifying the generated meningioma grades, receiving updates regarding the provided evidence, and updating the meningioma grades based on the received updates.

Description

Systems and Methods for Digital Pathology and Mitosis Detection Using Artificial IntelligenceCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 625,137 entitled “Systems and Methods for Mitosis Detection and Quantification Using Digital Pathology” filed January 25, 2024, the disclosure of which is hereby incorporated by reference in its entirety for all purposes.STATEMENT OF FEDERAL SUPPORT

[0002] This invention was made with government support under 2047297, awarded by the National Science Foundation. The government has certain rights in the invention.FIELD OF THE INVENTION

[0003] The present invention generally relates to digital pathology and, more specifically, using machine learning to detect and classify meningioma and mitosis.BACKGROUND

[0004] Digital pathology is the practice of digitizing traditional glass slides of biological tissues using high-resolution scanners to create digital images for analysis and diagnosis. Digital pathology leverages image analysis software, artificial intelligence (Al), and cloud-based storage to streamline workflows, enhance accuracy, and improve collaboration among pathologists, researchers, and clinicians. Digital pathology can enable remote consultations, support advanced research, and facilitate personalized medicine by integrating data with other diagnostic tools.

[0005] Mitosis is the process of cell division that occurs in eukaryotic cells, which are cells that contain a nucleus and other membrane-bound organelles. During mitosis, the cell undergoes a series of highly coordinated and regulated steps that result in the formation of two genetically identical daughter cells. These steps include chromosome condensation, spindle formation, chromosome alignment, sister chromatid separation,and cytokinesis. Mitosis plays a critical role in a variety of biological processes, including growth, development, tissue repair, and maintenance of chromosome stability. Dysregulation of mitosis can lead to abnormal cell proliferation and contribute to the development of cancer and other diseases. As such, mitosis is a fundamental process in cell biology and a topic of extensive research in both basic and clinical science.

[0006] Machine learning is a subset of Al that allows computers to learn from data without being explicitly programmed. The technology has the potential to revolutionize medical diagnosis by leveraging the power of image analysis and deep learning algorithms to detect and classify abnormalities in medical images. By analyzing vast amounts of data and recognizing patterns that are invisible to the human eye, machine learning can help doctors make more accurate and timely diagnoses, leading to better patient outcomes.SUMMARY OF THE INVENTION

[0007] Systems and methods for digital pathology and mitosis detection in accordance with embodiments of the invention are illustrated. One embodiment includes a method for performing human-AI digital pathology. The method includes receiving at least one whole slide images (WSIs) of tissues, pre-processing the received WSIs, and generating meningioma grades for the received WSIs. The method further includes providing evidence justifying the generated meningioma grades, receiving updates regarding the provided evidence, and updating the meningioma grades based on the received updates.

[0008] In another embodiment, the method includes detecting mitoses on each of the pre-processed WSIs, determining a plurality of regions of interest (ROIs) on each of the pre-processed WSIs based on detected mitoses using a depth-first search (DFS) algorithm, generating a mitosis count (MC) for each of the plurality of ROIs, determining the locations of mitoses in each ROI, and generating meningioma grades based on the mitosis count and the locations of all mitoses on each of the pre-processed WSIs.

[0009] In a further embodiment, each of the plurality of ROIs comprises ten consecutive high-power fields (HPFs).

[0010] In still another embodiment, the mitoses are detected using a combination of classification models comprising a nuclei segmentation model, a necrosis classificationmodel, a sheeting classification model, a mitosis classification model, and a prominent nucleoli classification model.

[0011] In a still further embodiment, the DFS algorithm determines each of the plurality of ROIs by determining a combination of ten consecutive HPFs based on the outputs of the combination of classification models.

[0012] In yet another embodiment, the DFS algorithm arranges HPFs as a sequence of 10 connected HPFs that form a connected path, wherein each HPF is passed exactly once.

[0013] In a yet further embodiment, the DFS algorithm searches all possibilities of 10 connected HPF paths to obtain the highest MC.

[0014] In another additional embodiment, the DFS algorithm comprises recursive depth-first searching with backtracking.

[0015] In a further additional embodiment, the WSIs comprises slides of tissues stained with hematoxylin and eosin (H&E), slides of tissues stained with phosphorylated histone H3 (PHH3), and Ki-67 immunohistochemistry (IHC) tests.

[0016] In another embodiment again, the meningioma grade is generated based on a combination of criteria comprising nuclei count, necrosis probability, sheeting probability, mitosis, prominent nucleoli, and Ki-67 proliferation index.

[0017] In a further embodiment again, the generated mmWave transmit signal is transmitted directionally using directional beams.

[0018] One embodiment includes a non-transitory machine readable medium containing processor instructions for performing human-AI digital pathology, where execution of the instructions by a processor causes the processor to perform a process that includes receiving at least one whole slide images (WSIs) of tissues, pre-processing the received WSIs, generating meningioma grades for the received WSIs, providing evidence justifying the generated meningioma grades, receiving updates regarding the provided evidence, and updating the meningioma grades based on the received updates.

[0019] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized byreference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0021] Fig. 1 illustrates a process for human-AI digital pathology diagnosis in accordance with an embodiment of the invention.

[0022] Fig. 2 illustrates a process for enhanced mitosis detection and quantification in accordance with an embodiment of the invention.

[0023] Fig. 3 illustrates a workflow for mitosis annotation in accordance with an embodiment of the invention.

[0024] Figs. 4A-B illustrate a depth-first search (DFS) algorithm in accordance with an embodiment of the invention.

[0025] Fig. 5 illustrates examples of various pathological features and criteria for meningioma grading in accordance with an embodiment of the invention.

[0026] Fig. 6 illustrates joint analyses of multiple criteria in meningioma grading in accordance with an embodiment of the invention.

[0027] Fig. 7 illustrates an additional view of the interface of the digital pathology system according to the top-down workflow discussed above in accordance with an embodiment of the invention.

[0028] Fig. 8 illustrates a mid-level sample used in the analysis in accordance with an embodiment of the invention.

[0029] Fig. 9 illustrates examples of evidence provided to pathologists using the digital pathology system in accordance with an embodiment of the invention.

[0030] Fig. 10 illustrates a data processing pipeline for the digital pathology system in accordance with an embodiment of the invention.

[0031] Fig. 11 illustrates a detection device that can be utilized to perform various functions in mitosis detection and quantification in accordance with an embodiment of the invention.

[0032] Figs. 12A-C illustrate an artificial intelligence (Al) assisted enhanced mitosis detection and quantification software suite in accordance with an embodiment of the invention.DETAILED DESCRIPTION

[0033] Mitosis is a fundamental process in cell biology that plays a critical role in a variety of biological processes, including growth, development, tissue repair, and maintenance of chromosome stability. Dysregulation of mitosis can lead to abnormal cell proliferation and contribute to the development of cancer and other diseases. Mitosis quantification is an important task in the diagnosis of tumors, such as meningiomas, and is based on the histopathological assessment of mitotic activity. However, this task can be challenging due to the subjective nature of manual assessment and the potential for inter-observer variability. To overcome these challenges, digital pathology and machine learning techniques can be used to enhance the accuracy and objectivity of mitosis quantification in meningiomas. By leveraging the power of image analysis and deep learning algorithms, it is possible to detect and classify mitotic figures in digital images of tissue samples with high precision and reliability.

[0034] In recent years, artificial intelligence (Al) has been increasingly used in digital pathology to improve the detection and classification of mitosis and associated cellular activities. Machine learning algorithms, including deep learning and convolutional neural networks (CNNs), have been developed to automatically identify and quantify mitotic figures in digital images of tissue samples. These algorithms can analyze vast amounts of data and recognize patterns that are difficult to recognize by the human eye, leading to more objective and accurate results.

[0035] While Al-assisted pathology systems have the potential to detect mitosis more accurately and comprehensively, current methods in digital pathology, specifically in mitosis detection, are still limited in a variety of ways. First, the incorporation of Al could potentially skew the judgments of medical professionals. Diagnosing certain conditionsultimately includes a subjective element where medical professionals use their subjective judgment in view of objective evidence to render a medical diagnosis. For example, mitosis itself is a naturally occurring process. However, too many mitoses in a tissue sample could mean that the cells are undergoing dysregulated mitosis or mitotic error, which may indicate a possible cancerous condition. Whether there is actually dysregulated mitosis or mitotic error would require medical professionals to exercise their subjective judgment, and Al-based suggestions could affect the judgment of medical professionals, which may cause them to be either too conservative or less conservative in their diagnoses.

[0036] Additionally, current Al-assisted pathology detection systems typically determine mitosis hotspots on whole slide images (WSIs) as either circle, square, or rectangular regions, which may not completely align with the actual distribution of mitosis in tumors. As a result, the number of mitoses detected in these recommended hotspot areas may be fewer than it should be, as the morphology of these hotspots lacks flexibility and nuance. Further, current Al-assisted mitosis detection systems are trained on data from limited sources, such as specific patients, hospitals, and scanners. Consequently, the mitosis detection systems may not provide optimal performance when confronted with new and diverse data, resulting in less accurate detection suggestions.

[0037] Systems and methods in accordance with many embodiments of the invention can improve upon the above-discussed limitations by providing a comprehensive digital pathology system that leverages multiple models to identify potentially relevant features on WSIs of tissues. WSIs may be stained using conventional techniques in pathology, such as hematoxylin and eosin (H&E) staining and phosphorylated histone H3 (PHH3) staining. In many embodiments, systems and methods can pre-process WSIs by applying a sliding window technique over the input WSIs. Systems and methods utilize a combination of classification models to segment pre-processed WSIs and detect various features relevant to mitosis based on the World Health Organization (WHO) meningioma grading criteria. In several embodiments, systems and methods can classify each detected feature to produce an intermediate classification.

[0038] In numerous embodiments, systems and methods utilize the intermediate classifications to determine regions of interest (ROIs) on a WSI. Systems and methods can utilize a depth-first search (DFS) algorithm to determine ROIs by identifying ten consecutive high-power fields (HPFs) on WSIs of tissues. Identifying ten consecutive HPFs can quantify mitosis more effectively in a manner that aligns more closely with medical diagnosis, such as cancer prevention. Systems and methods can generate a mitosis count in each ROI such that users can quantitatively conclude the condition of the tissue. In certain embodiments, systems and methods can provide suggestions based on all the identified features and HPFs to provide as much information as possible to the user. In numerous embodiments, systems and methods can be implemented as a software suite where the features discussed above can be displayed on graphical user displays (GUIs). Software suites may be interactive to allow users to perform in-depth analysis of the processed WSIs. In several embodiments, users can further verify any of the system outputs described above.

[0039] Turning to the drawings, a process for human-AI digital pathology diagnosis in accordance with an embodiment of the invention is illustrated in Fig. 1. Process 100 receives (110) a plurality of WSIs. In many embodiments, the plurality of WSIs are images of tissues. WSIs may be slides of tissues that are stained with either H&E or PHH3 and scanned for pre-processing. WSIs in some embodiments are H&E tiles and Ki-67 immunohistochemistry (IHC) tests.

[0040] Several embodiments pre-process (120) the received WSIs. Preprocessing in accordance with several embodiments includes cropping, resizing, and applying sliding windows to the WSIs, such that the WSIs may be used as input for classification models. In some embodiments, WSIs are processed with 240x240x3 sliding windows. WSIs may also be processed using 96x96x3 sliding windows depending on the classification model through which the WSI tiles are passed.

[0041] Process 100 generates (130) meningioma grades for the received WSIs. Many embodiments report multiple Al-computed pathology criteria using various models, which are mandatory for diagnosis according to medical guidelines. Digital pathology diagnosis systems in accordance with several embodiments provide (140) evidencejustifying the generated grades. Evidence justifying the generated grades may be provided for each of the pathology criteria used to generate the grades.

[0042] Process 100 receives (150) updates regarding the generated grades. Updates may be received through input made by human experts regarding the evidence justifying the generated grades. In numerous embodiments, systems and methods are able to update the generated grades in real time as human experts confirm or overrule the provided evidence.

[0043] While specific processes for human-AI digital pathology diagnosis are described above, any of a variety of processes can be utilized to perform human-AI digital pathology diagnosis as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0044] In certain embodiments, systems and methods are able to utilize the DFS algorithm for accurate mitosis detection and quantification to generate more accurate meningioma grading. A process for enhanced mitosis detection and quantification in accordance with an embodiment of the invention is illustrated in Fig. 2. Process 200 detects (210) mitoses on pre-processed WSIs. In numerous embodiments, mitoses are detected using a combination of classification models, including but not limited to a nuclei segmentation model, a necrosis classification model, a sheeting classification model, a mitosis classification model, and a prominent nucleoli classification model. In certain embodiments, 240x240x3 tiles are passed through mitosis classification models. In some embodiments, 96x96x3 tiles are passed through prominent nucleoli classification models to predict prominent nuclei probability.

[0045] Process 200 determines (220) ROIs. In numerous embodiments, a DFS algorithm is used to determine ROIs. The DFS algorithm can take outputs from the classification models described above and determine a combination of ten consecutive HPFs. Ten consecutive HPFs, as a standard provided by the WHO, can provide medical professionals with a more comprehensive description of the tissue such that medicalprofessionals are able to better determine if the tissue is undergoing dysregulated mitosis. DFS algorithms in accordance with many embodiments can calculate the mathematically maximum possible MC in 10 consecutive HPFs of a slide, along with their corresponding locations.

[0046] Process 200 generates (230) mitosis counts based on ROIs. For each of the identified ROIs, systems and methods can generate the location and the number of mitoses within the identified ROIs.

[0047] While specific processes for enhanced mitosis detection and quantification are described above, any of a variety of processes can be utilized to detect and quantify mitoses as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0048] Fig. 3 illustrates a workflow for mitosis annotation in accordance with an embodiment of the invention. In numerous embodiments, systems and methods provide a mitosis annotation tool for medical professionals to create their own training data in mitosis detection. Systems and methods in accordance with various embodiments of the invention can provide a mitosis annotation interface. Users can annotate mitosis on input stained WSIs using the mitosis annotation interface, and the annotations can be further reviewed and confirmed by an expert.

[0049] Many embodiments utilize a two-step image transform approach to align PHH3 and H&E WSIs to assist pathologists in verifying mitoses on H&E, where image alignment procedures were performed on PHH3 WSIs, while the H&E slides remained unaltered. In several embodiments, H&E and PHH3 image tiles with a size of one HPF (size = 1 ,600 x 1 ,600 pixels, or 0.4mm = 0.16mm2)1 were extracted from WSIs without overlapping (Figure 1 (b)). Extracted image tiles in accordance with many embodiments are annotated with the locations of mitoses.

[0050] Many embodiments quantify the mitotic count (MC) in 10 HPFs using six different methods. Methods 1 - 3 can count MCs in consecutive 10 HPFs and arecompatible with the current definition of WHO CNS 5 Blue Book, while the HPFs in methods 4 - 6 are not necessarily connected. Background and out-of-focus regions may be identified by a deep-learning algorithm and were excluded. Non-tumor regions can be marked by a neuropathologist (M.H.) with Aperio Imagescope software (version 12.3.2.8013)2 and were not used for calculation.

[0051] A DFS algorithm in accordance with an embodiment of the invention is illustrated in Figs. 4A-B. In several embodiments, locations of all mitosis annotations of a WSI, regardless of whether they were annotated by pathologists or detected by Al, are known in order to run the DFS algorithm. Firstly, the WSI can be split into non-overlapping HPF tiles. Then, a matrix of MC for each HPF in the WSI, denoted as WSI, may be built. The coordinates of each element in the WSI matrix can represent the location of an HPF, and the value represents the MC in this HPF. In various embodiments, the algorithm FindMaxMC is the main function, and algorithm Search is the helper function. To calculate with DFS, the WSI matrix should be passed to FindMaxMC function, and it will return two values: (1 ) maxSum: the highest MC in 10 consecutive HPFs, and (2) maxPath: the locations of these 10 HPFs.

[0052] The first method can produce MC in 10 HPFs using the DFS algorithm. In several embodiments, this method arranges HPFs as a sequence of 10 connected HPFs that form a connected path. HPFs may be squares when viewed in digital pathology, therefore, two HPFs that share at least one connected edge are considered connected. A path from the first HPF to the tenth, where each HPF is visited exactly once, may be formed, and the path can be calculated by the DFS algorithm. DFS algorithms in accordance with many embodiments include a recursive depth-first searching with backtracking. In a WSI, DFS algorithms can comprehensively search all possibilities of paths of 10 connected HPFs satisfying the constraint mentioned above such that the resulting number of MC is within the HPFs is maximized. In several embodiments, DFS algorithms prioritize exploring possible paths as deeply as possible into a given WSI from a given starting point. Once all potential paths have been fully explored, the algorithm can “backtrack” to the previous node to investigate alternative routes.

[0053] The second method can quantify MC in 10 linear HPFs. This method can quantify the highest MC of 10 HPFs with the linear arrangement Under this method, the highest MCs in 10 vertical HPFs and 10 horizontal HPFs may be reported.

[0054] The third method can quantify the highest MC of 10 HPFs under a rectangular arrangement. Under this method, the highest MCs in 5 x 2 HPFs in the vertical direction and 2 x 5 HPFs in the horizontal direction may be reported.

[0055] The fourth method can quantify the average number of MCs per 10 HPFs by dividing the total MC in tumor areas by the size of the tumors and multiplying by 10.

[0056] The fifth method may randomly sample 10 HPFs from each WS1 1 ,000 times without replacement and report the distribution of MCs in each set of 10 sampled HPFs.

[0057] The sixth method can rank the MC of each HPF in the entire WSI in descending order. The MC of the top 10 HPFs may be added together to obtain the maximum possible MC that can be observed in 10 HPFs within a WSI.Meningioma Grading

[0058] In many embodiments, digital pathology systems can correctly identify different grades of meningioma. Meningioma grading is complex as it typically involves multiple morphological and immunohistological features utilizing at least two kinds of pathology tests, including H&E slides and Ki-67 IHC tests, while also requiring pathologists to examine alternate high and low magnification images to detect large structures (i.e., brain invasion) or small events (i.e., mitosis). Further, pathologists need to examine the entire tumor, which may at times require pathologists to examine as many as 20 or more slides to identify frequently rare features (i.e., spontaneous necrosis).

[0059] According to the WHO guidelines (2016), meningiomas can be graded as Grade 1 , Grade 2, or Grade 3. The current grading of meningioma in the new WHO guideline (2021 ) still recommends the same criteria for grading, although the nomenclature is slightly different. Additionally, new molecular alterations are added to determine the tumor grade. The accurate grading of meningioma is vital for treatment planning, as Grade 1 tumors can be treated with either surgery or external beam radiation, while Grade 2 / 3 ones often need both treatments. Meanwhile, research shows thatpatients with Grade 3 meningiomas suffer a higher recurrence rate as well as a lower survival rate in comparison to Grade 2 patients.

[0060] Traditionally, pathologists need to search and locate features across various magnifications with optical microscopes or digital interfaces in order to determine the tumor grade. Specifically, they first localize the regions of interest (ROIs) in low magnification (x40), then switch to the patch level with a higher magnification (x100), and sometimes zoom further with the highest magnification (x400) to examine cellular architecture. These steps are usually repeated multiple times until pathologists have collected sufficient findings to conclude a grading and sign-out of the case. Fig. 5 illustrates examples of various pathological features and criteria for meningioma grading in accordance with an embodiment of the invention. Pathologists’ work starts with the H&E slides in Fig. 5a. Apart from the H&E, Ki-67 IHC tests can also be used to provide an estimated proliferation index, as illustrated in Figs. 5d and 5k, which is highly correlated to meningioma grading.

[0061] In several embodiments, resected tissues are stained with H&E solution, and users can locate mitotic cells in HPFs (illustrated in Fig. 5c) with the assistance of additional Ki-67 stains. Five pathological patterns, including hypercellularity (an abnormal excess of cells illustrated in Fig. 5f), prominent nucleoli (enlarged nucleoli pointed by the arrow in Fig. 5g), sheeting (loss of ‘whirling’ architecture illustrated in Fig. 5h), necrosis (irreversible injury to cells marked in the red box in Fig. 5i), and small cells (tumor cell aggregation with high nuclear / cytoplasmic ratio marked in the red box in Fig. 5j), may contribute to the grading of meningioma.

[0062] The WHO guidelines for meningioma grading include:• Grade 1 (benign) meningiomas include “histological variant other than clear cell, chordoid, papillary, and rhabdoid” with some exceptions and a lack of criteria for grade 2 and 3 meningiomas.• Grade 2 (formerly called atypical) meningiomas are recognized by meeting at least one of the four following criteria:1 ) The presence of > 2.5 mitoses / mm2(equating to > 4 mitoses per / 10 HPF of 0.16 mm2. Moreover, since mitoses are challenging to recognize inH&E, the Ki-67-positive nuclei (Fig. 5k) in the corresponding areas of Ki- 67 (Fig. 5d) are often compared for disambiguation;2) At least three out of five following histopathological features are observed: hypercellularity — an abnormal excess of cells in the specimen (Fig. 5f), prominent nucleoli — enlarged nucleoli in a cell (usually as a cluster) (Figs. 5g and 5m), sheeting — loss of ‘whirling’ architecture (Fig. 5h), necrosis — irreversible injury to cells (Fig. 5i), and small cell — cluster of cells with high nuclear / cytoplasmic ratio (Fig. 5j);3) Brain invasion — invasive tumor cells within the brain tissue are observed (Fig. 5e);4) The dominant appearance of a clear cell or chordoid subtype.• Grade 3 meningiomas are decided if at least one of the following criteria is met:1 ) Mitotic figures of > 12.5 mitoses / mm2(equal to > 20 mitoses / 10 HPF of 0.16 mm2);2) The appearance of frank anaplasia, papillary or rhabdoid subtype with some exceptions;3) Molecular alterations, such as a TERT promoter mutation and / or homozygous CDKN2A and / or CDKN2B deletion.

[0063] In view of the rigorous requirements for meningioma grading and mitosis detection, in various embodiments, digital pathology systems perform Al pathology through joint analyses of multiple criteria and explanations by hierarchically traceable evidence. Fig. 6 illustrates joint analyses of multiple criteria in meningioma grading in accordance with an embodiment of the invention.

[0064] In many embodiments, digital pathology systems are designed to follow the WHO guidelines and utilize Al to compute eight mandatory criteria for meningioma grading. These criteria can be split into two categories: quantitative and qualitative. For the quantitative criteria, such as mitotic count and Ki-67 proliferation index, systems and methods in accordance with various embodiments can predict these quantitative values directly. For the other criteria dealing with the presence or absence of a specific pathological pattern, digital pathology systems can provide recommendations of regions of interest (ROI) hotspots according to the largest aggregations of Als’ probabilities.

[0065] Fig. 6 demonstrates the interface of multiple criteria, which shows the current suggested grading for the tumor and a structured overview of each criterion. In several embodiments, digital pathology systems display an arrow to indicate the main contributing criterion, as illustrated in Fig. 6c, as the most deterministic Al findings for the suggested diagnosis, according to the meningioma grading guidelines. For example, in Fig. 6, “mitotic count” was suggested as the main contributing criterion, because 12 mitoses were detected in 10 HPFs. Such Al findings directly satisfy descriptions of WHO grade 2 meningiomas, making “mitotic count” the main contributing criterion. Going beyond the main contributing criterion, all the criteria may be linked with the evidence or regions of interest related to the findings. Moreover, Al’s recommendation on all the criteria can be confirmed and refined by the pathologist. Several embodiments use color bars to indicate the status. As illustrated in Figs. 6e, 6f, and 6g, red indicates a confirmed abnormal criterion (or presence), green indicates a confirmed normal criterion (or absence), orange indicates that the criterion is unconfirmed / uncertain (neither presence nor absence), and gray indicates the criterion is not applicable in this case. Once a pathologist overrides the result of any criterion (Fig. 6d), the color bar and the final grading can be updated correspondingly. Joint analyses of multiple criteria can address the challenge of comprehensiveness by providing mandatory criteria information for pathologists according to the medical guidelines. Systems and methods in accordance with many embodiments can achieve global explainability by presenting how different Al- computed criteria are combined to arrive at a diagnosis. Such a design can enhance Al’s workflow integration because it exposes the pathologist to high-level Al findings when they onboard the case. As such, they can establish an initial understanding and develop hypotheses, which also facilitates them to double-check with their examination later.

[0066] In numerous embodiments, systems and methods include top-down, hierarchically traceable evidence to justify the Al's grading. Such a workflow can mimic a common scenario found in actual practice where pathologists assign low-level tasks (e.g., marking ROIs, finding specific criteria) to trainees in practice. They can continue to perform a differential diagnosis (i.e., building hypotheses and ruling out less-probable cases with findings) based on trainees’ reports. By replacing trainees with Al, digitalpathology systems emulate the relationship between the pathologists and trainees, thus making Al integral to pathologists’ current practices.

[0067] Fig. 7 illustrates an additional view of the interface of the digital pathology system according to the top-down workflow discussed above in accordance with an embodiment of the invention. Users may start from the automatically-generated suggested grading result (Fig. 7a) and then examine the main contributing criterion (Fig. 7b). Users can further examine (Fig. 7c) the evidence list and register back into the original whole slide image in higher magnifications (Figs. 7d and 7e). Users might repeat the same workflow multiple times for examining other criteria (one criterion for each time). Meanwhile, suggested gradings may be updated as the user justifies Al’s findings.

[0068] Various embodiments present a hierarchical trace of evidence for each criterion with positive findings, which allows a pathologist to see a list of mid-level samples (Fig. 7c) that lead to a computed criterion (Fig. 7b). For the most important criterion — mitosis, digital pathology systems can provide a series of explanations in each mid-level sample. A mid-level sample used in the analysis in accordance with an embodiment of the invention is illustrated in Fig. 8, which includes probability (Fig. 8a), Al’s confidence level (Fig. 8b) that can be calculated by a probability threshold, and a saliency map that may be calculated by a Grad-CAM++ algorithm (Fig. 8c) that highlights the spatial support for the mitosis class in the reference image, allowing pathologists to check Al’s validity on each sample quickly. The probability threshold may be decided by a set of fixed threshold values. For example, if the thresholds are set as 0.4 (not confident), 0.6 (medium confident), 0.8 (high confident). Then, an Al probability of 0.7 should be “medium confident” because the probability is higher than 0.6 but lower than 0.8.

[0069] Further, at the low level, systems and methods in accordance with many embodiments support registering each sample of the mid-level evidence into the whole slide image (WSI) to enable pathologists to examine with higher magnification and search nearby for more contextual information (Figs. 7d and 7e). With the provided mid- and low- level information, a pathologist can approve / decline / declare uncertain a sample for a criterion with one click (Fig. 7f) or directly override Al’s results on each criterion (Fig. 7g). Correspondingly, the overall suggested grading (Fig. 7a) may be updated dynamically upon the user’s input. Such a diagnosis-contesting workflow allows pathologists tochallenge Al’s suggested diagnosis by seeing Al’s reasoning line and evidence, which increases the “contestability” as described in previous HCI research in healthcare.

[0070] Examples of evidence provided to pathologists using the digital pathology system in accordance with an embodiment of the invention are illustrated in Fig. 9. Evidence in accordance with many embodiments includes a highest focal region sampling result of mitotic count on the H&E slide, which is represented by the HPF in Fig. 9a, including multiple smaller boxes that identify the mid-level samples and locations of mitoses inside the mid-level samples. Evidence may further include a highest focal region sampling result on the Ki-67 IHC slide (Fig. 9b), a highest region sampling result of mitotic count on the H&E slide (Fig. 9c), a highest focal region sampling result on the Ki-67 IHC slide (Fig. 9d), a hypercellularity ROI sample (Fig. 9e), a necrosis ROI sample (Fig. 9f), a small cell ROI sample (Fig. 9g), and a prominent nucleoli ROI sample (Fig. 9h).

[0071] In several embodiments, digital pathology systems provide two “shortcuts” for pathologists to review generated results for the mitosis-related criteria (i.e., mitotic count and Ki-67 proliferation index). The first shortcut is the highest region sampling of WSIs. One WHO criterion in determining meningioma grading is “mitotic count in 10 consecutive HPFs”. However, the inter-observer consistency of “10 consecutive HPFs” is low. To address this problem, various embodiments provide a highest region sampling tool. The highest region in accordance with many embodiments is defined as a 2 x 5 HPF area with the highest number of mitotic counts (Fig. 9c) or the highest Ki-67 proliferation index (Fig. 9d). This tool speeds up a pathologist’s work by helping them locate 10 consecutive HPFs from the WHO guidelines.

[0072] The second shortcut is highest focal region sampling. High-grade meningiomas share a common feature of increased mitotic activities in a localized area. Hence, many embodiments provide a highest focal sampling tool to help pathologists better localize highly concentrated mitosis / Ki-67 proliferation index areas. The highest focal region may be calculated as the one HPF with the highest number of mitotic counts (Fig. 9a) or the highest Ki-67 proliferation index (Fig. 9b). Using this tool, pathologists can locate the foci of highly mitotic areas that the highest region sampling might miss.

[0073] In some embodiments, digital pathology systems generate heatmaps for WSIs. Pathologists can go beyond the sampled areas and navigate the high-heat areasusing heatmaps generated for the whole slide. For example, the mitosis heatmap registers all Al-detected positive mitotic cells as a mitotic density atlas, where high-heat areas indicate a high density of mitotic cells. As such, the heatmap can serve as a ‘screening tool’ to help pathologists filter out unrelated areas and rapidly narrow down to the ROIs that are scattered in an entire WSI. After pathologists finish examining one criterion, they can proceed to justify the rest of the criteria with the same top-down workflow (one iteration for each criterion). During such an iterative process, systems and methods in accordance with numerous embodiments update Al’s findings on individual criteria and, if necessary, the overall suggested grading as well. Finally, pathologists can make a diagnosis once they have collected sufficient confidence for the grading diagnosis.

[0074] Various embodiments implement an Al-aided pathology image processing backend to compute the eight pathological criteria of the mitotic count, Ki-67 proliferation index, hypercellularity, necrosis, small cell, prominent nucleoli, sheeting, and brain invasion. Digital pathology systems in accordance with many embodiments screen the entire WSI using six Al models and two rules, one for each criterion, to general initial Al results. For each WSI, many embodiments use a sliding window technique to cut it into smaller tiles.

[0075] A data processing pipeline for the digital pathology system in accordance with an embodiment is illustrated in Fig. 10. Systems and methods in accordance with various embodiments take H&E and Ki-67 WSIs as input and use a sliding window method to acquire H&E and Ki-67 tiles. Each H&E tile may be processed with resizing, a sliding window (240 x 240 x 3), and another sliding window (96 x 96 x 3) to fit the inputs of the down-stream Al models. For each tile, several embodiments utilize a series of Al models to calculate at least six criteria, including but not limited to nuclei count (Fig. 10f), necrosis probability (Fig. 10g), sheeting probability (Fig. 10h), mitosis (Fig. 10i), prominent nucleoli (Fig. 10j), and Ki-67 proliferation index (Fig. 10m).

[0076] For each pre-processed H&E tile, systems and methods in accordance with many embodiments use a nuclei segmentation model to count the number of nuclei (for hypercellularity judgment), a necrosis classification model to calculate necrosis probability, and a sheeting classification model to calculate sheeting probability. Nucleicounting results may be used for small cell recommendation and brain invasion visualization. For a 240x240x3 tile, systems and methods in accordance with several embodiments use a mitosis classification model to obtain the mitosis probability. A prominent nucleoli classification model may be used on 96 x 96 x 3 tiles to predict prominent nuclei probability. For each Ki-67 tile, systems and methods in accordance with many embodiments detect positive and negative nuclei to calculate the Ki-67 scores. In numerous embodiments, digital pathology systems can calculate ROIs based on all Al- computed results.

[0077] Based on the Al-computed nuclei count, many embodiments utilize two rules to support the reporting of small cell and brain invasion patterns. In selected embodiments, digital pathology systems can recommend small cell tiles based on the nuclei count of each tile (Fig. 10k). In certain embodiments, digital pathology systems can visualize brain invasion by classifying the brain vs. tumor regions according to the nuclei count (Fig. 101). This is because meningioma tumor areas usually have a high nuclei density, while normal brain tissues are not. After the Al models have processed each tile, digital pathology systems can calculate the ROIs.

[0078] Since there were no pre-trained models nor public meningioma datasets for the pathology patterns of mitosis, necrosis, prominent nucleoli, and sheeting, several embodiments utilize an in-house dataset built based on 30 WSIs (WSI total size - ~ 54.9 GB) to train Al models to classify these four patterns. The WSIs were scanned by an Aperio CS2 scanner in x400 magnification (pixel size=0.25 / zm). The ground truth labels were collected in two ways. For mitosis, the pathologist labeled ground truths with an online labeling system. For other criteria, the pathologist marked ROIs using the Imagescope software. Labeled ROIs can be cropped with a random-crop technique, and the tiles in different sets were generated from different groups of ROIs. In sum, the final dataset has a size of approximately 16.1 GB, consisting of four training and testing sets covering the four pathology patterns.

[0079] For each criterion, a subset of the training set was selected to be the validation set to train the models. Specific thresholds were decided by the maximum F1 scores achieved by each model in the validation set. Digital pathology systems can achieve F1 scores of 0.755, 0.904, 0.763, and 0.946 in identifying the pathologicalpatterns of mitosis, necrosis, prominent nucleolus, and sheeting. The scores indicate the effectiveness of the models. Moreover, for the tasks of cell-counting in hypercellularity and Ki-67 proliferation index criteria, the average error rates of nuclei counting and Ki-67 proliferation index were 12.08% and 29.36%, respectively, when tested with 150 randomly selected 512 x 512 x 3 tiles each.

[0080] Fig. 11 illustrates a detection device that can be utilized to perform various functions in mitosis detection and quantification in accordance with an embodiment of the invention.

[0081] Detection device 1100 includes a processor 1110. Processor 1110 may direct detection application 1142 to detect mitosis from input images. Detection application 1142 can be stored in memory 1140. Image data of whole slide images with potential mitosis can be stored in image data 1144. Memory 1140 may include model data 1146 for applications that interface with models for detection and / or training purposes.

[0082] In many embodiments, processor 1110 can include a processor, a microprocessor, a controller, or a combination of processors, microprocessor, and / or controllers that perform instructions stored in memory 1140 to perform mitosis detection and quantification. Processor instructions can configure the processor 1110 to perform processes in accordance with certain embodiments of the invention. In various embodiments, processor instructions can be stored on a non-transitory machine-readable medium. Detection device 1100 further includes a user interface 1120 that can receive user inputs on running the detection application. Detection device 1100 may further include a network interface 1130 that can receive various types of data from external sources.

[0083] Although a specific example of a detection device is illustrated in this figure, any of a variety of detection devices can be utilized in mitosis detection and quantification similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

[0084] An Al-assisted enhanced mitosis detection and quantification software suite, hereinafter referred to as MitoPath in accordance with an embodiment of the invention is illustrated in Figs. 12A-C.

[0085] In various embodiments, MitoPath displays multiple flexible-morphology ROI recommendations where users can select one for an in-depth examination. MitoPath can direct the user to the first area within the selected ROI and highlight mitosis events identified by the Al. MitoPath may offer enriched explanations of each Al-identified mitosis event. Upon completion of an area’s review, MitoPath in accordance with several embodiments navigates the user to the next area within the ROI, continuing this process until all areas have been visited. MitoPath can keep track of and report the sum of user- confirmed mitosis events after the user examines all areas of an ROI.

[0086] In many embodiments, the software suite leverages the DFS algorithm to determine ten consecutive HPFs and quantify ROIs based on the ten consecutive HPFs. These determinations can be adjusted based on different sensitivities as requested by the user. Once a user selects a DFS-quantified ROI, the software suite can guide the user to view each HPF within the ROI. In several embodiments, systems and methods outline the detected mitoses with boxes surrounding the potential mitoses. Users can perform this same process throughout the WSI to inspect all potential ROIs and mitoses. In certain embodiments, users can determine whether the Al-determined mitoses are, in fact, mitoses. Systems and methods in accordance with many embodiments can provide a variety of evidence to support their determination of ROIs and mitoses.

[0087] As shown in Fig. 12A, the interface of MitoPath consists of two parts: a pathology scan viewer on the left that can zoom up to 400x magnification in tumor scans, and an Al assistance panel on the right, which includes three feature groups. Al assistance panels in accordance with many embodiments include recommendation settings where users can toggle the visibility of MitoPath’s ROI and mitosis recommendations using the “Show ROI Recommendations” and “Show Mitosis Recommendations” switches, respectively. The “Number of ROIs” slider allows users to determine the number of visible ROIs. Each ROI is generated using the DFS algorithm. Several embodiments provide a flexible morphology and contain the maximum number of Al-detected mitosis events at a specified sensitivity level, adjustable via the “Al Sensitivity” slider.

[0088] As illustrated in Fig. 12B, users can examine areas within each ROI at higher magnification using “NEXT 1 HPF” to move to the next area and “PREV HPF” toreturn to the previous area. Once all areas have been visited, the user can click on the “PREV LEVEL” button to return to the lower magnification view. In many embodiments, users can record the number of confirmed mitosis events in each area of the ROI using the mitosis counter. MitoPath in accordance with various embodiments maintains a record of user-confirmed mitosis counts across areas within each ROI and can provide a total count once the user has finished examining all areas.

[0089] Fig. 12C demonstrates that MitoPath can present an evidence card for every Al-identified mitosis event, which includes four key XAI components, offering a holistic view of the Al’s findings. Evidence cards in accordance with several embodiments include a saliency map, where the hotspot indicates the spatial location of the detected event. Evidence cards may further include a probability score and a trust score, reflecting Al’s confidence, and a bar plot showcasing potential subclasses of the mitosis event (i. e. , pro-phase, meta-phase, ana / telo-phase, atypical and not mitosis), providing the user with a more granular categorization of the event. In some embodiments, evidence cards also include a set of (10) similar instances retrieved from an annotated dataset that includes paired H&E IHC staining, enabling pathologists to compare and make informed judgments.

[0090] A typical workflow with MitoPath begins as users enable the visibility of ROIs at low magnification. Users can control the number of displayed ROIs and Al sensitivity settings. Subsequently, users may select an ROI to examine more closely at a higher magnification. MitoPath can automatically transition into the first area of the chosen ROI. At this point, users can toggle on the “Show Mitosis Recommendations” feature to view potential mitosis events detected by the Al. Users can click on any mitosis recommendation box and view the evidence card detailing the Al detection. In this step, users can record the number of confirmed mitosis events in the examined area using the provided mitosis counter.

[0091] Upon completing the review of an area, users can proceed to the next one by clicking the “NEXT 1 HPF” button, which prompts MitoPath to guide them to the next area within the ROI. After completing the examination of all areas within an ROI, MitoPath may provide a total count of the user-recorded mitosis counts within that ROI, which aligns directly with the diagnosis guideline provided by the WHO.

[0092] Users can repeat the above-mentioned workflow for other ROIs. For regions not covered by any ROIs, users can enable the mitosis heatmap for manual navigation and to view Al-marked mitotic events. Users may continue this process until they have collected sufficient information for diagnosis and are ready to sign out the case.

[0093] Although specific methods of enhanced mitosis detection and quantification are discussed above, many different detection and quantification methods can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:1 . A method for performing human-AI digital pathology, the method comprising: receiving at least one whole slide images (WSIs) of tissues; pre-processing the received WSIs; generating meningioma grades for the received WSIs; providing evidence justifying the generated meningioma grades; receiving updates regarding the provided evidence; and updating the meningioma grades based on the received updates.

2. The method of claim 1 , wherein generating meningioma grades comprises: detecting mitoses on each of the pre-processed WSIs; determining a plurality of regions of interest (ROIs) on each of the pre-processed WSIs based on detected mitoses using a depth-first search (DFS) algorithm; generating a mitosis count (MC) for each of the plurality of ROIs; determining the locations of mitoses in each ROI; and generating meningioma grades based on the mitosis count and the locations of all mitoses on each of the pre-processed WSIs.

3. The method of claim 2, wherein each of the plurality of ROIs comprises ten consecutive high-power fields (HPFs).

4. The method of claim 2, wherein the mitoses are detected using a combination of classification models comprising a nuclei segmentation model, a necrosis classification model, a sheeting classification model, a mitosis classification model, and a prominent nucleoli classification model.

5. The method of claim 4, wherein the DFS algorithm determines each of the plurality of ROIs by determining a combination of ten consecutive HPFs based on the outputs of the combination of classification models.

6. The method of claim 2, wherein the DFS algorithm arranges HPFs as a sequence of 10 connected HPFs that form a connected path, wherein each HPF is passed exactly once.

7. The method of claim 6, wherein the DFS algorithm searches all possibilities of 10 connected HPF paths to obtain the highest MC.

8. The method of claim 2, wherein the DFS algorithm comprises recursive depth-first searching with backtracking.

9. The method of claim 1 , wherein the WSIs comprises slides of tissues stained with hematoxylin and eosin (H&E), slides of tissues stained with phosphorylated histone H3 (PHH3), and Ki-67 immunohistochemistry (IHC) tests.

10. The method of claim 1 , wherein the meningioma grade is generated based on a combination of criteria comprising nuclei count, necrosis probability, sheeting probability, mitosis, prominent nucleoli, and Ki-67 proliferation index.

11. A non-transitory machine readable medium containing processor instructions for performing human-AI digital pathology, where execution of the instructions by a processor causes the processor to perform a process that comprises: receiving at least one whole slide images (WSIs) of tissues; pre-processing the received WSIs; generating meningioma grades for the received WSIs; providing evidence justifying the generated meningioma grades; receiving updates regarding the provided evidence; and updating the meningioma grades based on the received updates.

12. The non-transitory machine readable medium of claim 11 , wherein generating meningioma grades comprises: detecting mitoses on each of the pre-processed WSIs; determining a plurality of regions of interest (ROIs) on each of the pre-processed WSIs based on detected mitoses using a depth-first search (DFS) algorithm; generating a mitosis count (MC) for each of the plurality of ROIs; determining the locations of mitoses in each ROI; and generating meningioma grades based on the mitosis count and the locations of all mitoses on each of the pre-processed WSIs.

13. The non-transitory machine readable medium of claim 12, wherein each of the plurality of ROIs comprises ten consecutive high-power fields (HPFs).

14. The non-transitory machine readable medium of claim 12, wherein the mitoses are detected using a combination of classification models comprising a nuclei segmentation model, a necrosis classification model, a sheeting classification model, a mitosis classification model, and a prominent nucleoli classification model.

15. The non-transitory machine readable medium of claim 14, wherein the DFS algorithm determines each of the plurality of ROIs by determining a combination of ten consecutive HPFs based on the outputs of the combination of classification models.

16. The non-transitory machine readable medium of claim 12, wherein the DFS algorithm arranges HPFs as a sequence of 10 connected HPFs that form a connected path, wherein each HPF is passed exactly once.

17. The non-transitory machine readable medium of claim 16, wherein the DFS algorithm searches all possibilities of 10 connected HPF paths to obtain the highest MC.

18. The non-transitory machine readable medium of claim 12, wherein the DFS algorithm comprises recursive depth-first searching with backtracking.

19. The non-transitory machine readable medium of claim 11 , wherein the WSIs comprises slides of tissues stained with hematoxylin and eosin (H&E), slides of tissues stained with phosphorylated histone H3 (PHH3), and Ki-67 immunohistochemistry (IHC) tests.

20. The non-transitory machine readable medium of claim 11 , wherein the meningioma grade is generated based on a combination of criteria comprising nuclei count, necrosis probability, sheeting probability, mitosis, prominent nucleoli, and Ki-67 proliferation index.

Images