Microscope slide image based machine learning image analysis for inflammatory bowel disease

Abstract

The present disclosure provides a method that includes determining, within an image of a biological sample from a patient's intestine, a plurality of image tiles depicting a portion of the biological sample. Based at least on the plurality of image blocks, a plurality of groups of bowel disease indications corresponding to a subset of the plurality of image blocks may be determined. A group-level histological score may be generated for each of the plurality of groups of bowel disease indications based at least on the subset of the plurality of image tiles included in each respective group. Based on histological scores of the generated group levels for each group of bowel disease indications, an aggregated histological score indicative of a disease load in the intestine of the patient may be generated. Related systems and computer program products are also provided.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 358,019 filed on July 1, 2022 and U.S. Provisional Patent Application No. 63 / 387,467 filed on December 14, 2022, the disclosures of which are incorporated by reference in their entireties. Technical Field

[0003] The subject matter described herein relates generally to digital pathology and, more specifically, to machine learning-based image analysis of microscope slide images for inflammatory bowel disease. Background Art

[0004] Inflammatory bowel disease (IBD), which includes the distinct entities of Crohn's disease and ulcerative colitis (UC), is a common disease with a high rate of poor long-term prognosis in terms of steroid dependence, hospitalization, and surgery, but few effective treatment options are available. UC affects the superficial layers of the intestine and mucosa and invariably involves the rectum. UC can also extend proximally to varying degrees to involve some or all of the colon. It is challenging to objectively assess the disease burden of IBD clinically, making it difficult to reliably determine the efficacy of various treatment options. Therefore, biological insights into IBD and reliable and reproducible methods for assessing the pathophysiology and disease burden of IBD remain crucial for identifying effective new therapeutic treatments, improving patient monitoring, etc. Summary of the invention

[0005] Systems, methods and articles of manufacture, including computer program products, for image-based (e.g., full-scan slide image-based) machine learning analysis of IBD are provided. In some exemplary embodiments, a system including at least one processor and at least one memory is provided. The at least one memory may include program code that provides operations when executed by at least one processor. These operations may include: determining multiple image blocks within an image of a biological sample from the patient's intestine. Each of the multiple image blocks depicts a portion of the biological sample. The operations also include determining multiple intestinal disease indicator groups based at least on the multiple image blocks. Each intestinal disease indicator group in the multiple intestinal disease indicator groups corresponds to a subset of the multiple image blocks. The operations also include generating a group-level histological score for each intestinal disease indicator group in the multiple intestinal disease indicator groups, based at least on a subset of the multiple image blocks contained in each corresponding group. The operations also include generating an aggregated histological score for the biological sample based on the generated group-level histological score for each intestinal disease indicator group. The aggregated histological score indicates the disease load in the intestine of the patient.

[0006] On the other hand, a computer-implemented method includes determining a plurality of image blocks within an image of a biological sample from the intestine of a patient. Each of the plurality of image blocks depicts a portion of the biological sample. The method also includes determining a plurality of intestinal disease indication groups based at least on the plurality of image blocks. Each of the plurality of intestinal disease indication groups corresponds to a subset of the plurality of image blocks. The method also includes generating a group-level histological score for each of the plurality of intestinal disease indication groups based at least on a subset of the plurality of image blocks contained in each corresponding group. The method also includes generating an aggregated histological score for the biological sample based on the generated group-level histological score for each intestinal disease indication group. The aggregated histological score indicates the disease burden in the intestine of the patient.

[0007] On the other hand, a computer program product is provided, which includes a non-transitory computer-readable medium storing instructions. The instructions may cause operations that can be performed by at least one data processor. These operations may include: determining multiple image blocks within an image of a biological sample from the intestine of a patient. Each image block in the multiple image blocks depicts a portion of the biological sample. The operations also include determining multiple intestinal disease indication groups based at least on the multiple image blocks. Each intestinal disease indication group in the multiple intestinal disease indication groups corresponds to a subset of the multiple image blocks. The operations also include generating a group-level histological score for each intestinal disease indication group in the multiple intestinal disease indication groups, based at least on a subset of the multiple image blocks contained in each corresponding group. The operations also include generating an aggregated histological score for the biological sample based on the group-level histological score generated for each intestinal disease indication group. The aggregated histological score indicates the disease load in the intestine of the patient.

[0008] In some variations, one or more features disclosed herein (including the following features) may optionally be included in any feasible combination of systems, methods, and / or non-transitory computer-readable media. In some variations, the group-level histology score and the aggregated histology score are each one of a Nancy Histology Index (NHI) score, a Robarts Histopathology Index (RHI) score, a Geboes scale score, and a global histology activity score (GHAS).

[0009] In some embodiments, the group-level histology score and the aggregated histology score are at least one of: a first score indicating no disease burden; a second score indicating a low disease burden in the patient's intestine; a third score indicating a moderate disease burden in the patient's intestine; and a fourth score indicating a high disease burden in the patient's intestine.

[0010] In some variations, the subset of the plurality of image patches is formed at least by clustering one or more similar image patches in the plurality of image patches based at least on one or more pixel-level features.

[0011] In some variations, the presence of a first pixel-level feature in the one or more pixel-level features in the plurality of image patches is associated with a first possible histology score. The absence of the first pixel-level feature is associated with a second possible histology score.

[0012] In some variations, generating the group-level histology score includes assigning a higher attention score to a first image block in the subset of the plurality of image blocks than to a second image block in the subset of the plurality of image blocks based at least on the presence of the first pixel-level feature or the absence of the first pixel-level feature. The group-level histology score is generated while determining the representation encoding of the subset of the plurality of image blocks.

[0013] In some embodiments, a higher attention score indicates that the first pixel-level feature of the first image patch contributes more to the representation encoding of the subset of the plurality of image patches than the second patch.

[0014] In some variations, the one or more pixel-level features indicate the presence of at least one of erosions of tissue, neutrophils, lymphoid structures, crypt abscesses, and intraepithelial debris of tissue in the biological sample.

[0015] In some variations, the one or more pixel-level features include at least one of shape, color, size, presence of a dye, and intensity associated with a pixel of the image of the biological sample.

[0016] In some variations, a subset of the plurality of image patches includes a common pixel-level feature of the one or more pixel-level features.

[0017] In some variations, the group-level histology score is generated based on at least one or more of a quantity of one or more pixel-level features within a subset of the plurality of image patches and a distribution of the one or more pixel-level features within the subset of the plurality of image patches.

[0018] In some variations, clustering is performed by applying a cluster analysis technique.

[0019] In some variations, the clustering analysis technique includes one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation-maximization (EM) clustering using Gaussian mixture models (GMM), and agglomerative hierarchical clustering.

[0020] In some variations, the method includes generating a first visual representation of a reduced-dimensional representation of the plurality of image patches.

[0021] In some variations, the first visual representation includes one or more visual indicators configured to indicate a contribution of the pixel-level features to the histology score at the possible group level.

[0022] In some variations, the first visual representation comprises one or more visual indicators configured to provide visual distinction between image patches in the plurality of image patches that indicate different possible histology scores.

[0023] In some variations, the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel level representation of each of the plurality of image blocks.

[0024] In some variations, the dimensionality reduction technique includes one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

[0025] In some embodiments, the group-level histology score and the aggregated histology score are each generated by applying at least one machine learning model trained to generate the group-level histology score and the aggregated histology score by determining at least a representation encoding of a subset of the plurality of image patches.

[0026] In some embodiments, the at least one machine learning model comprises a multiple instance learning (MIL) model.

[0027] In some exemplary embodiments, a system is provided that includes at least one processor and at least one memory. The at least one memory may include program code that provides operations when executed by the at least one processor. The operations may include: receiving an image of a biological sample from the intestine of a patient. The image depicts a plurality of cells of the biological sample. The operations may include segmenting the received image into a plurality of parts. Each of the plurality of parts corresponds to a cell in a plurality of cells. The operations may include identifying a first spatial coordinate associated with each of the plurality of cells in the image based at least on the segmented image. The operations may include identifying a first cell type associated with the first spatial coordinate. The operations may include generating a visual representation of the image including the biological sample based at least on the first spatial coordinate and the first cell type.

[0028] In some aspects, a computer-implemented method includes: receiving an image of a biological sample from the intestine of a patient. The image depicts a plurality of cells of the biological sample. The method may include segmenting the received image into a plurality of portions. Each of the plurality of portions corresponds to a cell in a plurality of cells. The method may include identifying a first spatial coordinate associated with each of the plurality of cells in the image based at least on the segmented image. The method may include identifying a first cell type associated with the first spatial coordinate. The method may include generating a visual representation of the image including the biological sample based at least on the first spatial coordinate and the first cell type.

[0029] In some aspects, a computer program product is provided, comprising a non-transitory computer-readable medium storing instructions. The instructions may cause operations that may be performed by at least one data processor. The operations may include: receiving an image of a biological sample from the intestine of a patient. The image depicts a plurality of cells of the biological sample. The operations may include segmenting the received image into a plurality of parts. Each of the plurality of parts corresponds to a cell in a plurality of cells. The operations may include identifying a first spatial coordinate associated with each of the plurality of cells in the image based at least on the segmented image. The operations may include identifying a first cell type associated with the first spatial coordinate. The operations may include generating a visual representation of the image comprising the biological sample based at least on the first spatial coordinate and the first cell type.

[0030] In some variations, one or more features disclosed herein (including the following) can optionally be included in any feasible combination of systems, methods, and / or non-transitory computer-readable media.

[0031] In some variations, the identifying is further based at least on a plurality of annotations identifying a plurality of cell types depicted in a plurality of images of the biological sample.

[0032] In some variations, the first cell type is at least one of a neutrophil, a plasma cell, a lymphocyte, an intraepithelial lymphocyte, an eosinophil, a mast cell, a macrophage, a goblet cell, an enterocyte, an endothelial cell, a fibroblast, a smooth muscle cell, and an endothelial cell.

[0033] In some variations, the image further depicts multiple tissue regions of the biological sample.

[0034] In some variations, the method includes identifying, based at least on the segmented image, second spatial coordinates associated with each tissue region of the plurality of tissue regions and a first tissue region type associated with the second spatial coordinates.

[0035] In some variations, the method includes generating a second visual representation of the image comprising the biological sample based at least on the second spatial coordinates and the first tissue region type.

[0036] In some variations, the tissue region type is at least one of epithelial, mucosa, submucosa, normal crypts, infiltrated crypts, lumen, vessel, lymphatic vessel, lamina propria, muscularis mucosae, basal plasmacytosis, ulcer, erosion, granulation tissue, infiltrated crypts, crypt abscess, normal collagen, abnormal collagen, stroma, a subtype of stroma, hyperplastic muscle, fissure, abscess, normal fat, abnormal fat, serosa, and serositis.

[0037] In some variations, identifying the second spatial coordinate and the first tissue region type is further based on at least a second plurality of annotations identifying a plurality of tissue region types depicted in a plurality of images of the biological sample.

[0038] In some variations, identifying includes generating a metric indicative of a confidence level associated with the identified first cell type for each cell in the plurality of cells.

[0039] In some variations, the method includes generating spatial tabular data including a first spatial coordinate associated with each cell of a plurality of cells within the image and a first cell type associated with the first spatial coordinate.

[0040] In some variations, the method includes generating a histology score for the biological sample based on at least the first spatial coordinate and a first cell type associated with the first spatial coordinate. The histology score is indicative of disease burden in the intestine of the patient.

[0041] In some variations, the histology score is one of a Nancy Histology Index (NHI) score, a Robarts Histopathology Index (RHI) score, a Geboes Scale score, a Global Histology Activity Score (GHAS), a muscle hyperplasia score, and a fibrosis score.

[0042] In some variations, the first cell type comprises neutrophils.

[0043] In some variations, the histology score is further generated based on the spatial distribution of the plurality of cells identified as having the first cell type.

[0044] In some variations, the histology score is further generated based on a number of cells identified as having a first cell type meeting a threshold number of cells.

[0045] In some variations, the histology score is further generated based on tissue region types of a plurality of tissue regions depicted in the image.

[0046] In some variations, the type of tissue region is at least one of an erosion and an ulcer.

[0047] In some variations, the first spatial coordinate is two-dimensional.

[0048] In some variations, the method includes generating an overlay indicating the first cell type at the first spatial coordinate based at least on the first spatial coordinate and the first cell type. The overlay includes at least one of a mask, a color, and a pattern.

[0049] In some embodiments, an image is segmented by applying a machine learning model that is trained to perform cell-based segmentation and tissue-based segmentation by assigning at least a cell segmentation label and a tissue region label to each pixel in the image, wherein the cell segmentation label indicates whether the pixel is associated with a cell type of a cell depicted in the image and the tissue region label indicates whether the pixel is associated with a tissue region type of a tissue region depicted in the image.

[0050] Specific implementations of the current subject matter may include, but are not limited to, methods consistent with the description provided herein and articles including tangibly embodied machine-readable media, which are operable to cause one or more machines (e.g., computers, etc.) to cause operations that implement one or more of the features described. Similarly, a computer system that may include one or more processors and one or more memories coupled to the one or more processors is also described. A memory that may include a non-transitory computer-readable or machine-readable storage medium may include, encode, store, etc., one or more programs that cause one or more processors to perform one or more of the operations described herein. A computer-implemented method consistent with one or more implementations of the current subject matter may be implemented by one or more data processors present in a single computing system or multiple computing systems. Such multiple computing systems may be connected and may exchange data and / or commands or other instructions, etc., via one or more connections, including, for example, via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.) via a direct connection between one or more of the multiple computing systems, etc.

[0051] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent with reference to the description and drawings, and to the claims. Although certain features of the subject matter disclosed herein are described for illustrative purposes related to IBD and UC, it should be readily understood that such features are not intended to be limiting. The claims that follow this disclosure are intended to define the scope of the protected subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some principles associated with the disclosed embodiments. This patent or application file contains at least one drawing in color. Copies of this patent or patent application publication with one or more color drawings will be provided by the Patent Office upon request and payment of the necessary fee.

[0053] In the accompanying drawings,

[0054] Figure 1depicts a system diagram illustrating an example of a mucosal biopsy analysis digital pathology system according to some exemplary embodiments;

[0055] Figure 2 depicts an image of a biological sample from the intestine of a patient according to some exemplary embodiments;

[0056] Figure 3 Depicted is an image of a patient's intestine compared to an image of another biological sample from the patient's intestine according to some exemplary embodiments. Figure 2 a portion of an image;

[0057] Figure 4A depicts screenshots showing examples of visual representations according to some exemplary embodiments;

[0058] Figure 4B depicts screenshots showing examples of visual representations according to some exemplary embodiments;

[0059] Figure 4C depicts screenshots showing examples of visual representations according to some exemplary embodiments;

[0060] Figure 4D depicts screenshots showing examples of visual representations according to some exemplary embodiments;

[0061] Figure 5 depicts a bar graph according to some exemplary embodiments;

[0062] Figure 6 depicts a confusion matrix according to some exemplary embodiments;

[0063] Figure 7 depicts a confusion matrix according to some exemplary embodiments;

[0064] Figure 8 depicts a performance table according to some exemplary embodiments;

[0065] Fig. 9 depicts an exemplary end-to-end model according to some exemplary embodiments;

[0066] Fig. 10A depicts screenshots showing exemplary visual representations according to some exemplary embodiments;

[0067] Fig. 10B depicts screenshots showing exemplary visual representations according to some exemplary embodiments;

[0068] Fig. 10C depicts screenshots showing exemplary visual representations according to some exemplary embodiments;

[0069] Fig.11 depicts a comparison matrix according to some exemplary embodiments;

[0070] Fig.12 depicts a flow chart showing an example of a process for image segmentation according to some exemplary embodiments;

[0071] Fig.13 depicts a flow chart showing an example of a process for histology score generation according to some exemplary embodiments; and

[0072] Fig.14 Depicted is a block diagram illustrating an example of a computing system in accordance with some example embodiments.

[0073] When applicable, like reference numerals refer to like structures, features, or elements. DETAILED DESCRIPTION

[0074] IBD (including UC) has a high rate of long-term poor prognosis in terms of hospitalization and surgery, but there are few effective treatment options. As mentioned above, biological insights into IBD and consistent methods for assessing clinical disease burden remain crucial for identifying effective new therapeutic treatments, improving patient monitoring, etc. Historically, patient-reported outcomes, such as stool frequency and rectal bleeding, and endoscopic examinations (e.g., sigmoidoscopy) have been used to assess the health of the intestinal mucosa. For example, endoscopy has been used to provide a visual assessment of the patient's intestinal mucosa using a camera that passes through the intestinal lumen. Although the endoscopic appearance of the intestine is often used to determine whether the intestine shows a meaningful improvement in intestinal health (formerly known as the concept of mucosal healing), this visual assessment is highly subjective and variable. In addition, there is often a significant inconsistency between the endoscopic appearance of intestinal tissue and the actual inflammatory state of the tissue, as can be seen at the microscopic level. For example, significant active microscopic inflammation may persist in tissue that is normal on endoscopy; this disconnect highlights the importance of histological assessment of tissue biopsies taken from patients with IBD.

[0075] In order to solve the defect of using endoscopy to assess intestinal health, histological scoring has been developed, verified and implemented to classify the microscopic inflammation in the intestine more objectively. Although there are many scoring systems, they all assess the presence of neutrophil inflammation in the tissue, which defines the active inflammation in the intestine. The degree of epithelial damage caused by active inflammation (neutrophils) can also be a sign of tissue disease load. For example, a classification histological score can be assigned to a tissue sample. When there is epithelial erosion or ulcer in a tissue sample, the classification histological score can be classified as severe active disease. When neutrophil infiltration epithelium, the score can be classified as moderate active disease or mild active disease, specifically depending on neutrophil density and positioning in the tissue compartment. In the absence of active disease, there may be increased chronic (inactive) inflammation, and there will be an associated classification score. The lowest score (usually scored as zero) will be in the absence of a significant increase in chronic or active inflammation.

[0076] However, conventional methods for assigning such classification scores can be highly subjective, provide limited underlying data, and lack the granularity required for consistent and accurate assessment of intestinal health. Although training of pathologist central readers is expensive and slow, conventional methods for assigning such classification scores can result in high levels of variability, making scores not easily reproducible over time between readers (inter-reader variability) and by the same reader (intra-reader variability) for the same tissue sample, thereby reducing the ability of studies to identify treatment efficacy. For example, certain cells (such as neutrophils) may be confused with eosinophils or lymphocytes, and vice versa. Similarly, the presence of ulcers may also be caused by small, isolated, easily overlooked tissue fragments. Variability may further increase when classifying lesions as mild or moderate disease, which is a highly qualitative and subjective practice. Therefore, conventional techniques for assessing intestinal health and assigning histological scores are impractical due to high costs, high complexity of implementation, huge variability in scoring and interpretation, and lack of unbiased data.

[0077] In accordance with the embodiments of the current theme, the digital pathology platform can provide robust and reproducible data and classification histological scores. In particular, the digital pathology platform described herein can identify relevant cells and features to quantify the corresponding spatial positioning of the features within the cell content and tissue samples of each tissue sample. The digital pathology platform can also generate unbiased data sets, such as hypothesis testing in clinical trials, determination of clinical responses, and correlation with endoscopy, microbiome determination, gene sequencing, biomarkers, etc. For example, the digital pathology platform can segment an image of a biological sample from the patient's intestine into multiple cells, identify spatial coordinates associated with multiple cells and cell types and / or tissue region types associated with the spatial coordinates, and generate a visual representation based on the identified spatial coordinates and cell types and / or tissue region types. These spatially oriented and quantified features and cell-specific segmentations can be extrapolated to classification scores. Additionally and / or alternatively, the digital pathology platform can use an end-to-end model to predict the intestinal disease indication group enabled by machine learning and predict the reproducible histological score of the entire intestinal tissue sample. Therefore, the digital pathology platform can generate comprehensive, quantitative and unbiased data based on tissue samples and predict reproducible histological scores based on tissue samples.

[0078] Figure 1 A system diagram illustrating an example of a digital pathology system 100 is depicted in accordance with some exemplary embodiments. Figure 1 , the digital pathology system 100 may include a mucosal biopsy analysis digital pathology platform 110, an imaging system 120, and a client device 130. Figure 1 As shown, the digital pathology platform 110, the imaging system 120, and the client device 130 can be communicatively connected via a network 140. The network 140 can be a wired network and / or a wireless network, including, for example, a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, etc. The imaging system 120 may include one or more imaging devices (including, for example, a microscope, a digital camera, a whole slide scanner, an automated microscope, etc.). The client device 130 can be a processor-based device, including, for example, a workstation, a desktop computer, a laptop computer, a smart phone, a tablet computer, a wearable device, etc.

[0079] The digital pathology platform 110 may include an IBD analysis engine 115 and a mucosal biopsy segmentation engine 116. The analysis engine 115 and / or the segmentation engine 116 may perform one or more of the various processes and / or workflows described herein. The analysis engine 115 and the segmentation engine 116 may communicate with each other. For example, the segmentation engine 116 may execute at least a portion of a workflow, and the analysis engine 115 may execute another portion of the workflow based on the first portion of the workflow executed by the segmentation engine 116, and vice versa. Therefore, one or more aspects of the analysis engine 115 described herein may be applied to the segmentation engine 116, and one or more aspects of the segmentation engine 116 may be applied to the analysis engine 115. The digital pathology platform 110 may be hosted on a cloud-based infrastructure so that the functionality of the digital pathology platform 110 is remotely accessible, such as as part of a web-based application, a native mobile application, software as a service (SaaS), etc.

[0080] Reference again Figure 1 , the digital pathology platform 110 may receive one or more images of a biological sample of the intestine of the patient, such as from an imaging system 120. The one or more images of the biological sample may be a whole slide image (WSI). In some embodiments, the one or more images of the biological sample are full scan images of hematoxylin and eosin stained (H&E) formalin fixed, paraffin embedded (FFPE) tissue. The one or more images may be images of a microscope slide and / or a microscope slide.

[0081] The one or more images may depict multiple cells of a biological sample, which may include a mucosal biopsy. For example, a biological sample from the intestine of a patient may include at least one neutrophil, plasma cell, lymphocyte, intraepithelial lymphocyte, eosinophil, mast cell, macrophage, goblet cell, enterocyte, endothelial cell, fibroblast, smooth muscle cell, endothelial cell, etc. The one or more images may additionally and / or alternatively depict multiple tissue regions, including epithelium, mucosa, submucosa, normal crypts, infiltrated crypts, lumen, blood vessels, lymphatic vessels, lamina propria, muscularis mucosae, basal plasmacytosis, ulcers, erosions, granulation tissue, infiltrated crypts, crypt abscesses, normal collagen, abnormal collagen, stroma, subtypes of stroma, hyperplastic muscle, fissures, abscesses, normal fat, abnormal fat, serosa, serositis, etc. Exemplary images are shown in Figure 2 and Figure 3 Described in.

[0082] As mentioned previously, images based on biological samples of the intestine from patients (such as Figure 2 Image 200 and / or Figure 3Unbiased data generated by the image 300 of the tissue sample. Unbiased data includes data generated due to subjective analysis. For example, as previously described, the generated data may be biased based on the highly subjective and variability of visual assessment, classification score assignment and interpretation of images of tissue samples.

[0083] According to some exemplary embodiments, the segmentation engine 116 generates a plurality of unbiased data based on such images. For example, the segmentation engine 116 can employ a machine learning model to detect various features of a mucosal biopsy sample, including neutrophils, plasma cells, lymphocytes, intraepithelial lymphocytes, eosinophils, mast cells, macrophages, goblet cells, enterocytes, endothelial cells, fibroblasts, smooth muscle cells, endothelial cells, epithelium, mucosa, submucosa, normal crypts, infiltrated crypts, lumen, blood vessels, lymphatic vessels, lamina propria, muscularis mucosa, basal plasmacytosis, ulcers, erosions, granulation tissue, infiltrated crypts, crypt abscesses, normal collagen, abnormal collagen, stroma, subtypes of stroma, hyperplastic muscle, fissures, abscesses, normal fat, abnormal fat, serosa, serositis, etc. For example, the segmentation engine 116 can generate spatial tabular data, heat map visualization / overlays, predicted histological scores, etc. The spatial table data includes at least one table having a plurality of rows corresponding to specific cell types and / or tissue region types and a plurality of columns corresponding to at least one spatial coordinate associated with cells having the specific cell types and / or tissue region types. At least one spatial coordinate can be a two-dimensional spatial coordinate so that it identifies the two-dimensional position of at least a portion of cells having the corresponding cell types and / or tissue region types. In other words, the spatial coordinate can include at least an x-coordinate and a y-coordinate associated with the position of at least a portion of cells having the corresponding cell types and / or tissue region types.

[0084] The generated spatial tabular data, visualization, predicted histological scores, etc., help to identify, locate and quantify cell types and tissue area types in the image with improved granularity, and thus identify, locate and quantify disease load. For example, spatial tabular data allows each cell and associated features depicted or detected in the positioning image. The positioning of cells and associated features can provide key biomarkers indicating the disease load in the intestine of the patient. For example, since the positioning of the cells depicted in the image provides the two-dimensional position of each cell and / or tissue area, the distribution of cells in the image can be determined and used to effectively and accurately assess the disease load in the intestinal biopsy of the patient. The spatial position, quantity and distribution of cells and / or tissue areas in the image can also be used to determine the therapeutic efficacy of the treatment regimen for treating IBD (such as UC). For example, spatial position, quantity and distribution can be associated with a specific level of disease load (e.g., none, mild, moderate, severe, etc.). Spatial position, quantity and distribution can be determined and compared at different time points to determine whether the spatial position, quantity and distribution indicate the improvement of disease load in the intestine of the patient, and determine an effective treatment regimen for treating the disease.

[0085] Based at least on the two-dimensional coordinates of each cell and / or tissue region, segmentation engine 116 can generate multiple visual representations of the image including the biological sample. Visual representation can indicate the position of at least one cell type and / or tissue region type in the image. For example, segmentation engine 116 can generate a superposition associated with the cell type and / or tissue region type of the cell and / or tissue region depicted in the image of the biological sample, such as at least one of a mask, a color, a pattern, a shadow, etc. In some embodiments, the generated visual representation can be manipulated via the selection on user interface 135 to indicate the spatial position of at least one cell and / or tissue region. For example, the generated visual representation can be manipulated via the selection on user interface 135 to display the superposition associated with the selected cell type and / or tissue region type. In other words, the selection of specific cell type and / or tissue region type can display the position of the cell and / or tissue region with corresponding cell type and / or tissue region type respectively. Figures 4A to 4D Examples of such visual representations are provided.

[0086] Additionally and / or alternatively, the segmentation engine 116 can predict the histological score of the biological sample depicted in the image based at least on the two-dimensional coordinates of each cell and / or tissue region, the spatial distribution of the cell and / or tissue region, the number of cells and / or tissue regions meeting the threshold number, the detection of specific cell types and / or tissue regions, and / or the positioning of specific cell types and / or tissue region types within the image. In some embodiments, the segmentation engine 116 provides the spatial table data and / or positioning data to the analysis engine 115 to predict the histological score. The predicted histological score can indicate the disease load in the intestine of the patient. The predicted histological score can be at least one of the Nancy histological index (NHI) score, the Robarts histopathological index (RHI) score, the Geboes scale score, the overall histological activity score (GHAS), the muscle hyperplasia score, and the fibrosis score. Although the prediction score consistent with the implementation of the current theme is referred to herein as indicating the disease load in the intestine of the patient, the prediction score can additionally and / or alternatively indicate the disease load in the biopsy (e.g., biological sample) or the image of the biopsy (e.g., the image of the biological sample). The prediction score can still indicate the overall disease burden in the patient's intestine. In some embodiments, the prediction score indicates the overall disease burden in the patient's intestine, such as based on a prediction score associated with one or more biopsies (e.g., biological samples) or images of the one or more biopsies (e.g., images of biological samples).

[0087] In some embodiments, the classification histology score can be scored from zero to four based at least on the spatial coordinates, distribution, etc. of the cells and / or tissue regions identified by the segmentation engine 116. For example, a 4-level score can be predicted based on a determination of severe active disease (such as when there is erosion or ulceration in the tissue sample). A 3-level score can be predicted based on a determination of moderately active disease, or a 2-level score can be predicted based on a determination of mildly active disease (such as when it is determined that acute inflammatory cells have infiltrated the epithelium of the tissue sample). A 1-level score can be predicted based on a determination of a moderate or significant increase in chronic inflammatory cells, which are identified as infiltrating the tissue sample lamina propria without acute inflammatory infiltration. When no significant disease activity is identified, a 0-level score can be predicted based on a determination that there is no or a slight increase in chronic inflammation.

[0088] To generate spatial tabular data, visualization, and / or histological scores, the segmentation engine 116 can segment the image of the biological sample into a plurality of portions corresponding to cells and / or tissue regions. The segmentation engine 116 can segment the image to locate individual cells and / or tissue regions present in the image depicting the biological sample. For example, the segmentation engine 116 can segment the image to locate individual cells, including at least one of neutrophils, plasma cells, lymphocytes, intraepithelial lymphocytes, eosinophils, mast cells, macrophages, goblet cells, enterocytes, endothelial cells, fibroblasts, smooth muscle cells, endothelial cells, etc., and / or individual tissue regions, such as at least one of epithelium, mucosa, submucosa, normal crypts, infiltrated crypts, lumen, blood vessels, lymphatic vessels, lamina propria, muscularis mucosae, basal plasmacytosis, ulcers, erosions, granulation tissue, infiltrated crypts, crypt abscesses, normal collagen, abnormal collagen, stroma, subtypes of stroma, hyperplastic muscle, fissures, abscesses, normal fat, abnormal fat, serosa, serositis, etc.

[0089] For example, the segmentation engine 116 can perform cell-by-cell and / or tissue-by-region segmentation to assign to each pixel within an image of a biological sample a cell segmentation label identifying the cell to which the pixel belongs and / or a tissue-by-region label identifying the tissue-by-region to which the pixel belongs. In addition, the task of cell-by-cell and / or tissue-by-region segmentation can include operating on high-dimensional image data, particularly when the image is high-resolution and / or acquired at a high level of magnification.

[0090] In some exemplary embodiments, the segmentation engine 116 may apply various cell and / or tissue region segmentation techniques, including, for example, watershed cell segmentation, deep learning-based cell segmentation, etc. For example, the segmentation engine 116 may train and apply a machine learning model (e.g., a convolutional neural network, etc.) to perform cell segmentation and / or tissue region segmentation. The machine learning model may perform cell segmentation by determining, at least for each pixel in the image, whether the pixel is part of the background of the image or part of the cell and / or tissue region depicted in the image (see the illustration of the segmentation background). Figure 4A ). To this end, the machine learning model can further predict, for each pixel in the image, the probability or confidence level that the pixel is part of the cell and / or tissue region depicted in the image.

[0091] In some exemplary embodiments, the segmentation engine 116 can apply a variety of cell segmentation techniques to perform cell segmentation and locate individual cells and / or tissue regions present in the image. In some cases, the segmentation label of a single pixel can be determined based on a first result of a first cell segmentation technique and a second result of a second cell segmentation technique. For example, the segmentation engine 116 can apply a first machine learning model to a pixel within an image to determine a first cell and / or tissue region segmentation label, and apply a second machine learning model to determine a second cell and / or tissue region segmentation label. In addition, the segmentation engine 116 can determine a third cell and / or tissue region segmentation label for the pixel based at least on the first cell and / or tissue region segmentation label and the second cell and / or tissue region segmentation label.

[0092] The segmentation engine 116 can identify at least one spatial coordinate based on the segmented image. For example, by at least locating the cell and / or tissue region, the segmentation engine 116 can identify at least one spatial coordinate associated with each cell and / or tissue region within the image. As described herein, the spatial coordinate can include an x-coordinate and / or a y-coordinate of the corresponding image. The identified spatial coordinate can be stored for use in generating spatial tabular data, visualization, histological scoring, etc.

[0093] When locating individual cells and / or tissue regions present in the image, the segmentation engine 116 can identify at least one cell type and / or tissue region type associated with the identified spatial coordinates. For example, the segmentation engine 116 can identify at least one cell type (including neutrophils, plasma cells, lymphocytes, intraepithelial lymphocytes, eosinophils, mast cells, macrophages, goblet cells, enterocytes, endothelial cells, fibroblasts, smooth muscle cells, endothelial cells, etc.) and / or tissue regions (such as epithelium, mucosa, submucosa, normal crypts, infiltrated crypts, lumens, blood vessels, lymphatic vessels, lamina propria, muscularis mucosae, basal plasma cell hyperplasia, ulcers, erosions, granulation tissue, infiltrated crypts, crypt abscesses, normal collagen, abnormal collagen, stroma, subtypes of stroma, hyperplastic muscle, fissures, abscesses, normal fat, abnormal fat, serosa, serositis, etc.) as being associated with the identified pixels. The segmentation engine 116 may store the assigned cell type and / or tissue region type in association with the corresponding pixel. For example, the segmentation engine 116 may store the assigned cell type and / or tissue region type with the corresponding pixel for use in generating spatial tabular data, visualization, histological scoring, etc.

[0094] In some embodiments, the segmentation engine 116 identifies the cell type and / or tissue region type corresponding to a particular pixel based on at least one annotation identifying a plurality of cell types and / or tissue region types in a plurality of images of the biological sample. For example, at least one annotation may be applied by a pathologist or other trained professional. The pathologist may provide annotations to various images to identify each of the cell type and / or tissue region type within the image. The machine learning model may be trained based at least in part on the annotations. In this way, the machine learning model may include a supervised machine learning model.

[0095] In some embodiments, the segmentation engine 116 identifies the cell type and / or tissue region type corresponding to a particular pixel based on one or more features identified in the pixel. For example, a pixel may depict one or more features associated with a particular cell type and / or tissue region type. Based on the one or more features depicted in the pixel, the segmentation engine 116 assigns a cell segmentation label and / or a tissue region segmentation label.

[0096] As an example, Figure 2 It is an image 200 of a biological sample from the intestine of a patient. In particular, image 200 is a FFPE tissue that has been sliced ​​on a glass slide and stained with hematoxylin and eosin (H&E). Image 200 depicts a normal or healthy colon mucosa without disease. Image 200 depicts normal colon mucosa as having epithelium (including crypt and surface epithelium) and lamina propria (which includes sparse inflammatory cells, such as plasma cells and lymphocytes). Here, epithelial crypts have uniform shape and size, have roughly equal spatial distribution, and indicate that there is no disease or disease load.

[0097] Figure 3 A portion of image 200 is shown for comparison with image 300 of another biological sample of the intestine from a patient with UC. In image 300, the lamina propria is filled with dense chronic inflammatory cells, such as lymphocytes and plasma cells, with associated features of basal plasmacytosis, which is an inflammatory zone between the crypt base and the muscularis mucosa. In addition, as shown in FIG. Figure 3 As shown, the epithelium is irregular in size, shape, and distribution (e.g., architecturally distorted). In addition to these features of chronic colitis, there is active inflammation, as defined by the presence of neutrophil infiltration. The presence of cryptitis, ulcers, etc. indicates epithelial damage. The segmentation engine 116 can determine that the image 300 depicts chronic, active colitis.

[0098] Figure 4A An example of a visual representation 400 including an image 401 of a biological sample is shown in accordance with some exemplary embodiments. Figure 4AAs shown, visual representation 400 includes a first selectable element 402 and a second selectable element 404. Selection of first selectable element 402 may result in an overlay being displayed on image 401 indicating an artifact in visual representation 400. Selection of second selectable element 404 may result in an overlay being displayed on image 401 indicating a background in visual representation 400. Thus, visual representation 400 may be used to more clearly indicate features of image 401 for evaluation.

[0099] Figure 4B An example of a visual representation 410 including an image 411 (which may be the same as or different from image 401) of a biological sample is shown in accordance with some exemplary embodiments. Figure 4B As shown, visual representation 410 includes a first selectable element 412, a second selectable element 414, a third selectable element 416, a fourth selectable element 418, a fifth selectable element 420, a sixth selectable element 422, a seventh selectable element 424, an eighth selectable element 426, a ninth selectable element 428, and a tenth selectable element 429. In this example, first selectable element 412, second selectable element 414, third selectable element 416, fourth selectable element 418, fifth selectable element 420, sixth selectable element 422, seventh selectable element 424, eighth selectable element 426, ninth selectable element 428, and tenth selectable element 429 are associated with different cell types and / or tissue region types. For example, the first selectable element 412 is associated with basal plasmacytosis, the second selectable element 414 is associated with granulation tissue, the third selectable element 416 is associated with blood vessels, the fourth selectable element 418 is associated with erosion or ulcer, the fifth selectable element 420 is associated with the lamina propria, the sixth selectable element 422 is associated with the muscularis mucosa, the seventh selectable element 424 is associated with crypt abscesses, the eighth selectable element 426 is associated with the crypt cavity, the ninth selectable element 428 is associated with infiltrated epithelium, and the tenth selectable element 429 is associated with normal crypts. The first selectable element 412, the second selectable element 414, the third selectable element 416, the fourth selectable element 418, the fifth selectable element 420, the sixth selectable element 422, the seventh selectable element 424, the eighth selectable element 426, the ninth selectable element 428, and the tenth selectable element 429 can each be selected to display an overlay indicating the associated cell type and / or tissue region type.

[0100] Figure 4C An example of a visual representation 430 including an image 431 of a biological sample is shown in accordance with some exemplary embodiments. Figure 4CAs shown, visual representation 430 includes a first selectable element 432 and a second selectable element 434. In this example, first selectable element 432 and second selectable element 434 are associated with different cell types and / or tissue region types. For example, first selectable element 432 is associated with a vascular lumen and second selectable element 434 is associated with a goblet cell cytoplasm. First selectable element 432 and second selectable element 434 can each be selected to display an overlay indicating the associated cell type and / or tissue region type.

[0101] Figure 4D An example of a visual representation 440 including an image 441 of a biological sample is shown in accordance with some exemplary embodiments. Figure 4B As shown, visual representation 440 includes a first selectable element 442, a second selectable element 444, a third selectable element 446, a fourth selectable element 448, a fifth selectable element 450, a sixth selectable element 452, a seventh selectable element 454, and an eighth selectable element 456. In this example, first selectable element 442, second selectable element 444, third selectable element 446, fourth selectable element 448, fifth selectable element 450, sixth selectable element 452, seventh selectable element 454, and eighth selectable element 456 are associated with different cell types and / or tissue region types. For example, the first selectable element 442 is associated with goblet cell nuclei, the second selectable element 444 is associated with epithelial non-goblet cell enterocytes, the third selectable element 446 is associated with intraepithelial lymphocytes, the fourth selectable element 448 is associated with non-intraepithelial lymphocytes, the fifth selectable element 450 is associated with plasma cells, the sixth selectable element 452 is associated with eosinophils, the seventh selectable element 454 is associated with neutrophils, and the eighth selectable element 426 is associated with other cells. The first selectable element 442, the second selectable element 444, the third selectable element 446, the fourth selectable element 448, the fifth selectable element 450, the sixth selectable element 452, the seventh selectable element 454, and the eighth selectable element 456 can each be selected to display an overlay indicating the associated cell type and / or tissue region type.

[0102] Figure 5 An exemplary bar graph 500 according to some exemplary embodiments is shown. The bar graph 500 compares the cell segmentation labels and / or tissue region segmentation labels generated by the segmentation engine 116 (see 502) for a set of 240 images with the manual annotations provided by five pathologists for these images (see 504). As shown in the bar graph 500, the cell segmentation labels and / or tissue region segmentation labels generated by the segmentation engine 116 are positively correlated with the manual annotations for each of the cell types and / or tissue region types. Figure 6 A confusion matrix is ​​depicted, which further illustrates a positive correlation between the cell segmentation labels and / or tissue region segmentation labels generated by the segmentation engine 116 and the manually applied annotations. Figure 8 The second column shows that the segmentation engine 116 consistently and accurately identifies cell segmentation labels and / or tissue region segmentation labels. Figure 8 As shown, the model employed by the segmentation engine 116 to identify cell segmentation labels and / or tissue region segmentation labels associated with pixels in an image has a weighted κ of 0.80 and a Spearman correlation of 0.79.

[0103] Fig. 9 An end-to-end model 900 according to some exemplary embodiments is shown. The end-to-end model 900 can be implemented at least in part by an analysis engine 115. The analysis engine 115 can implement the end-to-end model 900 to generate an aggregated histology score (e.g., an overall score) of a biological sample depicted in an image (such as a whole slide scan image of an H&E-stained FFPE tissue, etc.). As described herein, a histology score, such as an aggregated histology score, is a Nancy histology index (NHI) score, a Robarts histopathology index (RHI) score, a Geboes scale score, a global histology activity score (GHAS), etc. In some instances, the histology score includes a first score indicating no disease load (e.g., no disease) or a low disease load in the intestine of a patient, a second score indicating a mild disease load in the intestine of a patient, a third score indicating a moderate disease load in the intestine of a patient, a fourth score indicating a high disease load, and the like. As previously described, although the prediction score consistent with embodiments of the current subject matter is referred to herein as indicating the disease load in the patient's intestine, the prediction score can additionally and / or alternatively indicate the disease load in the biopsy (e.g., biological sample) or an image of the biopsy (e.g., an image of a biological sample). The prediction score can still indicate the overall disease load in the patient's intestine. In some embodiments, the prediction score indicates the overall disease load in the patient's intestine, such as based on a prediction score associated with one or more biopsies (e.g., biological samples) or images of the one or more biopsied tissues (e.g., images of biological samples).

[0104] As described herein, scores are typically assigned to images of biological samples. However, such scores are typically unreliable, at least in part due to the subjectivity involved in assigning scores, variability within and between readers when assigning scores to the same biological sample, and variability in interpreting and defining various aspects of the biological sample. The analysis engine 115 can employ an end-to-end model 900 to generate reliable and reproducible histological scores for the biological samples depicted in the image. Therefore, the analysis engine 115 can reduce the variability previously associated with histological scores, which improves the ability to rely on histological scores and the ability to more accurately assess and / or treat the patient's physical condition (e.g., the disease load in the patient's intestine).

[0105] The end-to-end model 900 may include one or more machine learning models 910, such as a multi-instance machine learning model and other models. The analysis engine 115 may apply the one or more machine learning models 910 to determine a histological score indicating the level of disease burden in the portion of the intestine depicted in the image based at least on an image of a biological sample depicting at least a portion of the patient's intestine. For example, Fig. 9 As shown, the analysis engine 115 can determine one or more intestinal disease indication groups within an image of at least a portion of the patient's intestine (such as image 200 and / or image 300), including, for example, a first intestinal disease indication group 902A, a second intestinal disease indication group 902B, a third intestinal disease indication group 902C, etc.

[0106] One or more intestinal disease indication groups include a plurality of image patches, each image patch depicting a portion of a biological sample. Fig. 9 An image block 905 of the plurality of image blocks is shown. Image block 905 may depict at least a portion of a cell present in a biological sample.

[0107] refer to Fig. 9 , the first intestinal disease indication group 902A, the second intestinal disease indication group 902B, and the third intestinal disease indication group 902C may each include a subset of a plurality of image blocks representing one or more portions of a biological sample depicted in an image. For example, the first intestinal disease indication group 902A includes a first plurality of image blocks 904A, the second intestinal disease indication group 902B includes a second plurality of image blocks 904B, and the third intestinal disease indication group 902C includes a third plurality of image blocks 904C. Each of the first plurality of image blocks 904A, the second plurality of image blocks 904B, and the third plurality of image blocks 904C includes one or more portions of a biological sample depicted in an image. Therefore, the first plurality of image blocks 904A, the second plurality of image blocks 904B, and the third plurality of image blocks 904C may each be a subset of a plurality of image blocks.

[0108] In some embodiments, the analysis engine 115 can exclude image patches that do not depict cells and / or cell parts present in the biological sample above a threshold number from the histology score generation. Examples of image patches that are excluded from the histology score generation can include image patches with an image background above a threshold ratio, image patches with an average color channel variance below a threshold (e.g., gray patches), etc.

[0109] The analysis engine 115 may form subsets of the plurality of image patches (e.g., a first plurality of image patches 904A, a second plurality of image patches 904B, a third plurality of image patches 904C, etc.) by at least clustering one or more similar image patches in the plurality of image patches based at least on pixel-level representations of the plurality of image patches including one or more pixel-level features. The analysis engine 115 may form a first intestinal disease indication group 902A, a second intestinal disease indication group 902B, and a third intestinal disease indication group 902C by at least clustering one or more image patches having similar pixel-level features. In some embodiments, the analysis engine 115 forms intestinal disease burden groups by at least clustering image patches based on pixel-level features, such that each intestinal disease indication group includes a distribution of image patches representing an overall image of the biological sample.

[0110] In some embodiments, the analysis engine 115 (e.g., the machine learning model 910) includes a feature extraction mechanism to extract one or more pixel-level features from multiple image patches (such as from a pixel-level representation of the multiple image patches). Pixel-level features can be associated with specific pixels of an image. Pixel-level features can include features associated with specific cells and / or tissue regions. For example, pixel-level features can include shapes, colors, sizes, presence of dyes, intensity, etc. associated with specific pixels of an image of a biological sample. Pixel-level features can indicate the presence of tissue region types and / or cell types in a biological sample, such as, for example, erosions of tissue, neutrophils, lymphoid structures, crypt abscesses, intraepithelial debris of tissue, etc.

[0111] The analysis engine 115 may form subsets of the plurality of image patches for each of the bowel disease burden groups by applying a cluster analysis technique such as k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), agglomerative hierarchical clustering, etc. to the pixel-level representation of the image of the biological sample. In doing so, the analysis engine 115 may identify the number of clusters that maximizes the intra-cluster correlation between members of each cluster.

[0112] Additionally and / or alternatively, the analysis engine 115 identifies the group of intestinal disease indications by applying a dimensionality reduction technique (such as principal component analysis (PCA), uniform manifold approximation and projection (UMAP), T-distributed stochastic neighbor embedding (t-SNE), etc.) to the pixel-level representation of each image block in the image. The resulting dimensionality reduction representation of the image blocks in the image may correspond to a projection of the pixel-level representation of each image block onto a lower dimensional subspace.

[0113] The analysis engine 115 may generate one or more visual representations of the reduced dimensionality representation of the plurality of image patches. The one or more visual representations may include one or more visual indicators (e.g., overlays, colors, masks, etc.) that display different pixel-level features and / or indicate the contribution of the pixel-level features to the histological scores of the possible group levels (see Figures 10A to 10C ). One or more visual indicators provide visual distinction between image blocks, indicating different possible histological scores.

[0114] Reference again Fig. 9 , the analysis engine 115 generates a group-level histological score for each of the multiple intestinal disease indication groups via the machine learning model 910, based at least on a subset of the multiple image blocks included in each corresponding group. In other words, the analysis engine 115 can generate a group-level histological score by at least applying one or more machine learning models to predict the histological score of each intestinal disease indication group of the image blocks. One or more machine learning models can be trained to generate a group-level histological score by at least determining a representation encoding of a subset of the multiple image blocks. In some embodiments, at least one machine learning model is trained on the cell segmentation labels and tissue region segmentation labels generated by the segmentation engine 116 and / or based on annotations provided by a pathologist. Reference Fig. 9 , the machine learning model 910 can generate a first set of hierarchical histological scores 908A corresponding to the first intestinal disease indication group 902A, a second set of hierarchical histological scores 908B corresponding to the second intestinal disease indication group 902B, and a third set of hierarchical histological scores 908C corresponding to the third intestinal disease indication group 902C.

[0115] In some embodiments, a group-level histological score can be generated while determining a representation encoding for a subset of multiple image blocks. For example, the machine learning model 910 may include an attention mechanism that is configured to assign an attention score to each image block, the attention score representing the contribution (e.g., relevance) of the pixel-level features in each image block to the group-level histological score of each corresponding intestinal disease indication group. Therefore, the attention mechanism of the machine learning model 910 can assign a higher attention score to a first image block in a subset of multiple image blocks compared to a second image block in the subset of multiple image blocks based at least on the presence of a first pixel-level feature or the absence of a first pixel-level feature. A higher attention score indicates that the first pixel-level feature of the first image block contributes more to the representation encoding of the subset of multiple image blocks than the second block.

[0116] In some exemplary embodiments, the analysis engine 115 may generate one or more visual representations, for example for display in a user interface 135 at a client device 130, that display the contribution of particular pixel-level features to a particular group-level histology score and the associated attention score. The one or more visual representations may include an overlay showing the image patch-level contribution to each group-level histology score. Typically, the group-level histology scores and / or aggregated histology score predictions may be driven by a greater number of image patches that have a high contribution to a given histology score.

[0117] For example, Fig. 10A Shown are a first visual representation 1002 corresponding to a first histological score indicating a low disease burden, a second visual representation 1004 corresponding to a second histological score indicating a mild or marked increase in disease burden, a third visual representation 1006 corresponding to a third histological score indicating a mild disease burden, a fourth visual representation 1008 corresponding to a fourth histological score indicating a moderate disease burden, and a fifth visual representation 1010 corresponding to a fifth histological score indicating a high disease burden. As shown, the second visual representation 1004 displays a greater number of image blocks with high contributions than the first visual representation 1002, the third visual representation 1006 displays a greater number of image blocks with high contributions than the second visual representation 1004 and / or the first visual representation 1002, the fourth visual representation 1008 displays a greater number of image blocks with high contributions than the first visual representation 1002, the second visual representation 1004, and the third visual representation, and the fifth visual representation 1010 displays a greater number of image blocks with high contributions than the first visual representation 1002, the second visual representation 1004, the third visual representation 1006, and the fourth visual representation 1008.

[0118] As another example, Fig. 10BShown are a first visual representation 1012 corresponding to a first histological score indicating a low disease burden, a second visual representation 1014 corresponding to a second histological score indicating a mild or marked increase in disease burden, a third visual representation 1016 corresponding to a third histological score indicating a mild disease burden, a fourth visual representation 1018 corresponding to a fourth histological score indicating a moderate disease burden, and a fifth visual representation 1020 corresponding to a fifth histological score indicating a high disease burden. As shown, the second visual representation 1014 displays a greater number of image blocks with high contributions than the first visual representation 1012, the third visual representation 1016 displays a greater number of image blocks with high contributions than the second visual representation 1014 and / or the first visual representation 1012, the fourth visual representation 1018 displays a greater number of image blocks with high contributions than the first visual representation 1012, the second visual representation 1014, and the third visual representation, and the fifth visual representation 1020 displays a greater number of image blocks with high contributions than the first visual representation 1012, the second visual representation 1014, the third visual representation 1016, and the fourth visual representation 1018.

[0119] As another example, Fig. 10C Shown are a first visual representation 1022 corresponding to a first histological score indicating a low disease burden, a second visual representation 1024 corresponding to a second histological score indicating a mild or marked increase in disease burden, a third visual representation 1026 corresponding to a third histological score indicating a mild disease burden, a fourth visual representation 1028 corresponding to a fourth histological score indicating a moderate disease burden, and a fifth visual representation 1030 corresponding to a fifth histological score indicating a high disease burden. As shown, the second visual representation 1024 displays a greater number of image blocks with high contributions than the first visual representation 1022, the third visual representation 1026 displays a greater number of image blocks with high contributions than the second visual representation 1024 and / or the first visual representation 1022, the fourth visual representation 1028 displays a greater number of image blocks with high contributions than the first visual representation 1022, the second visual representation 1024, and the third visual representation, and the fifth visual representation 1030 displays a greater number of image blocks with high contributions than the first visual representation 1022, the second visual representation 1024, the third visual representation 1026, and the fourth visual representation 1028.

[0120] Additionally and / or alternatively, the machine learning model 910 generates a group-level histology score based at least on the number of one or more pixel-level features within a subset of the plurality of image patches, the distribution of one or more pixel-level features within a subset of the plurality of image patches, and the like.

[0121] refer to Fig. 9, the analysis engine 115 can generate an aggregated histology score 912 for the biological sample based on the generated group-level histology scores for each intestinal disease indication group at 911. The aggregated histology score indicates the disease burden in the intestine of the patient and can be an overall score for the overall image prediction of the biological sample. In some embodiments, the analysis engine 115 generates the aggregated histology score 912 by at least applying a machine learning model (such as the machine learning model 910 or another machine learning model trained to determine the overall histology score based on the generated group-level histology scores). In some embodiments, the analysis engine 115 generates the aggregated histology score 912 by at least determining the average of the generated group-level histology scores (e.g., the first group-level histology score 908A, the second group-level histology score 908B, the third group-level histology score 908C, etc.). In some embodiments, the analysis engine 115 averages the group-level histology scores by applying a weight to at least one of the group-level histology scores. For example, the analysis engine 115 may weight one or more group-level histology scores associated with one or more intestinal disease indication groups and / or one or more image patches that include one or more pixel-level features and / or a threshold number of pixel-level features indicated as having a higher contribution to the aggregated histology score. The analysis engine 115 may aggregate the group-level histology scores via one or more other aggregation techniques to generate an aggregated histology score 912.

[0122] Figure 7 A confusion matrix is ​​depicted, which further shows a positive correlation between the aggregated histology scores generated by the analysis engine 115 and the manually assigned histology scores. Figure 8 The third column shows that the analysis engine 115 generates histology scores consistently and accurately at both the slide level and the bowel disease burden group level. Figure 8 As shown, the model employed by the analysis engine 115 to generate the aggregated histology score has a weighted κ of 0.83 and a Spearman correlation of 0.80. In addition, Fig.11 Positive correlations are shown between the end-to-end model employed by the analytics engine 115 and scores, biomarkers, and other tests such as manual NHI scores, machine NHI scores, endoscopy subscores (ES), physician's global assessment (PGA), bowel frequency (SF) biomarkers, rectal bleeding (RB) biomarkers, C-reactive protein (CRP) biomarkers, and fecal calprotectin (FCP) biomarkers.

[0123] Fig.12Depicted is a flow chart showing an example of a process 1200 for cell segmentation and / or tissue region segmentation according to some exemplary embodiments. The process 1200 may be implemented by the analysis engine 115, the segmentation engine 116, the digital pathology platform 110, and / or other components therein.

[0124] At 1202, the segmentation engine 116 receives an image of a biological sample of the intestine from a patient. The image may depict a plurality of cells of the biological sample. The image may further depict a plurality of tissue regions of the biological sample. The image may include a whole slide image, such as a scanned image of H&E stained FFPE tissue.

[0125] At 1204, the segmentation engine 116 segments the received image into a plurality of parts. Each part in the plurality of parts corresponds to a cell in a plurality of cells and / or a tissue region in a plurality of tissue regions. For example, the segmentation engine 116 can segment the image by applying a machine learning model trained to perform segmentation by cells and by tissue. The segmentation engine 116 can assign a cell segmentation label and a tissue region label to each pixel in the image, the cell segmentation label indicating whether the pixel is associated with the cell type of the cell depicted in the image, and the tissue region label indicating whether the pixel is associated with the tissue region type of the tissue region depicted in the image. In certain embodiments, the image is manually annotated with a plurality of annotations identifying a plurality of cell types and a plurality of tissue region types.

[0126] At 1206, the segmentation engine 116 identifies a first spatial coordinate associated with each of the plurality of cells within the image based on at least the segmented image. The segmentation engine 116 may additionally and / or alternatively identify a second spatial coordinate associated with each of the plurality of tissue regions within the image based on at least one segmented image. The first spatial coordinate and / or the second spatial coordinate may be two-dimensional.

[0127] At 1208, the segmentation engine 116 identifies a first cell type associated with the first spatial coordinate. The first cell type is at least one of a neutrophil, a plasma cell, a lymphocyte, an intraepithelial lymphocyte, an eosinophil, a mast cell, a macrophage, a goblet cell, an enterocyte, an endothelial cell, a fibroblast, a smooth muscle cell, and an endothelial cell. Additionally and / or alternatively, the segmentation engine 116 identifies a first tissue region type associated with the second spatial coordinate. The tissue region type is at least one of epithelium, mucosa, submucosa, normal crypt, infiltrated crypt, lumen, blood vessel, lymphatic vessel, lamina propria, muscularis mucosa, basal plasma cell hyperplasia, ulcer, erosion, granulation tissue, infiltrated crypt, crypt abscess, normal collagen, abnormal collagen, stroma, subtype of stroma, hyperplastic muscle, fissure, abscess, normal fat, abnormal fat, serosal membrane, and serositis.

[0128] In some embodiments, the segmentation engine 116 identifies a first cell type based on a plurality of annotations identifying a plurality of cell types depicted in a plurality of images of the biological sample. For example, the plurality of annotations may be manually applied by one or more pathologists and / or may be generated by the digital pathology platform 110. Similarly, the segmentation engine 116 may identify a first tissue region type based on a second plurality of annotations identifying a plurality of tissue region types depicted in a plurality of images of the biological sample. The second plurality of annotations may be manually applied by one or more pathologists and / or may be generated by the digital pathology platform 110. In some embodiments, the segmentation engine 116 generates a metric indicating a confidence level associated with the identified first cell type for each cell in the plurality of cells and / or the identified first tissue region type for each tissue region in the plurality of tissue regions.

[0129] At 1210, the segmentation engine 116 generates a visual representation of an image including the biological sample based at least on the first spatial coordinate and the first cell type. The segmentation engine 116 may further generate a visual representation based at least on the second spatial coordinate and the first tissue region type. The segmentation engine 116 may generate an overlay indicating the first cell type at the first spatial coordinate and / or the first tissue region type at the second spatial coordinate. The overlay may include at least one of a mask, a color, a pattern, etc.

[0130] Additionally and / or alternatively, the segmentation engine 116 generates spatial tabular data including a first spatial coordinate associated with each of a plurality of cells within the image and a first cell type associated with the first spatial coordinate. The segmentation engine 116 may generate spatial tabular data including a second spatial coordinate associated with each of a plurality of tissue regions within the image and a first tissue region type associated with the second spatial coordinate. The spatial tabular data may provide a location of each of the cells and / or tissue regions within the image. The spatial tabular data may be stored and used to generate a visual representation and / or histological score.

[0131] Additionally and / or alternatively, the segmentation engine 116 generates a histology score for the biological sample based on at least the first spatial coordinate and the first cell type associated with the first spatial coordinate and / or the second spatial coordinate and the first tissue region type associated with the second spatial coordinate. In some embodiments, the segmentation engine 116 communicates with the analysis engine 115 to generate the histology score.

[0132] The histological score indicates the disease load in the intestine of the patient. The histological score includes one of the Nancy histological index (NHI) score, the Robarts histopathological index (RHI) score, the Geboes scale score, the overall histological activity score (GHAS), the muscle hyperplasia score, the fibrosis score, etc. The histological score can be generated based on the spatial distribution of multiple cells identified as having a first cell type and / or the spatial distribution of multiple tissue regions identified as having a first tissue region type. Additionally and / or alternatively, the histological score is generated based on the number of cells identified as having a first cell type meeting a threshold cell number. Additionally and / or alternatively, such as when the tissue region type is erosion and / or ulcer, the histological score is generated based on the tissue region type of multiple tissue regions depicted in the image.

[0133] Thus, the digital pathology platform 110 can efficiently generate unbiased localization data, such as spatial tabular data, and other data describing images of biological samples, which can be used to generate visual representations, predict cell and / or tissue regions within images of biological samples, generate histological scores, etc.

[0134] In some embodiments, the spatial table data, visualization and / or histological scores generated by the segmentation engine 116 can be used to calculate the classification histological scores. For example, a graph neural network (GNN) can be implemented by the analysis engine 115 to receive the spatial table data, visualization and / or predicted histological scores (e.g., Nancy scores) and output the classification histological scores (e.g., 0-4 levels). In one or more instances, the GNN can be trained to learn the association between the spatial table data and the classification score. The analysis engine 115 can obtain the spatial table data, the visualization and predicted histological scores, and input the data into the trained GNN to obtain the classification score of a given image. Therefore, the extracted features (such as spatial table data) can be connected to the molecular data of the tissue sample depicted by the image. Then the molecular data / marker can be associated with the phenotype to assess inflammation, which can further support the disease load in the IBD analysis. In some embodiments, the spatial table data can be clustered together using the trained GNN to predict the classification score. The spatial table data includes multifactorial data, and the GNN can be used to compress the data by identifying the patterns and relationships between the data.

[0135] Fig.13 A flow chart illustrating an example of a process 1300 for histology score generation according to some exemplary embodiments is depicted. The process 1300 may be implemented by the analysis engine 115, the segmentation engine 116, the digital pathology platform 110, and / or other components therein.

[0136] At 1302, the analysis engine 115 may determine a plurality of image patches within an image of a biological sample from the intestine of a patient. Each image patch in the plurality of image patches depicts a portion of the biological sample. The image may include a whole slide image, such as a scanned image of H&E stained FFPE tissue.

[0137] At 1304, the analysis engine 115 may determine a plurality of bowel disease indication groups based at least on the plurality of image blocks. Each bowel disease indication group in the plurality of bowel disease indication groups corresponds to a subset of the plurality of image blocks.

[0138] In some embodiments, a subset of multiple image blocks includes common pixel-level features in one or more pixel-level features. For example, a subset of multiple image blocks can be formed by clustering one or more similar image blocks in multiple image blocks at least based on one or more pixel-level features. The analysis engine 115 can perform clustering by applying clustering analysis techniques. Clustering analysis techniques may include one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), and agglomerative hierarchical clustering. In some embodiments, GNN can be used to cluster spatial tabular data to predict classification scores.

[0139] The pixel-level features may be associated with a particular pixel of the image. The pixel-level features may include shape, color, size, presence of dye, intensity, etc. associated with a particular pixel of the image of the biological sample. The pixel-level features may indicate the presence of at least one of erosion of tissue, neutrophils, lymphoid structures, crypt abscesses, and / or intraepithelial debris of tissue in the biological sample.

[0140] At 1304, the analysis engine 115 generates a group-level histology score for each of the plurality of intestinal disease indication groups based at least on a subset of the plurality of image blocks contained in each respective group. The group-level histology score is at least one of a Nancy Histology Index (NHI) score, a Robarts Histopathology Index (RHI) score, a Geboes Scale score, and a Global Histology Activity Score (GHAS). Additionally and / or alternatively, the group-level histology score is at least one of: a first score indicating that there is no disease load in the patient's intestine; a second score indicating a mild disease load in the patient's intestine; and a third score indicating a moderate disease load in the patient's intestine. The score range is 0-4 (or other scales, depending on the scoring system). In the example, the first score is 0, the second score is 1, and the third score is 2.

[0141] The analysis engine 115 can generate a group-level histology score by at least applying at least one machine learning model to predict the histology score for each intestinal disease indication group of the image block. The at least one machine learning model can include a multiple instance learning (MIL) model and other models, which can be trained to generate a group-level histology score by at least determining a representation encoding of a subset of multiple image blocks. In some embodiments, at least one machine learning model is trained on the cell segmentation labels and tissue region segmentation labels generated by the segmentation engine 116. For example, process 1200 or similar processes can be used for cell segmentation and / or tissue region segmentation.

[0142] In some embodiments, a group-level histology score is generated while determining a representation code for a subset of the plurality of image blocks. For example, the analysis engine 115 may assign a higher attention score to a first image block in the subset of the plurality of image blocks compared to a second image block in the subset of the plurality of image blocks based at least on the presence of the first pixel-level feature or the absence of the first pixel-level feature. The higher attention score indicates that the first pixel-level feature of the first image block contributes more to the representation code for the subset of the plurality of image blocks than the second block.

[0143] Additionally and / or alternatively, the analysis engine 115 generates a group-level histology score based on at least one or more of the number of one or more pixel-level features within a subset of the plurality of image patches and the distribution of the one or more pixel-level features within the subset of the plurality of image patches. In some embodiments, the presence of a first pixel-level feature of the one or more pixel-level features in the plurality of image patches is associated with a first possible histology score, and the absence of the first pixel-level feature is associated with a second possible histology score. Thus, the presence and / or absence, number and / or distribution of pixel-level features within a subset of the plurality of image patches may be used to generate a histology score.

[0144] At 1306, the analysis engine 115 generates an aggregated histology score for the biological sample based on the generated group-level histology scores for each bowel disease indication group. The aggregated histology score indicates the disease burden in the patient's intestine. The aggregated histology score is an overall score predicted for the image of the microscope slide.

[0145] The aggregated histology score is at least one of a Nancy histology index (NHI) score, a Robarts histopathology index (RHI) score, a Geboes scale score, and a global histology activity score (GHAS). Additionally and / or alternatively, the aggregated histology score is at least one of: a first score indicating a low disease burden in the patient's intestine; a second score indicating a moderate disease burden in the patient's intestine; and a third score indicating a high disease burden in the patient's intestine. The score ranges from 0-4 (or other scales, depending on the scoring system). In an example, the first score is 1, the second score is 2 or 3, and the third score is 4.

[0146] In some embodiments, the analysis engine 115 generates a first visual representation of the reduced dimensionality representation of the plurality of image blocks. The first visual representation may include one or more visual indicators (e.g., overlays, colors, masks, etc.). The one or more visual indicators may display different pixel-level features. The one or more visual indicators may indicate the contribution of the pixel-level features to the histological scores of the possible group levels. The one or more visual indicators provide a visual distinction between the image blocks in the plurality of image blocks, which indicates different possible histological scores.

[0147] The analysis engine 115 can generate a first visual representation by applying a dimensionality reduction technique to at least the pixel-level representation of each of the plurality of image blocks. The dimensionality reduction technique includes one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and T-distributed stochastic neighbor embedding (t-SNE). In addition, GNNs can be used to reduce the dimensionality of spatial tabular data by detecting patterns and relationships in multidimensional data sets.

[0148] Thus, the digital pathology platform 110 can efficiently generate reproducible and consistent aggregated histology scores for biological samples. Reproducible and consistent aggregated histology scores can be generated according to the embodiments described herein while limiting or eliminating subjectivity that previously reduced the reliability of such histology scores.

[0149] Fig.14 A block diagram illustrating an example of a computing system 1400 is depicted in accordance with some example embodiments. Figure 1 and 14 , the computing system 1400 may be used to implement the digital pathology platform 110 , the client device 130 , the analysis engine 115 , the segmentation engine 116 , and / or any components thereof.

[0150] like Fig.14As shown, the computing system 1400 may include a processor 1410, a memory 1420, a storage device 1430, and an input / output device 1440. The processor 1410, the memory 1420, the storage device 1430, and the input / output device 1440 may be interconnected via a system bus 1450. The processor 1410 is capable of processing instructions for execution within the computing system 1400. Such executed instructions may implement, for example, one or more components of the digital pathology platform 110, the client device 130, the analysis engine 115, the segmentation engine 116, etc. In some exemplary embodiments, the processor 1410 may be a single-threaded processor. Alternatively, the processor 1410 may be a multi-threaded processor. The processor 1410 is capable of processing instructions stored on the memory 1420 and / or the storage device 1430 to display graphical information for a user interface provided via the input / output device 1440.

[0151] Memory 1420 is a computer-readable medium, such as a volatile or non-volatile computer-readable medium, that stores information within computing system 1400. For example, memory 1420 may store a data structure representing a configuration object database. Storage device 1430 is capable of providing persistent storage for computing system 1400. Storage device 1430 may be a floppy disk device, a hard disk device, an optical disk device, a tape device, or other suitable persistent storage device. Input / output device 1440 provides input / output operations for computing system 1400. In some exemplary embodiments, input / output device 1440 includes a keyboard and / or a pointing device. In various specific implementations, input / output device 1440 includes a display unit for displaying a graphical user interface.

[0152] According to some exemplary embodiments, the input / output device 1440 may provide input / output operations for network devices. For example, the input / output device 1440 may include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0153] In some exemplary embodiments, computing system 1400 can be used to execute various interactive computer software applications that can be used to organize, analyze and / or store data in various formats. Alternatively, computing system 1400 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents and / or any other objects, etc.), computing functions, communication functions, etc. The application can include various additional functions or can be an independent computing product and / or function. After activation within the application, the function can be used to generate a user interface provided via input / output device 1440. The user interface can be generated by computing system 1400 and presented to the user (e.g., on a computer screen monitor, etc.).

[0154] One or more aspects or features of the subject matter described herein can be implemented with digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include a specific implementation in one or more computer programs, which are executable and / or interpretable on a programmable system, which includes at least one programmable processor (which can be dedicated or general, coupled to receive data and instructions from it and send data and instructions to it), a storage system, at least one input device, and at least one output device. A programmable system or computing system can include a client and a server. Typically, the client and the server are remotely arranged from each other, and generally interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on each computer and a client-server relationship between each other.

[0155] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, device and / or device (such as, for example, a disk, an optical disk, a memory and a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily (such as, for example, a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium). A machine-readable medium may store such machine instructions alternatively or additionally in a transient manner (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores).

[0156] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and a pointing device (such as, for example, a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive tracking pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0157] In the above description and claims, phrases such as "at least one" or "one or more of" may appear, followed by a list of combinations of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, the phrase is intended to represent any element or feature listed alone, or any other described element or feature in combination with any other described element or feature. For example, the phrases "at least one of A and B"; "one or more of A and B"; "A and / or B" are each intended to represent "A alone, B alone, or A and B together". A similar interpretation also applies to lists including three or more items. For example, the phrases "at least one of A and B"; "one or more of A, B, and C" and "A, B, and / or C" are each intended to represent "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together". Use of the term "based on" above and in the claims is intended to mean "based, at least in part, on" such that non-recited features or elements are also permissible.

[0158] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The embodiments described in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are only some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to those features and / or variations described herein, other features and / or variations may also be provided. For example, the above-mentioned specific implementations may be directed to various combinations and sub-combinations of the disclosed features and / or to combinations and sub-combinations of several further features disclosed above. In addition, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other specific implementations may be within the scope of the following claims.

Claims

1. A computer-implemented method comprising: determining a plurality of image patches within an image of a biological sample from the intestine of a patient, wherein each image patch of the plurality of image patches depicts a portion of the biological sample; determining a plurality of intestinal disease indication groups based at least on the plurality of image blocks, wherein each intestinal disease indication group in the plurality of intestinal disease indication groups corresponds to a subset of the plurality of image blocks; For each of the plurality of intestinal disease indication groups, generating a group-level histological score based at least on the subset of the plurality of image blocks included in each corresponding group; as well as Based on the generated group-level histology scores for each bowel disease indication group, an aggregated histology score is generated for the biological sample, wherein the aggregated histology score is indicative of a disease burden in the intestine of the patient.

2. The method of claim 1, wherein the group-level histology score and the aggregated histology score are each one of a Nancy Histology Index (NHI) score, a Robarts Histopathology Index (RHI) score, a Geboes Scale score, and a Global Histology Activity Score (GHAS).

3. The method of claim 1 or claim 2, wherein the group-level histology score and the aggregated histology score are at least one of: a first score indicating no disease burden; a second score indicating a low disease burden in the intestine of the patient; a third score indicating a moderate disease burden in the intestine of the patient; and a fourth score indicating a high disease burden in the intestine of the patient.

4. The method according to any one of claims 1 to 3, wherein the subset of the plurality of image blocks is formed at least by clustering one or more similar image blocks in the plurality of image blocks based at least on one or more pixel-level features.

5. The method of claim 4, wherein the presence of a first pixel-level feature among the one or more pixel-level features in the plurality of image blocks is associated with a first possible histological score; and wherein the absence of the first pixel-level feature is associated with a second possible histological score.

6. A method according to claim 5, wherein generating the group-level histological score includes assigning a higher attention score to a first image block in the subset of the multiple image blocks compared to a second image block in the subset of the multiple image blocks based at least on the presence of the first pixel-level feature or the absence of the first pixel-level feature; and wherein the group-level histological score is generated while determining a representation encoding of the subset of the multiple image blocks.

7. The method of claim 6, wherein the higher attention score indicates that the first pixel-level feature of the first image block contributes more to the representation encoding of the subset of the plurality of image blocks than the second block.

8. The method of claim 4, wherein the one or more pixel-level features represent the presence of at least one of erosion of tissue, neutrophils, lymphoid structures, crypt abscesses, and intraepithelial debris of tissue in the biological sample.

9. The method of claim 4, wherein the one or more pixel-level features include at least one of shape, color, size, presence of a dye, and intensity associated with a pixel of the image of the biological sample.

10. The method of claim 4, wherein the subset of the plurality of image blocks comprises common pixel-level features among the one or more pixel-level features.

11. The method of claim 4, wherein the group-level histology score is generated based on at least one or more of a quantity of the one or more pixel-level features within the subset of the plurality of image patches and a distribution of the one or more pixel-level features within the subset of the plurality of image patches.

12. The method according to claim 4, wherein the clustering is performed by applying a cluster analysis technique.

13. The method of claim 12, wherein the clustering analysis technique comprises one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

14. The method of any one of claims 1 to 13, further comprising generating a first visual representation of a reduced-dimensional representation of the plurality of image patches. 15 . The method of claim 14 , wherein the first visual representation comprises one or more visual indicators configured to indicate a contribution of pixel-level features to a possible group-level histology score. 16 . The method of claim 14 , wherein the first visual representation comprises one or more visual indicators configured to provide visual distinction between image blocks in the plurality of image blocks indicating different possible histological scores.

17. The method of claim 14, wherein the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel-level representation of each of the plurality of image blocks.

18. The method of claim 17, wherein the dimensionality reduction technique comprises one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

19. A method according to any one of claims 1 to 18, wherein the group-level histological scores and the aggregated histological scores are each generated by applying at least one machine learning model, and the at least one machine learning model is trained to generate the group-level histological scores and the aggregated histological scores by at least determining a representation encoding of the subset of the multiple image blocks.

20. The method of claim 19, wherein the at least one machine learning model comprises a multiple instance learning (MIL) model.

21. A system comprising: at least one data processor; as well as at least one memory storing instructions that, when executed by the at least one data processor, cause the at least one data processor to: determining a plurality of image patches within an image of a biological sample from the intestine of a patient, wherein each image patch of the plurality of image patches depicts a portion of the biological sample; determining a plurality of intestinal disease indication groups based at least on the plurality of image blocks, wherein each intestinal disease indication group in the plurality of intestinal disease indication groups corresponds to a subset of the plurality of image blocks; For each of the plurality of intestinal disease indication groups, generating a group-level histological score based at least on the subset of the plurality of image blocks included in each corresponding group; as well as Based on the generated group-level histology scores for each bowel disease indication group, an aggregated histology score is generated for the biological sample, wherein the aggregated histology score is indicative of a disease burden in the intestine of the patient.

22. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, perform operations comprising: determining a plurality of image patches within an image of a biological sample from the intestine of a patient, wherein each image patch of the plurality of image patches depicts a portion of the biological sample; determining a plurality of intestinal disease indication groups based at least on the plurality of image blocks, wherein each intestinal disease indication group in the plurality of intestinal disease indication groups corresponds to a subset of the plurality of image blocks; For each of the plurality of intestinal disease indication groups, generating a group-level histological score based at least on the subset of the plurality of image blocks included in each corresponding group; as well as Based on the generated group-level histology scores for each bowel disease indication group, an aggregated histology score is generated for the biological sample, wherein the aggregated histology score is indicative of a disease burden in the intestine of the patient.

23. A computer-implemented method comprising: receiving an image of a biological sample from the intestine of a patient, the image depicting a plurality of cells of the biological sample; segmenting the received image into a plurality of portions, each portion of the plurality of portions corresponding to a cell of the plurality of cells; identifying, based at least on the segmented image, a first spatial coordinate associated with each of the plurality of cells within the image; identifying a first cell type associated with the first spatial coordinate; as well as A visual representation comprising the image of the biological sample is generated based on at least the first spatial coordinates and the first cell type.

24. The method of claim 23, wherein the identifying is further based at least on a plurality of annotations identifying a plurality of cell types depicted in a plurality of images of the biological sample.

25. The method of claim 23 or claim 24, wherein the first cell type is at least one of a neutrophil, a plasma cell, a lymphocyte, an intraepithelial lymphocyte, an eosinophil, a mast cell, a macrophage, a goblet cell, an enterocyte, an endothelial cell, a fibroblast, a smooth muscle cell, and an endothelial cell.

26. The method of any one of claims 23 to 25, wherein the image further depicts a plurality of tissue regions of the biological sample.

27. The method of claim 26, further comprising: Second spatial coordinates associated with each tissue region of the plurality of tissue regions and a first tissue region type associated with the second spatial coordinates are identified based at least on the segmented image.

28. The method of claim 27, further comprising: A second visual representation of the image including the biological sample is generated based on at least the second spatial coordinates and the first tissue region type.

29. The method of claim 27, wherein the tissue region type is at least one of epithelial, mucosa, submucosa, normal crypts, infiltrated crypts, lumen, vessel, lymphatic vessel, lamina propria, muscularis mucosae, basal plasmacytosis, ulcer, erosion, granulation tissue, infiltrated crypts, crypt abscesses, normal collagen, abnormal collagen, stroma, a subtype of stroma, hyperplastic muscle, fissure, abscess, normal fat, abnormal fat, serosa, and serositis.

30. The method of claim 27, wherein identifying the second spatial coordinates and the first tissue region type is further based on at least a second plurality of annotations identifying a plurality of tissue region types depicted in a plurality of images of the biological sample.

31. The method of any one of claims 23 to 30, wherein the identifying comprises generating a metric indicative of a confidence level associated with the identified first cell type for each cell in the plurality of cells.

32. The method according to any one of claims 23 to 31, further comprising: Spatial tabular data is generated, the spatial tabular data including the first spatial coordinates associated with each cell of the plurality of cells within the image and the first cell type associated with the first spatial coordinates.

33. The method according to any one of claims 23 to 32, further comprising: A histology score is generated for the biological sample based on at least the first spatial coordinate and the first cell type associated with the first spatial coordinate, wherein the histology score is indicative of a disease burden in the intestine of the patient.

34. The method of claim 33, wherein the histological score is one of a Nancy Histological Index (NHI) score, a Robarts Histopathological Index (RHI) score, a Geboes Scale score, a Global Histological Activity Score (GHAS), a muscle hyperplasia score, and a fibrosis score.

35. The method of claim 33, wherein the first cell type comprises neutrophils.

36. The method of claim 33, wherein the histological score is further generated based on a spatial distribution of the plurality of cells identified as having the first cell type.

37. The method of claim 33, wherein the histological score is further generated based on a number of cells of the plurality of cells identified as having the first cell type meeting a threshold number of cells.

38. The method of claim 33, wherein the histology score is further generated based on tissue region types of a plurality of tissue regions depicted in the image.

39. The method of claim 38, wherein the tissue region type is at least one of an erosion and an ulcer.

40. A method according to any one of claims 23 to 39, wherein the first spatial coordinate is two-dimensional.

41. The method according to any one of claims 23 to 40, further comprising: generating an overlay indicating the first cell type at the first spatial coordinate based at least on the first spatial coordinate and the first cell type; Wherein the overlay comprises at least one of a mask, a color and a pattern.

42. A method according to any one of claims 23 to 41, wherein the image is segmented by applying a machine learning model, and the machine learning model is trained to perform cell-based segmentation and tissue-based segmentation by assigning a cell segmentation label and a tissue region label to at least each pixel in the image, the cell segmentation label indicating whether the pixel is associated with a cell type of a cell depicted in the image, and the tissue region label indicating whether the pixel is associated with a tissue region type of a tissue region depicted in the image.

43. A system comprising: at least one data processor; as well as at least one memory storing instructions that, when executed by the at least one data processor, cause the at least one data processor to: receiving an image of a biological sample from the intestine of a patient, the image depicting a plurality of cells of the biological sample; segmenting the received image into a plurality of portions, each portion of the plurality of portions corresponding to a cell of the plurality of cells; identifying, based at least on the segmented image, a first spatial coordinate associated with each of the plurality of cells within the image; identifying a first cell type associated with the first spatial coordinate; as well as A visual representation comprising the image of the biological sample is generated based on at least the first spatial coordinates and the first cell type.

44. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, perform operations comprising: receiving an image of a biological sample from the intestine of a patient, the image depicting a plurality of cells of the biological sample; segmenting the received image into a plurality of portions, each portion of the plurality of portions corresponding to a cell of the plurality of cells; identifying, based at least on the segmented image, a first spatial coordinate associated with each of the plurality of cells within the image; identifying a first cell type associated with the first spatial coordinate; as well as A visual representation comprising the image of the biological sample is generated based on at least the first spatial coordinates and the first cell type.

Images