Transformer-based multi-modal and multi-region data fusion framework

Abstract

The present disclosure relates to a method. The method includes applying one or more multi-stage hierarchical transformations on data corresponding to one or more first regions of interest (ROIs) within a digitized pathology image of a patient to generate a pathology feature map and on data corresponding to one or more second ROIs from a radiological image of the patient to generate a radiology feature map. Respective stages of the one or more multi-stage hierarchical transformations comprise a self-attention operation. The radiology feature map and the pathology feature map are combined to generate a fused feature map. The fused feature map is operated upon with a machine learning model to generate one or more medical predictions for the patient.

Description

TRANSFORMER-BASED MULTI-MODAL AND MULTI-REGION DATA FUSION FRAMEWORK REFERENCE TO RELATED APPLICATION

[0001] This Application claims the benefit of U.S. Provisional Application No. 63 / 588,402, filed on October 6, 2023, the contents of which are incorporated by reference in their entirety. FEDERAL FUNDING NOTICE

[0002] This invention was made with government support under R01CA249992- 01A1 awarded by the National Institutes of Health. The government has certain rights in the invention. BACKGROUND

[0003] In recent years, there has been significant interest in developing computer vision systems for interrogating medical images. Computer vision systems are computer systems that utilize artificial intelligence to analyze images (e.g., medical images). Such systems have the potential to improve health care for patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various example operations, apparatus, methods, and other example embodiments of various aspects discussed herein. It will be appreciated that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one example of the boundaries. One of ordinary skill in the art will appreciate that, in some examples, one element can be designed as multiple elements or that multiple elements can be designed as one element. In some examples, an element shown as an internal component of another element may be implemented as an external component and vice versa. Furthermore, elements may not be drawn to scale.

[0005] Fig.1 illustrates a block diagram of some embodiments of a computer vision assessment system configured to incorporate multi-modal and multi-region imaging analysis to determine one or more medical predictions.

[0006] Fig.2 illustrates a block diagram of some additional embodiments of a computer vision assessment system configured to incorporate multi-modal and multi- region imaging analysis to determine one or more medical predictions.

[0007] Fig.3 illustrates a block diagram of some additional embodiments of a computer vision assessment system configured to incorporate multi-modal and multi- region imaging analysis to determine one or more medical predictions.

[0008] Fig.4 illustrates some embodiments of a flow diagram showing a method for incorporating multi-modal and multi-region imaging analysis to determine one or more medical predictions.

[0009] Fig.5 illustrates exemplary Kaplan-Meier (KM) graphs showing a predicted overall survival for different feature types.

[0010] Fig.6 illustrates a table showing exemplary performance parameters for different combinations of features maps and machine learning systems.

[0011] Fig.7 illustrates a block diagram of some additional embodiments of a computer vision assessment system configured to incorporate multi-modal and multi- region imaging analysis to determine one or more medical predictions.

[0012] Fig.8 illustrates some additional embodiments of a computer vision assessment system configured to incorporate multi-modal and multi-region imaging analysis to determine one or more medical predictions. DETAILED DESCRIPTION

[0013] The description herein is made with reference to the drawings, wherein like reference numerals are generally utilized to refer to like elements throughout, and wherein the various structures are not necessarily drawn to scale. In the following description, for purposes of explanation, numerous specific details are set forth in order to facilitate understanding. It may be evident, however, to one of ordinary skill in the art, that one or more aspects described herein may be practiced with a lesser degree of these specific details. In other instances, known structures and devices are shown in block diagram form to facilitate understanding.

[0014] Modern-day oncology depends on the accurate examination of various sources of data such as radiology, histology, genomics, and / or clinical records. There has been significant recent interest in developing computer vision tools for interrogating radiologic images (e.g., computed tomography (CT) scans) or pathology images (e.g.,Whole Slide Images) for oncology outcome and treatment response prediction. However, most computer vision tools comprise unimodal deep learning (DL) models that focus on a single image modality. For example, a computer vision tool may focus on either radiomic approaches to characterize anatomical information from radiology images (e.g., using texture features of a tumor, shape features of a tumor, etc.) or computational pathology (e.g., immune architecture) to build prognostic models at a cellular level for cancer diagnosis and / or outcome prediction.

[0015] In this regard, different types of medical images are often processed independently and target a single region of interest. However, such approaches are unable to take into consideration complex interactions between scales and locations that these similar data structures implicitly share. It has been appreciated that given the complementary nature of radiology and pathology imaging data (e.g., both containing enriched outcome-related information), combining these two modalities for outcome prediction may improve a prognostic ability of a computer vision system compared to unimodal data modeling strategies.

[0016] The present disclosure relates a computer vision assessment system, and an associated method, that incorporates both multi-modal imaging analysis (e.g., using different image types) and multi-region imaging analysis (e.g., using different locations within an image) to determine one or more medical predictions for a patient. In some embodiments, the method may be performed by obtaining a plurality of fragments associated with one or more regions of interest (ROIs) of digitized pathology imaging data from a patient and further obtaining a plurality of patches associated with one or more ROIs of radiological imaging data from the patient. Multiple stages of hierarchical transformations are performed on the plurality of fragments to generate a pathology feature map and on the plurality of patches to generate a radiology feature map. The pathology feature map and the radiology feature map are subsequently combined to form a fused feature map. A machine learning model is then operated upon the fused feature map to determine one or more medical predictions for the patient. By using multiple stages of hierarchical transformations to generate separate feature maps that are subsequently combined and provided to a machine learning model, features from both the digitized pathology imaging data and the radiology imaging data may be taken into consideration when generating the one or more medical predictions of the patient, thereby improving an accuracy of the one or more medical predictions.

[0017] Fig.1 illustrates a block diagram of some embodiments of a computer vision assessment system 100 configured to incorporate multi-modal and multi-region imaging analysis to determine one or more medical predictions.

[0018] The computer vision assessment system 100 comprises a data set 102 including multi-modal patient data 103 having a plurality of different modalities. In some embodiments, the multi-modal patient data 103 may include digitized pathology imaging data 104 and radiological imaging data 106. In some embodiments, the digitized pathology imaging data 104 includes one or more digitized images of a tissue sample taken from a patient. For example, the digitized pathology imaging data 104 may comprise a whole slide image (WSI) of an H&E (Hematoxylin and eosin) stained slide. In some embodiments, the radiological imaging data 106 includes one or more digital images that have been obtained through radiological imaging (e.g., a CT (computed tomography) scan, an MRI (magnetic resonance imaging) scan, an MRE (Magnetic Resonance Enterography) scan, an x-ray, an ultrasound scan, and / or the like). In other embodiments, the multi-modal patient data 103 may comprise different and / or additional modalities of patient data (e.g., clinical data, molecular data, etc.)

[0019] In some embodiments, a pre-processing tool 108 may be configured to respectively separate the multi-modal patient data 103 into a plurality of data fragments 109 associated with one or more regions of interest (ROIs) within a respective modality of data. For example, the pre-processing tool 108 may separate a first modality of patient data into a first plurality of data fragments associated with a first modality of data, separate a second modality of patient data into a second plurality of data fragments associated with a second modality of data, etc.

[0020] In some embodiments, the pre-processing tool 108 is configured to separate the digitized pathology imaging data 104 into a plurality of fragments 110 that respectively relate to one or more non-overlapping regions of interest (ROIs) within the digitized pathology imaging data 104. In some embodiments, the pre-processing tool 108 is further configured to separate the radiological imaging data 106 into a plurality of patches 112 that respectively relate to different non-overlapping ROIs within the radiological imaging data 106. For example, the radiological imaging data 106 may be broken into a first patch 112a that corresponds to a tumor region and a second patch 112b that corresponds to a lymph node. Although Fig.1 is illustrated as having two fragments and two patches, it will be appreciated that in other embodiments, thedigitized pathology imaging data 104 and the radiological imaging data 106 may be broken into more than two fragments and / or patches.

[0021] Data corresponding to the one or more ROIs within the multi-modal patient data 103 is provided to a multi-stage hierarchical transformer tool 114. In some embodiments, the data may be provided to the multi-stage hierarchical transformer tool 114 as the plurality of data fragments 109. In other embodiments, the data may be provided to the multi-stage hierarchical transformer tool 114 in its entirety (e.g., as a whole digitized pathology image and / or radiological image). The multi-stage hierarchical transformer tool 114 is configured to operate one or more multi-stage hierarchical transformations on the data to generate a plurality of feature maps 129 respectively corresponding to the different modalities of data.

[0022] For example, in some embodiments the plurality of fragments 110 and the plurality of patches 112 are provided to the multi-stage hierarchical transformer tool 114. The multi-stage hierarchical transformer tool 114 is configured to perform multi-scale and multi-region processing upon the plurality of fragments 110 to form a pathology feature map 130 and upon the plurality of patches 112 to form a radiology feature map 132.

[0023] In some embodiments, the multi-stage hierarchical transformer tool 114 comprises a series of transformer stages 116a-116n. The series of transformer stages 116a-116n are respectively configured to operate at different scales, thereby allowing the multi-stage hierarchical transformer tool 114 to perform one or more multi-stage transformations that take into consideration different scales (e.g., macro-scale tumor texture and shape along with cellular architecture) across different modalities. The series of transformer stages 116a-116n may also perform attention at each stage. By performing attention at each stage, the multi-stage hierarchical transformer tool 114 is able to provide for cross-region attention between different regions of an image, thereby allowing for different regions to be accounted for by the computer vision assessment system 100 (e.g., allowing for the computer vision assessment system 100 to take into consideration relevant features not only a tumor region, but also surrounding or adjacent regions). In some embodiments, the series of transformer stages 116a-116n may comprise four stages. In other embodiments, the series of transformer stages 116a-116n may comprise more or less than four stages.

[0024] In some embodiments, the multi-stage hierarchical transformer tool 114 may comprise a multi-stage Swin Transformer tool having a pathology Swin Transformerpath 118 configured to generate the pathology feature map 130 and a radiology Swin Transformer path 124 configured to generate the radiology feature map 132. In some embodiments, the pathology Swin Transformer path 118 comprises a pathology processing stage 119 within each of the series of transformer stages 116a-116n. The pathology processing stage 119 includes a pathology Swin Transformer block 120 and a pathology self-attention tool 122. The pathology Swin Transformer block 120 is configured to process inputs (e.g., embedded tokens comprising the plurality of fragments 110) and to generate outputs (e.g., encoded pathology vectors). The outputs are provided to the pathology self-attention tool 122, which is configured to perform self- attention operations on the different outputs (e.g., on encoded pathology vectors output from an upstream radiology Swin Transformer).

[0025] In some embodiments, the radiology Swin Transformer path 124 comprises a radiology processing stage 123 disposed within each of the series of transformer stages 116a-116n. The radiology processing stage 123 includes a plurality of radiology Swin Transformers 126 and a radiology self-attention tool 128. The plurality of radiology Swin Transformers 126 are respectively configured to process an input (e.g., a token comprising one of the plurality of patches 112) and to generate an output (e.g., an encoded radiology vector). The outputs of the plurality of radiology Swin Transformers 126 are provided to the radiology self-attention tool 128, which is configured to perform self-attention operations on the different outputs (e.g., on encoded radiology vectors output from upstream radiology Swin Transformers).

[0026] The plurality of feature maps 129 are provided to a fusion tool 134 configured to combine the plurality of feature maps 129 to generate a fused feature map 136 comprising information relating to one or more ROI (e.g., different ROIs) of the multi- modal patient data 103. For example, in some embodiments, the pathology feature map 130 and the radiology feature map 132 are provided to the fusion tool 134, which is configured to combine the pathology feature map 130 and the radiology feature map 132 to form a fused feature map 136. The fused feature map 136 may comprise information relating to different regions of the digitized pathology imaging data 104 and the radiological imaging data 106.

[0027] The fused feature map 136 may be subsequently provided to a machine learning model 138. The machine learning model 138 is configured to operate upon the fused feature map 136 to generate one or more medical predictions 140. In someembodiments, the one or more medical predictions 140 may relate to a clinical endpoint having a binary output (e.g., a cancer grade, a genotype, a benign / malignancy status, and / or the like) and / or a time based event (e.g., recurrence, 5-year survival, overall survival, and / or the like). Operating the machine learning model 138 on the fused feature map 136 allows for the one or more medical predictions 140 to be generated based upon information relating to different image modalities and different regions within each of the image modalities. The one or more medical predictions 140 are therefore able to achieve improved results (e.g., AUC) over machine learning models that utilize single regions of radiological or pathological imaging data.

[0028] Fig.2 illustrates a block diagram of some additional embodiments of a computer vision assessment system 200 configured to incorporate multi-modal and multi-region analysis to determine one or more medical predictions.

[0029] The computer vision assessment system 200 comprises a data set 102 (e.g., an imaging data set) that includes digitized pathology imaging data 104 and radiological imaging data 106 from a patient. In some embodiments, the patient may comprise a cancer patient (e.g., a patient having, suspected of having, and / or having had cancer). In some embodiments, the cancer patient may have head and neck cancer, lung cancer, prostate cancer, breast cancer, and / or the like. The data set 102 may be stored in electronic memory 201. In various embodiments, the electronic memory 201 may comprise solid state memory, SRAM (static random-access memory), DRAM (dynamic random-access memory), and / or the like.

[0030] In some embodiments, the digitized pathology imaging data 104 may comprise two-dimensional (2D) data. For example, the digitized pathology imaging data 104 may comprise a digitized 2D stained whole slide image (WSI) 202. In some embodiments, the radiological imaging data 106 may comprise three-dimensional (3D) data. For example, the radiology imaging data 106 may comprise a 3D computed tomography (CT) image 204.

[0031] A pre-processing tool 108 is configured to separate the digitized pathology imaging data 104 into a plurality of fragments 110a-110b that respectively relate to different regions of interest (ROIs). In some embodiments, the plurality of fragments 110 may comprise tokens (e.g., embedded vectors corresponding to the different ROI). In some such embodiments, the pre-processing tool 108 may comprise an embedding layer configured to project pixel values within the different ROI onto the tokens. Thepre-processing tool 108 is further configured to separate the radiological imaging data 106 into a plurality of patches 112 that respectively relate to different ROIs. In some embodiments, the plurality of patches 112 may comprise non-overlapping 3D patches respectively including pixel values from the different ROIs. In some embodiments, the plurality of patches 112 are discontinuous with one another (e.g., taken from different parts of the image that are not adjacent to one another). In some embodiments, the pre- processing tool 108 may be implemented as computer code run by a processing unit (e.g., a central processing unit including one or more transistor devices configured to operate computer code to achieve a result, a microcontroller, or the like).

[0032] The plurality of fragments 110 and the plurality of patches 112 are provided to a multi-stage hierarchical transformer tool 114 configured to perform a hierarchical encoding that reconciles multi-scale representational differences (e.g., macro-scale tumor texture and shape with cellular architecture) across the plurality of fragments 110 and the plurality of patches 112, thereby leading to improved performance of the computer vision assessment system 200. The multi-stage hierarchical transformer tool 114 is configured to operate upon the plurality of fragments 110 to build a pathology feature map 130 and to operate upon the plurality of patches 112 to build a radiology feature map 132. In some embodiments, the multi-stage hierarchical transformer tool 114 may comprise a machine learning model (e.g., a deep learning model) run by a processing unit (e.g., a central processing unit including one or more transistor devices configured to operate computer code to achieve a result).

[0033] In some embodiments, the multi-stage hierarchical transformer tool 114 comprises a series of transformer stages 116a-116n respectively configured to operate at different scales. The series of transformer stages 116a-116n are configured to perform a series of hierarchical transformation operations to generate the pathology feature map 130 and the radiology feature map 132. For example, a first transformer stage 116a is configured to operate upon plurality of fragments 110 and the plurality of patches 112 to generate an output that is provided to a second transformer stage 116b, etc. In some embodiments, the multi-stage hierarchical transformer tool 114 comprises a pathological Swin Transformer path 118 including a plurality of pathology processing stages 119a-119n and a radiological Swin Transformer path 124 including a plurality of radiology processing stages 123a-123n.

[0034] In some embodiments, the plurality of pathology processing stages 119a- 119n respectively comprise a pathology Swin Transformer block 120 and a pathology self-attention tool 122. In some embodiments, the pathology Swin Transformer block 120 is a 2D Swin Transformer. The plurality of fragments 110 (e.g., respectively represented as separate embeddings) flow through the pathology Swin Transformer block 120 in parallel. The pathology self-attention tool 122 is applied to outputs of the pathology Swin Transformer block 120 to provide for cross-region attention between the plurality of fragments 110. By applying cross-region attention between the plurality of fragments 110, different information (e.g., tumor heterogeneity) from the different ROI can be considered by a downstream machine learning model. Furthermore, applying the attention in consistent stepwise resolutions (e.g., after each hierarchical transformation operation) provides for inherent multi-scale associations between imaging at the micro- and macro-scales.

[0035] In some embodiments, the plurality of radiology processing stages 123a-123n respectively comprise a plurality of radiology Swin Transformers 126a-126J and a radiology self-attention tool 128. The plurality of radiology Swin Transformers 126a- 126J may have a same structure and are configured to respectively operate on one of the plurality of patches 112 (e.g., represented as vectors) in parallel. For example, a first radiology Swin Transformer 126a within a first transformer stage 116a may operate on a first patch, a second radiology Swin Transformer within the first transformer stage 116a may operate on a second patch 112b, etc. In some embodiments, the number of the plurality of radiology Swin Transformers 126a-126J is equal to a number of the plurality of patches 114a-114J (e.g., if there are 10 patches there will be 10 radiology Swin Transformers). In some embodiments, the plurality of radiology Swin Transformers 126a-126J are 3D Swin Transformers. The radiology self-attention tool 128 is configured to apply self-attention to outputs of the plurality of radiology Swin Transformers 126a-126J. The application of self-attention allows for the plurality of radiology processing stages 123a-123n to correlate information between the different patches (e.g., between patches within a primary tumor and patches within a lymph node).

[0036] In some embodiments, the pathology self-attention tool 122 and the radiology self-attention tool 128 are configured to perform self-attention at a plurality of self- attention nodes, 121 and 127, by multiplying encoded vectors output from an upstreamSwin Transformer block with weights to get learnable weight matrices called query, key, and value matrices. A resulting attention matrix is then calculated using the query, key, and value matrices. It will be appreciated that each of the self-attention nodes, 121 and 127, calculates a self-attention value based on different parameters (e.g., based on different query, key, and value matrices) since each of the self-attention nodes generates their query, key, and value matrices based upon different upstream inputs. For example, a self-attention node downstream of a Swin Transformer block that receives a lymph node patch may generate query, key, and value matrices in a manner that places more emphasis on the lymph node (e.g., the query may be generated from the lymph node and the key from the tumor), while a self-attention node downstream of a Swin Transformer block that receives a tumor patch may generate query, key, and value matrices in a manner that places more emphasis on the tumor patch (e.g., the query may be generated from the tumor and the key from the lymph node). Therefore, the self-attentions calculated by each of the self-attention nodes, 121 and 127, jointly contribute to an overall interaction between the different fragments and / or patches.

[0037] Combination blocks 206-208 are configured to receive the outputs of the pathological Swin Transformer path 118 and the radiological Swin Transformer path 124. The combination blocks 206-208 are configured to respectively use the outputs to generate a pathology feature map 130 and a radiology feature map 132. In some embodiments, the combination blocks 206-208 may comprise attention pooling blocks. The attention pooling blocks may include a pathology attention pooling block configured to use (e.g., sum) outputs from the pathological Swin Transformer path 118 to form the pathology feature map 130 and a radiology attention pooling block configured to use (e.g., sum) outputs from the radiological Swin Transformer path 124 to form the radiology feature map 132. In other embodiments, the combination blocks 206-208 may comprise concatenation blocks configured to concatenate outputs of the pathological Swin Transformer path 118 to form the pathology feature map 130 and to further concatenate outputs of the radiological Swin Transformer path 124 to form the radiology feature map 132.

[0038] In some embodiments, the pathology feature map 130 and / or the radiology feature map 132 may respectively comprise hand-crafted (e.g., pre-defined) features in addition to deep learning features (e.g., features generated through deep learning). For example, the pathology feature map may comprise both hand crafted pathomic features(e.g., texture features, density features, proportion features, entropy features, Haralick features, and / or the like) and deep learning features, while the radiological feature map 132 may comprise both hand crafted radiomic features (e.g., shape features, texture features, size features, histogram based features, gray-level cooccurrence matric (GLCM features), Gabor features, and / or the like). In some embodiments, the hand- crafted features may be provided as an input to the multi-stage hierarchical transformer tool 114. In other embodiments, the hand-crafted features may be provided to the pathology feature map 130 and / or the radiology feature map 132 downstream of the multi-stage hierarchical transformer tool 114.

[0039] A fusion tool 134 is configured to act upon the pathology feature map 130 and the radiology feature map 132 to generate a fused feature map 136. In some embodiments, the fused feature map 136 comprises a fusion tensor that contains information relating to interactions between the different image modalities and different regions within the image modalities. The fused feature map 136 may subsequently be provided to a machine learning model 138 that is configured to operate upon the fused feature map 136 to generate one or more medical predictions 140. In some embodiments, the fusion tool 134 may be configured to form the fused feature map 136 by taking a Kronecker product of the pathology feature map 130 and the radiology feature map 132.

[0040] In some embodiments, the machine learning model 138 may comprise a deep learning neural network (e.g., a multitask fully-connected network). In some such embodiments, the machine learning model 138 may be configured to generate two medical predictions, 210 and 212. For example, in some embodiments the machine learning model 138 may be supervised by a Cox partial negative log-likelihood loss for overall survival prediction, as well as a cross-entropy loss for grade classification (GC). In such embodiments, the machine learning model 138 is configured to generate a risk score 210 for overall survival and a grade class 212 for cancer grade determination. In some embodiments, the risk score 210 for overall survival may classify the patient as a having a low-risk or a high-risk of death. In some embodiments, the grade class 212 may comprise a probability that the patient has a specific grade of cancer (e.g., grade 3).

[0041] Fig.3 illustrates a block diagram of some additional embodiments of a computer vision assessment system 300 configured to incorporate multi-modal and multi-region imaging analysis to determine one or more medical predictions.

[0042] The computer vision assessment system 300 comprises a data set 102 including digitized pathology imaging data 104 and radiological imaging data 106. In some embodiments, the digitized pathology imaging data 104 may comprise a digitized histology WSI. In some embodiments, the radiological imaging data 106 may comprise a computed tomography (CT) image. In some embodiments, the data set 102 may comprise imaging data for i patients. For each patient, the imaging data may include both J 3D fragments of radiological images and K 2D patches of histological images. In some embodiments, the data set 102 may be segmented into one or more training sets, one or more testing sets, and one or more validation sets. The one or more training, testing, and validation sets may be used to train machine learning models (e.g., multi- stage hierarchical transformer tool 114) within the computer vision assessment system 300.

[0043] A pre-processing tool 108 may operate upon the digitized pathology imaging data 104 to generate a plurality of fragments 110. Typically, a WSI may have a relatively large size (e.g., 150,000 x 150,000 pixels at 20 x magnification). Separating the WSI into the plurality of fragments 110 enables the WSI to be operated upon by a downstream transformer. In some embodiments, the plurality of fragments 110 may comprise a plurality of two-dimensional (2D) embeddings. In some embodiments, the WSI may be broken into the plurality of fragments 110 using a pre-trained transformer model. For example, in some embodiments the pre-processing tool 108 may comprise a hierarchical image pyramid transformer (HIPT) architecture configured to break the WSI into non-overlapping 4096 x 4096 image regions, which are subsequently broken into non-overlapping 256 x 256 image patches, which are subsequently broken into non-overlapping 16 x 16 image segments. The regions, patches, and segments are subsequently embedded into 2D feature vectors that are provided to a downstream transformer as the plurality of fragments 110.

[0044] The pre-processing tool 108 may further operate upon the radiological imaging data 106 to generate a plurality of patches 112. In some embodiments, the plurality of patches 112 may comprise a plurality of three-dimensional (3D) non- overlapping patches (e.g., cropped even quadratic sub-volumes) respectively having awidth, a, height, and a depth (e.g., 96 pixels x 96 pixels x 3 pixels). The 3D non- overlapping patches may include raw pixel values from the radiological imaging data 106. In some embodiments, the pre-processing tool 108 is configured to segment the radiological imaging data 106 to identify a primary tumor region and a lymph node (e.g., a largest suspicious lymph node). In some embodiments, the radiological imaging data 106 may be further segmented to identify a peritumoral region surrounding the tumor (e.g., a 15 mm ring surrounding the tumor region). In some embodiments, the plurality of patches 112 may comprise patches (e.g., sub-volumes) taken from a part of the primary tumor region and the lymph node. In some additional embodiments, the plurality of patches 112 may comprise patches taken from a part of the primary tumor region, the lymph node, and the peritumoral region. After pre-processing, imaging data (Xi) associated with each patient includes both J 3D fragments of radiological images (Rij) and K 2D patches of histological images (Hik) (e.g., so that for each patient i, Xi = [Ri,Hi]).

[0045] The computer vision assessment system 300 further comprises a multi-stage hierarchical transformer tool 114 configured to operate upon the plurality of fragments 110 and the plurality of patches 112 to generate a pathology feature map 130 and a radiology feature map 132. The multi-stage hierarchical transformer tool 114 comprises a series of transformer stages 116a-116d configured to perform transformation operations on the plurality of fragments 110 and the plurality of patches 112. In some embodiments, the multi-stage hierarchical transformer tool 114 may comprise B = 4 transformer stages 116a-116d that gradually decrease a resolution of the inputs and model a multi-scale embedding.

[0046] To generate the pathology feature map 130 from the plurality of fragments 110, the multi-stage hierarchical transformer tool 114 uses a pathology Swin Transformer path 118, which contains a plurality of pathology Swin Transformer blocks 120 respectively having two successive modules 302a-302b. A first module 302a uses window based multi-head self-attention (W-MSA), which has a relatively small computational cost that makes it scalable to bigger images. Features are extracted using a multilayer perceptron (MLP) with GELU activation, layer normalization (LN) and residual connections, and W-MSA model interactions within non-overlapping windows of M × M patches. A second module 302b uses a shifted-window multi-head self-attention ^ (SW-MSA) strategy, which displaces the focus by ^^^ ^ × ^^^ pixels. The same featureextraction architecture remains and is followed by a merging layer to group neighboring ^ ^ patches and reduce a size of the feature map to 2C ×^×^. The plurality of pathology Swin Transformer blocks 120 are respectively configured to apply a model, ^^for b ∈ [1,B], to respective ones of the plurality of fragments 110 corresponding to a patient i, so that a fragment output from stage b, (^^^^ ), is equal to ^^^^^^^^^), as shown below for k fragments:Asize c.

[0047] To generate the radiology feature map 132 from the plurality of patches 112, the multi-stage hierarchical transformer tool 114 uses a radiology Swin Transformer path 124, which contains a plurality of stages respectively having a plurality of Swin Transformer blocks 126a-126b having a same structure. The plurality of Swin Transformer blocks 126a-126b are configured to handle a depth dimension. For example, a volume is partitioned into 3D patches and the window regions are shifted by ^ ^ ^ ^^^ × ^^^ × ^^^ voxels, resulting in an intermediate output whose width, height, and and C is doubled. The plurality of Swin Transformer blocks 126a-126bare respectively configured to operate on one of the plurality of patches 112 as they all carry a different type of information the model is to distinguish. For example, for each patch j, a different 3D Swin Transformer model, Ψ^^, is used. The final feature map for the radiological image is ri= [R^^^, ..., R^^^] =, ..., Ψ^^(R^^)] with a feature size c.

[0048] Self-attentionof each of the Swin Transformer blocks, 120 and 126a-126b. The self-attention tools, 122 and 128, perform unimodal inter-region attention. For example, for a patient i, a radiology self- attention tool 128 is configured to generate unimodal intermediate feature maps z = [z1, ..., zJ / K]T, with z being either the set of J fragments R^^^or the set of K patches H^^^ after any Swin Transformer block b. In some embodiments, self-attention mayfor any region m according to: whereto the query Q = zWv (where x and Wv are matrices), key K = zWk, and value V = zWvof the attention mechanism, σ isthe Softmax operation, d is the size of their output dimension and αm is the attention score for region m. For the multi-region set, this simplifies to an attention matrix A with ^ ^∗^ ^ ^^^√!" # ^ $ % ^&'. blocks 206-208 are configured to operate on outputs of thepath 118 and the radiological Swin Transformer path 124 to respectively form a pathology feature map 130 and a radiology feature map 132. In some embodiments, the combination blocks 206-208 are configured to apply an attention-gated mechanism to form the pathology feature map 130 and the radiology feature map 132. For example, to form the radiology feature map (^∗the following attention-gated mechanism may be used: map 132 are vectors of size c,which model a relative expressiveness of each region and limit memory consumption.

[0050] A fusion tool 134 is configured to generate a fused feature map comprising a bimodal representation between the digitized pathology imaging data 104 (e.g., WSI) and the radiological imaging data 106 (e.g., CT). In some embodiments, the fusion tool 134 may form an outer product of the pathology feature map 130 and the radiology feature map 132. In some embodiments, the outer product may comprise a Kronecker product with 1 appended to each of the pathology feature map 130 and the radiology feature map 132. Appending a 1 to each of the pathology feature map 130 and the radiology feature map 132, allows for the resulting outer product to retain information relating to both unimodal features ((^∗, ℎ^∗) and bimodal interactions (^∗⊗ ℎ^∗. By retaining information relating to both thefeatures (e.g., the original CT and pathology embeddings) and the bimodal interactions, an improved result can be achieved.

[0051] The fused feature map is input into a downstream machine learning model 138. In some embodiments, the machine learning model 138 is configured to generate two outputs: a risk score 210 and a grade class 212. In such embodiments, the machine learning model 138 may be a multitask fully-connected network (e.g., a deep neural network) supervised by a Cox partial negative log-likelihood loss for overall survival prediction, as well as a cross-entropy loss for grade classification. The riskscore 210 models a classification of the patient as a high-risk or a low-risk. The grade class 212 is a probability that the patient has a specific grade of cancer (e.g., grade 3). Given the true grade class yi, U the set of uncensored patients and tithe observation time, a total objective L of the system may be a γ-linear combination of both losses: 400for more medical predictions.

[0053] While the disclosed method 400 is illustrated and described herein as a series of acts or events, it will be appreciated that the illustrated ordering of such acts or events are not to be interpreted in a limiting sense. For example, some acts may occur in different orders and / or concurrently with other acts or events apart from those illustrated and / or described herein. In addition, not all illustrated acts may be required to implement one or more aspects or embodiments of the description herein. Further, one or more of the acts depicted herein may be carried out in one or more separate acts and / or phases.

[0054] At act 402, digitized pathology imaging data of a patient is accessed. In some embodiments, the patient may comprise a cancer patient. In some embodiments, the cancer patient may have, have had, or be suspected of having head and neck squamous cell carcinoma, prostate cancer, lung cancer, breast cancer, and / or the like.

[0055] At act 404, a plurality of fragments associated with one or more regions of interest (ROIs) are obtained from the digitized pathology imaging data.

[0056] At act 406, radiological imaging data of the patient is accessed. The radiological imaging data may correspond to a same region of the cancer patient as the pathology imaging data. For example, both the pathology imaging data and the radiological image data may correspond to a neck and head of the cancer patient.

[0057] At act 408, a plurality of patches associated with one or more ROIs are obtained from the radiological imaging data.

[0058] At act 410, multiple stages of hierarchical transformations are applied on the plurality of fragments to generate a pathology feature map and on the plurality of patches to generate a radiology feature map. In some embodiments, the application of hierarchical transformations may be performed according to acts 412-418.

[0059] At act 412, a two-dimensional (2D) Swin transformation operation is performed on embedded vectors comprising different ones of the plurality of fragments to form encoded pathology vectors.

[0060] At act 414, a self-attention operation is performed on the encoded pathology vectors. In some embodiments, acts 412-414 may be repeated during each stage of the multiple stages of hierarchical transformations to form a pathology feature map.

[0061] At act 416, attention pooling is performed on the encoded pathology vectors to form a pathology feature map.

[0062] At act 418, separate three-dimensional (3D) Swin transformation operations are performed on input vectors comprising different ones of the plurality of patches to form encoded radiology vectors.

[0063] At act 420, a self-attention operation is performed on the encoded radiology vectors. In some embodiments, acts 418-420 may be repeated during each stage of the multiple stages of hierarchical transformations.

[0064] At act 422, attention pooling is performed on the encoded radiology vectors to form a radiology feature map. In some embodiments, the pathology feature map and the radiology feature map may have identical sizes.

[0065] At act 424, the radiology feature map and the pathology feature map are combined to generate a fused feature map. In some embodiments, the radiology feature map and the pathology feature map may be combined by taking an outer product of the radiology feature map and the pathology feature map.

[0066] At act 426, the fused feature map is operated upon with a machine learning model (e.g., a deep learning model) to generate one or more medical predictions for the cancer patient. In some embodiments, the machine learning model may comprise a fully connected network.

[0067] Fig.5 illustrates exemplary Kaplan-Meier (KM) graphs 500-510 showing a predicted overall survival for different feature types.

[0068] As shown in Fig.5, KM graphs 500-510 show KM curves corresponding to an overall survival predicted using different feature sets. The first graph 500 corresponds to a computer vision assessment system that operates upon features extracted from a tumor region within a radiological image. The second graph 502 corresponds to a computer vision assessment system that operates upon features extracted from a lymph node region within a radiological image. The third graph 504 corresponds to acomputer vision assessment system that operates upon features extracted from tumor and lymph node regions within a radiological image. The fourth graph 506 corresponds to a computer vision assessment system that operates upon features extracted from a digitized pathology image. The fifth graph 508 corresponds to a computer vision assessment system that operates upon features extracted from digitized pathology and radiological images fused using vector concatenation. The sixth graph 510 corresponds to a disclosed computer vision assessment system.

[0069] For each of the KM graphs 500-510, the x-axis represents a time in months and y-axis represents the estimated survival probability. The KM graphs 500-510 respectively have a high-risk stratification group (red line) and a low-risk stratification group (blue line). In some embodiments, stratification between the high-risk stratification group and the low-risk stratification group may be based on a median of risk scores generated during training of an associated system.

[0070] As can be seen in the KM graphs 500-510, the sixth graph 510 (associated with the disclosed computer vision assessment system) achieves the highest hazard ratio (H=8.95), thereby indicating that the disclosed computer vision assessment system outperforms the other systems. The high hazard ratio also indicates a good stratification between the high-risk stratification group and the low-risk stratification group and a good prognostic ability of the disclosed computer vision assessment system.

[0071] Fig.6 illustrates a table 600 showing exemplary performance parameters for different combinations of features maps and machine learning systems.

[0072] Table 600 illustrates different feature maps 602 and different machine learning systems 604. The different feature maps and machine learning systems show 24 different combinations. For each combination, an area under curve (AIC) is shown for a grade classification. As can be seen by table 600, when comparing the feature extraction strategies, the three deep learning-based approaches CNN (convolutional neural network), VIT (Vision Transformer), and ST (Swin Transformer) always outperformed handcrafted (HF) machine learning, with ST being the best. A highest area under curve (AUC) is obtained using the disclosed computer vision system (e.g., the computer vision system that comprises Swin Transformers and feature maps generated using attention and Kronecker fusion. Specifically, the disclosed computervision system outperforms the CNN and VIT using Kronecker fusion (AK) by 3.8% for overall survival and grade classification (e.g., 0.81 vs 0.78).

[0073] It will be appreciated that the disclosed methods and / or block diagrams may be implemented as computer-executable instructions, in some embodiments. Thus, in one example, a computer-readable storage device (e.g., a non-transitory computer- readable medium) may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform the disclosed methods and / or block diagrams. While executable instructions associated with the disclosed methods and / or block diagrams are described as being stored on a computer- readable storage device, it is to be appreciated that executable instructions associated with other example disclosed methods and / or block diagrams described or claimed herein may also be stored on a computer-readable storage device.

[0074] Furthermore, although the disclosed method and apparatus have been described as performing a multi-modal and multi-region analysis of cancer for radiological and histological imaging data, the disclosed method and apparatus are not limited to such application and / or multi-modal data. Rather, the disclosed method and apparatus may also be applied to other applications (e.g., allograft rejection, heart transplant rejection, etc.) and may use other and / or additional data modalities (e.g., genomic, clinical, and / or the like).

[0075] For example, Fig.7 illustrates a block diagram of some additional embodiments of a computer vision assessment system 700 configured to incorporate multi-modal and multi-region imaging analysis to determine one or more medical predictions.

[0076] The computer vision assessment system 700 comprises electronic memory 201. The electronic memory 201 is configured to store multi-modal patient data 103 from a plurality of different modalities. For example, the multi-modal patient data 103 may comprise two or more of digitized pathology imaging data 104, radiological imaging data 106, genomic data 704, clinical data 706 (e.g., clinical records), and / or the like.

[0077] A pre-processing tool 108 may be configured to operate upon the multi-modal patient data 103 prior to providing the multi-modal patient data 103 to a multi-stage hierarchical transformer tool 114. The pre-processing tool 108 may generate pre- processed patient data 710 that has been separated and / or embedded into different data structures and / or formats.

[0078] The pre-processed patient data 710 is provided to a multi-stage hierarchical transformer tool 114 that is configured to perform multi-scale and multi-region processing upon the pre-processed patient data 710 to form a plurality of feature maps 129. For example, the multi-stage hierarchical transformer tool 114 may operate upon the digitized pathology imaging data 104 to form a pathology feature map 130, upon the radiological imaging data 106 to form a radiology feature map 132, upon the genomic data 704 to form a genomic feature map 724, and / or upon the clinical data 706 to form a clinical feature map 726. In some embodiments, one or more of the plurality of feature maps 129 may comprise hand-crafted (e.g., pre-defined) features in addition to deep learning features (e.g., features generated through deep learning).

[0079] The multi-stage hierarchical transformer tool 114 comprises a plurality of transformer paths 118, 124, 712, and 718 respectively configured to perform multi-stage transformations on a specific modality of data. For example, the multi-stage hierarchical transformer tool 114 may comprise a pathological Swin Transformer path 118 configured to operate upon the digitized pathology imaging data 104 to form the pathology feature map 130, a radiological Swin Transformer path 124 configured to operate upon the radiological imaging data 106 to form the radiology feature map 132, a genomic transformer path 712 (e.g., including a transformer 714 and a self-attention tool 716) configured to operate upon the genomic data 704 to form a genomic feature map 724, and a clinical transformer path 718 (e.g., including a transformer 720 and a self- attention tool 722) configured to operate upon the clinical data 706 to form a clinical feature map 726. In some embodiments, one or more of the transformer paths, 118 and 124, may comprise Swin Transformers, 120 and 126, while one or more of the transformer paths, 712 and 718, may comprise another type of transformer, 714 and 720. In some embodiments, the multi-stage hierarchical transformer tool 114 is configured to perform an attention based aggregation of the plurality of fragments 110 during formation of the pathology feature map 130 and to perform an attention based aggregation of the plurality of patches 112 during formation of the radiology feature map 132.

[0080] The plurality of feature maps 129 are provided to a fusion tool 134 that is configured to combine the plurality of feature maps 129 to form a fused feature map 136. The fused feature map 136 may comprise information relating to data havingdifferent modalities and interactions between different regions of the data having different modalities.

[0081] The fused feature map 136 may be subsequently provided to a machine learning model 138. The machine learning model 138 is configured to operate upon the fused feature map 136 to generate one or more medical predictions 140. Operating the machine learning model 138 on the fused feature map 136 allows for the one or more medical predictions 140 to be generated based upon information relating to different regions and different modalities. The one or more medical predictions 140 are therefore able to achieve improved results (e.g., AUC) over machine learning models that utilize single regions or modalities of data.

[0082] Fig.8 illustrates a block diagram of some additional embodiments of a disclosed computer vision assessment system 800.

[0083] The computer vision assessment system 800 comprises an assessment apparatus 802. The assessment apparatus 802 is coupled to a slide digitization tool 804 that is configured to obtain digitized pathology imaging data 104 from a cancer patient 806. In some embodiments, the slide digitization tool 804 may comprise a CMOS image sensor, a CCD camera, and / or the like. In some embodiments, the digitized pathology imaging data 104 may be generated from a pathological tissue sample taken from the cancer patient 806. In some such embodiments, a tissue sample collection tool 808 (e.g., a cannular, forceps, needle, punch, scalpel, and / or the like) is used to excise tissue from the cancer patient 806 to obtain a tissue block. The tissue block is fixed, embedded in paraffin, and sliced into thin slices (e.g., using a microtome), which are subsequently placed on one or more transparent slides (e.g., one or more glass slides). The tissue on the one or more transparent slides is then stained to generate one or more biopsy slides 810. The slide digitization tool 804 is used to convert the one or more biopsy slides 810 to one or more whole slide images (WSIs).

[0084] The assessment apparatus 802 is further coupled to a radiological imaging tool 812 that is configured to obtain one or more radiological images from the cancer patient 806. In various embodiments, the radiological imaging tool 812 may comprise a CT (computed tomography) scanner, an MRI (magnetic resonance imaging) scanner, an MRE (Magnetic Resonance Enterography) scanner, an x-ray machine, an ultrasound machine, and / or the like.

[0085] The assessment apparatus 802 comprises one or more processors 814 and a memory 816. The one or more processors 814, in various embodiments, comprise circuitry such as, but not limited to, one or more single-core or multi-core processors. The one or more processors 814 can include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, etc.). The one or more processors 814 can be coupled with and / or can comprise memory (e.g., memory 816) or storage and can be configured to execute instructions stored in the memory 816 or storage to enable various apparatus, applications, or operating systems to perform operations and / or methods discussed herein.

[0086] The memory 816 can be configured to store data having different modalities. For example, the memory 816 may store the digitized pathology imaging data 104 and the radiology imaging data 106 from the cancer patient 806. The digitized pathology imaging data 104 and the radiology imaging data 106 may comprise digitized images including a plurality of pixels, each pixel having an associated intensity. In some additional embodiments, the digitized pathology imaging data 104 and the radiology imaging data 106 may comprise a plurality of digitized images stored in the memory 816 as one or more training sets, one or more test sets, and / or one or more validation sets.

[0087] The assessment apparatus 802 also comprises an input / output (I / O) interface 818 (e.g., associated with one or more I / O devices), a display 820, one or more circuits 824, and an interface 822 that connects the one or more processors 814, the memory 816, the I / O interface 818, the display 820, and the one or more circuits 824. The I / O interface 818 can be configured to transfer data between the memory 816, the one or more processors 814, the one or more circuits 824, and external devices (e.g., slide digitization tool 804, the radiological imaging tool 812, etc.).

[0088] The one or more circuits 824 can comprise a pre-processing circuit 826. The pre-processing circuit 826 is configured to separate the digitized pathology imaging data 104 into a plurality of fragments 110 and to separate the radiology imaging data 106 into a plurality of patches 112. In some embodiments, the pre-processing circuit 826 may comprise one or more deep learning segmentation circuits implemented in hardware components and / or in software components.

[0089] In some embodiments, the one or more circuits 824 may further comprise a first Swin Transformer circuit 828 and a second Swin Transformer circuit 830. The firstSwin Transformer circuit 828 is configured to perform a series of Swin Transformations on vectors comprising the plurality of fragments 110. The second Swin Transformer circuit 830 is configured to perform a series of Swin Transformations on vectors comprising the plurality of patches 112. After each of the Swin transformations, an attention circuit 832 is configured to perform self-attention on vectors output from the first Swin Transformer circuit 828 and the second Swin Transformer circuit 830.

[0090] In some embodiments, the one or more circuits 824 may further comprise a fusion circuit 834. The fusion circuit 834 is configured to combine a pathology feature map 130 generated from the plurality of fragments 110 and a radiology feature map 132 generated from the plurality of patches 112 to form a fused feature map 136. The fused feature map 136 comprises information relating to different regions and different image modalities of the digitized pathology imaging data 104 and radiological imaging data 106.

[0091] In some embodiments, the one or more circuits 824 may further comprise a machine learning circuit 836. The machine learning circuit 836 is configured to operate upon the fused feature map 136 to generate one or more medical predictions 140 relating to the cancer patient 806.

[0092] Therefore, the present disclosure relates a computer vision assessment system, and an associated method, that incorporates both multi-modal imaging analysis and multi-region imaging analysis to determine one or more medical predictions for a patient.

[0093] In some embodiments, the disclosure relates to a method that includes applying one or more multi-stage hierarchical transformations on data corresponding to one or more first regions of interest (ROIs) within a digitized pathology image of a patient to generate a pathology feature map and on data corresponding to one or more second ROIs from a radiological image of the patient to generate a radiology feature map, respective stages of the one or more multi-stage hierarchical transformations including a self-attention operation; combining the radiology feature map and the pathology feature map to generate a fused feature map; and operating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient. In some embodiments, the method may further include obtaining a plurality of fragments associated with the one or more first ROIs from the digitized pathology image of the patient; obtaining a plurality of patches associated with the oneor more second ROIs from the radiological image of the patient; and applying the one or more multi-stage hierarchical transformations to the plurality of fragments to generate the pathology feature map and to the plurality of patches to generate the radiological feature map. In some embodiments, the method may further include performing a first transformation operation on the plurality of fragments, the plurality of fragments including embedded vectors; performing a first self-attention operation on first encoded vectors output from the first transformation operation; performing a second transformation operation on vectors output from the first self-attention operation; and performing a second self-attention operation on second encoded vectors output from the second transformation operation. In some embodiments, the one or more multi- stage hierarchical transformations are performed using a plurality of hierarchical transformation operations having different scales; and the self-attention operation is performed after each of the plurality of hierarchical transformation operations. In some embodiments, the one or more multi-stage hierarchical transformations include a pathology Swin Transformer path having a plurality of pathology processing stages configured to perform a first plurality of Swin transformations upon the data corresponding to the one or more first ROIs; and the one or more multi-stage hierarchical transformations further include a radiology Swin Transformer path having a plurality of radiology processing stages configured to perform a second plurality of Swin transformations upon the data corresponding to the one or more second ROIs. In some embodiments, the pathology Swin Transformer path includes a first Swin Transformer configured to operate upon the data corresponding to the one or more first ROIs within a first pathology processing stage of the plurality of pathology processing stages; and the radiology Swin Transformer path includes a plurality of second Swin Transformers configured to operate upon the data corresponding to respective ones of the one or more second ROIs within a first radiology processing stage of the plurality of radiology processing stages. In some embodiments, the method further includes performing a first combining operation on outputs of the pathology Swin Transformer path to form the pathology feature map; and performing a second combining operation on outputs of the radiology Swin Transformer path to form the radiology feature map. In some embodiments, one or more of the pathology feature map and the radiology feature map include hand-crafted features and deep learning features. In some embodiments, the one or more multi-stage hierarchical transformations are performed by a plurality ofSwin Transformers. In some embodiments, the data corresponding to the one or more first ROIs is operated upon by a 2D Swin Transformer module; and the data corresponding to the one or more second ROIs is operated upon by separate 3D Swin Transformer modules. In some embodiments, the radiology feature map and the pathology feature map are combined by taking a Kronecker product of the radiology feature map with 1 appended thereto and the pathology feature map with 1 appended thereto. In some embodiments, the data corresponding to the one or more second ROIs includes a first patch containing a part of a tumor region and a second patch containing a part of a lymph node.

[0094] In other embodiments, the disclosure relates to a non-transitory computer- readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, including obtaining a plurality of fragments associated with one or more regions of interest (ROIs) from a digitized pathology image of a patient; obtaining a plurality of patches associated with one or more ROIs from a radiological image of the patient; operating upon the plurality of fragments with a first plurality of Swin transformation operations to form a pathology feature map, self-attention being performed between outputs of respective ones of the first plurality of Swin transformation operations; operating upon the plurality of patches with a second plurality of Swin transformation operations to form a radiology feature map, wherein self-attention is performed between outputs of respective ones of the second plurality of Swin transformation operations; combining the radiology feature map and the pathology feature map to generate a fused feature map; and operating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient. In some embodiments, the radiology feature map and the pathology feature map are combined by operating upon the radiology feature map and the pathology feature map with a Kronecker product. In some embodiments, the first plurality of Swin transformation operations are configured to operate at different scales; and the second plurality of Swin transformation operations are configured to operate at the different scales. In some embodiments, the plurality of fragments are operated upon by a 2D Swin Transformer module; and the plurality of patches are respectively operated upon by separate 3D Swin Transformer modules.

[0095] In yet other embodiments, the disclosure relates to an apparatus, including electronic memory configured to store an imaging data set having digitized pathology imaging data and radiological imaging data of a patient; a pre-processing tool configuredto break the digitized pathology imaging data into a plurality of fragments and further configured to break the radiological imaging data into a plurality of patches; a multi-stage Swin Transformer tool configured to perform multi-scale and multi-region processing upon the plurality of fragments to form a pathology feature map and upon the plurality of patches to build a radiology feature map; a fusion tool configured to act upon the pathology feature map and the radiology feature map to generate a fused feature map; and a machine learning model configured to operate upon the fused feature map to generate one or more medical predictions. In some embodiments, the multi-stage Swin Transformer tool includes a pathology Swin Transformer path having a plurality of pathology processing stages respectively configured to perform a first Swin transformation on the plurality of fragments; and the multi-stage Swin Transformer tool includes a radiology Swin Transformer path having a plurality of radiology processing stages respectively configured to perform second Swin transformations on respective ones of the plurality of patches. In some embodiments, the pathology Swin Transformer path includes a series of 2D Swin Transformer blocks and the radiology Swin Transformer path includes a series of 3D Swin Transformer blocks respectively having a plurality of radiology Swin Transformers. In some embodiments, the multi-stage Swin Transformer tool is configured to perform a series of Swin transformations with self-attention applied after each of the series of Swin transformations.

[0096] In yet other embodiments, the disclosure relates to a method that includes applying a first multi-stage hierarchical transformation on data corresponding to multiple regions of interest (ROIs) from a first modality of patient data of a patient to generate a first feature map, respective stages of the first multi-stage hierarchical transformation having a self-attention operation; applying a second multi-stage hierarchical transformation on data corresponding to multiple ROIs from a second modality of patient data to generate a second feature map, respective stages of the second multi-stage hierarchical transformation having a self-attention operation; combining the first feature map and the second feature map to generate a fused feature map; and operating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient.

[0097] Examples herein can include subject matter such as an apparatus, including a digital whole slide scanner, a CT system, an MRI system, a personalized medicine system, a CADx system, a processor, a system, circuitry, a method, means forperforming acts, steps, or blocks of the method, at least one machine-readable medium including executable instructions that, when performed by a machine (e.g., a processor with memory, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like) cause the machine to perform acts of the method or of an apparatus or system, according to embodiments and examples described.

[0098] References to “one embodiment”, “an embodiment”, “one example”, and “an example” indicate that the embodiment(s) or example(s) so described may include a particular feature, structure, characteristic, property, element, or limitation, but that not every embodiment or example necessarily includes that particular feature, structure, characteristic, property, element or limitation. Furthermore, repeated use of the phrase “in one embodiment” does not necessarily refer to the same embodiment, though it may.

[0099] “Computer-readable storage device”, as used herein, refers to a device that stores instructions or data. “Computer-readable storage device” does not refer to propagated signals. A computer-readable storage device may take forms, including, but not limited to, non-volatile media, and volatile media. Non-volatile media may include, for example, optical disks, magnetic disks, tapes, and other media. Volatile media may include, for example, semiconductor memories, dynamic memory, and other media. Common forms of a computer-readable storage device may include, but are not limited to, a floppy disk, a flexible disk, a hard disk, a magnetic tape, other magnetic medium, an application specific integrated circuit (ASIC), a compact disk (CD), other optical medium, a random access memory (RAM), a read only memory (ROM), a memory chip or card, a memory stick, and other media from which a computer, a processor or other electronic device can read. [000100] “Circuit”, as used herein, includes but is not limited to hardware, firmware, software in execution on a machine, or combinations of each to perform a function(s) or an action(s), or to cause a function or action from another logic, method, or system. A circuit may include a software controlled microprocessor, a discrete logic (e.g., ASIC), an analog circuit, a digital circuit, a programmed logic device, a memory device containing instructions, and other physical devices. A circuit may include one or more gates, combinations of gates, or other circuit components. Where multiple logical circuits are described, it may be possible to incorporate the multiple logical circuits intoone physical circuit. Similarly, where a single logical circuit is described, it may be possible to distribute that single logical circuit between multiple physical circuits. [000101] To the extent that the term “includes” or “including” is employed in the detailed description or the claims, it is intended to be inclusive in a manner similar to the term “comprising” as that term is interpreted when employed as a transitional word in a claim. [000102] Throughout this specification and the claims that follow, unless the context requires otherwise, the words 'comprise' and 'include' and variations such as 'comprising' and 'including' will be understood to be terms of inclusion and not exclusion. For example, when such terms are used to refer to a stated integer or group of integers, such terms do not imply the exclusion of any other integer or group of integers. [000103] To the extent that the term “or” is employed in the detailed description or claims (e.g., A or B) it is intended to mean “A or B or both”. When the applicants intend to indicate “only A or B but not both” then the term “only A or B but not both” will be employed. Thus, use of the term “or” herein is the inclusive, and not the exclusive use. See, Bryan A. Garner, A Dictionary of Modern Legal Usage 624 (2d. Ed.1995). [000104] While example systems, methods, and other embodiments have been illustrated by describing examples, and while the examples have been described in considerable detail, it is not the intention of the applicants to restrict or in any way limit the scope of the appended claims to such detail. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the systems, methods, and other embodiments described herein. Therefore, the invention is not limited to the specific details, the representative apparatus, and illustrative examples shown and described. Thus, this application is intended to embrace alterations, modifications, and variations that fall within the scope of the appended claims.

Claims

What is claimed is:

1. A method, comprising: applying one or more multi-stage hierarchical transformations on data corresponding to one or more first regions of interest (ROIs) within a digitized pathology image of a patient to generate a pathology feature map and on data corresponding to one or more second ROIs from a radiological image of the patient to generate a radiology feature map, wherein respective stages of the one or more multi-stage hierarchical transformations comprise a self-attention operation; combining the radiology feature map and the pathology feature map to generate a fused feature map; and operating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient.

2. The method of claim 1, further comprising: obtaining a plurality of fragments associated with the one or more first ROIs from the digitized pathology image of the patient; obtaining a plurality of patches associated with the one or more second ROIs from the radiological image of the patient; and applying the one or more multi-stage hierarchical transformations to the plurality of fragments to generate the pathology feature map and to the plurality of patches to generate the radiological feature map.

3. The method of claim 2, further comprising: performing a first transformation operation on the plurality of fragments, wherein the plurality of fragments comprise embedded vectors; performing a first self-attention operation on first encoded vectors output from the first transformation operation; performing a second transformation operation on vectors output from the first self- attention operation; and performing a second self-attention operation on second encoded vectors output from the second transformation operation.

4. The method of claim 1, wherein the one or more multi-stage hierarchical transformations are performed using a plurality of hierarchical transformation operations having different scales; and wherein the self-attention operation is performed after each of the plurality of hierarchical transformation operations.

5. The method of claim 1, wherein the one or more multi-stage hierarchical transformations comprise a pathology Swin Transformer path comprising a plurality of pathology processing stages configured to perform a first plurality of Swin transformations upon the data corresponding to the one or more first ROIs; and wherein the one or more multi-stage hierarchical transformations further comprise a radiology Swin Transformer path comprising a plurality of radiology processing stages configured to perform a second plurality of Swin transformations upon the data corresponding to the one or more second ROIs.

6. The method of claim 5, wherein the pathology Swin Transformer path comprises a first Swin Transformer configured to operate upon the data corresponding to the one or more first ROIs within a first pathology processing stage of the plurality of pathology processing stages; and wherein the radiology Swin Transformer path comprises a plurality of second Swin Transformers configured to operate upon the data corresponding to respective ones of the one or more second ROIs within a first radiology processing stage of the plurality of radiology processing stages.

7. The method of claim 5, further comprising: performing a first combining operation on outputs of the pathology Swin Transformer path to form the pathology feature map; and performing a second combining operation on outputs of the radiology Swin Transformer path to form the radiology feature map.

8. The method of claim 1, wherein one or more of the pathology feature map and the radiology feature map include hand-crafted features and deep learning features.

9. The method of claim 1, wherein the one or more multi-stage hierarchical transformations are performed by a plurality of Swin Transformers.

10. The method of claim 1, wherein the data corresponding to the one or more first ROIs is operated upon by a 2D Swin Transformer module; and wherein the data corresponding to the one or more second ROIs is operated upon by separate 3D Swin Transformer modules.

11. The method of claim 1, wherein the radiology feature map and the pathology feature map are combined by taking a Kronecker product of the radiology feature map with 1 appended thereto and the pathology feature map with 1 appended thereto.

12. The method of claim 1, wherein the data corresponding to the one or more second ROIs includes a first patch containing a part of a tumor region and a second patch containing a part of a lymph node.

13. A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, comprising: obtaining a plurality of fragments associated with one or more regions of interest (ROIs) from a digitized pathology image of a patient; obtaining a plurality of patches associated with one or more ROIs from a radiological image of the patient; operating upon the plurality of fragments with a first plurality of Swin transformation operations to form a pathology feature map, wherein self-attention is performed between outputs of respective ones of the first plurality of Swin transformation operations; operating upon the plurality of patches with a second plurality of Swin transformation operations to form a radiology feature map, wherein self-attention is performed between outputs of respective ones of the second plurality of Swin transformation operations; combining the radiology feature map and the pathology feature map to generate a fused feature map; andoperating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient.

14. The non-transitory computer-readable medium of claim 13, wherein the radiology feature map and the pathology feature map are combined by operating upon the radiology feature map and the pathology feature map with a Kronecker product.

15. The non-transitory computer-readable medium of claim 13, wherein the first plurality of Swin transformation operations are configured to operate at different scales; and wherein the second plurality of Swin transformation operations are configured to operate at the different scales.

16. The non-transitory computer-readable medium of claim 13, wherein the plurality of fragments are operated upon by a 2D Swin Transformer module; and wherein the plurality of patches are respectively operated upon by separate 3D Swin Transformer modules.

17. An apparatus, comprising: electronic memory configured to store an imaging data set comprising digitized pathology imaging data and radiological imaging data of a patient; a pre-processing tool configured to break the digitized pathology imaging data into a plurality of fragments and further configured to break the radiological imaging data into a plurality of patches; a multi-stage Swin Transformer tool configured to perform multi-scale and multi- region processing upon the plurality of fragments to form a pathology feature map and upon the plurality of patches to build a radiology feature map; a fusion tool configured to act upon the pathology feature map and the radiology feature map to generate a fused feature map; and a machine learning model configured to operate upon the fused feature map to generate one or more medical predictions.

18. The apparatus of claim 17, wherein the multi-stage Swin Transformer tool comprises a pathology Swin Transformer path including a plurality of pathology processing stages respectively configured to perform a first Swin transformation on the plurality of fragments; and wherein the multi-stage Swin Transformer tool comprises a radiology Swin Transformer path including a plurality of radiology processing stages respectively configured to perform second Swin transformations on respective ones of the plurality of patches.

19. The apparatus of claim 18, wherein the pathology Swin Transformer path comprises a series of 2D Swin Transformer blocks and the radiology Swin Transformer path comprises a series of 3D Swin Transformer blocks respectively comprising a plurality of radiology Swin Transformers.

20. The apparatus of claim 17, wherein the multi-stage Swin Transformer tool is configured to perform a series of Swin transformations with self-attention applied after each of the series of Swin transformations.

21. A method, comprising: applying a first multi-stage hierarchical transformation on data corresponding to one or more regions of interest (ROIs) from a first modality of patient data of a patient to generate a first feature map, wherein respective stages of the first multi-stage hierarchical transformation comprise a self-attention operation; applying a second multi-stage hierarchical transformation on data corresponding to one or more ROIs from a second modality of patient data to generate a second feature map, wherein respective stages of the second multi-stage hierarchical transformation comprise a self-attention operation; combining the first feature map and the second feature map to generate a fused feature map; and operating upon the fused feature map with a machine learning model to generate one or more medical predictions for the patient.

Images