Attention-based multimodal fusion for patient survival prediction

Abstract

A method for predicting a patient's overall survival probability using a machine learning-based prediction system includes: the prediction system receiving first modality data and second modality data corresponding to the patient; the prediction system generating first intermediate feature vectors and second intermediate feature vectors based on the first and second modality data; the prediction system determining first attention scores and second attention scores based on the first and second intermediate feature vectors; the prediction system generating aggregate feature vectors based on the first and second intermediate feature vectors and the first and second attention scores; and the prediction system generating a survival probability prediction corresponding to the patient based on the aggregate feature vectors.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority and benefit from U.S. Provisional Application No. 63 / 533,572, “ATTENTION-BASED MULTIMODAL-FUSION FOR NON-SMALL CELL LUNG CANCER (NSCLC) PATIENT SURVIVAL PREDICTION,” filed on 18 August 2023, and U.S. Provisional Application No. 63 / 555,256, “ATTENTION-BASED MULTIMODAL-FUSION FOR PATIENT SURVIVAL PREDICTION,” filed on 19 February 2024.

[0002] One or more embodiments of the present disclosure relate to systems and methods for predicting patient outcomes. [Background technology]

[0003] Various forms of cancer are among the leading causes of death worldwide. Lung cancer, in particular, is one of the most common malignancies, accounting for approximately 25% of all cancer-related deaths. About 84% of lung cancers are non-small cell lung cancer (NSCLC), a group of lung cancers that behave similarly. Immunotherapy with checkpoint inhibitors such as anti-PD1 and anti-PD-L1 drugs has shown promising clinical outcomes in patients with locally advanced (ad) or metastatic (m) NSCLC. However, the biomarkers currently used to select patients who may benefit from targeted or immunotherapy are inaccurate and have high potential for improvement.

[0004] Predicting cancer prognosis and survival outcomes is crucial for predicting treatment response and stratifying patients into different treatment groups. Integrating various data formats into survival prediction models can enhance their predictive power, benefiting both clinical research and practice.

[0005] The information disclosed in this background section is intended solely to enhance the understanding of the background, and therefore, the information described in this background section does not necessarily constitute prior art. [Overview of the Initiative]

[0006] Embodiments of this disclosure relate to a multimodal prediction system that utilizes an attention-based deep learning framework for predicting overall survival of patients (e.g., NSCLC patients) in combination with histopathology and genomics data. In some embodiments, the attention-based deep learning framework utilizes a cross-modality attention-based multimodal fusion (CM-MMF) approach, which integrates image and RNA sequence modalities to achieve superior patient survival prediction. Here, attention scores derived from the fusion layer can highlight the importance of each modality in the fusion for clinical diagnosis.

[0007] According to some embodiments of the present disclosure, a method for predicting a patient's overall viability by a machine learning-based prediction system includes: the prediction system receiving first modality data and second modality data corresponding to a patient; the prediction system generating first intermediate feature vectors and second intermediate feature vectors based on the first and second modality data; the prediction system determining first attention scores and second attention scores based on the first and second intermediate feature vectors; the prediction system generating aggregate feature vectors based on the first and second intermediate feature vectors and the first and second attention scores; and the prediction system generating a patient's viability prediction based on the aggregate feature vectors.

[0008] According to some embodiments, the first modality data includes histological hematoxylin and eosin (H&E) image data, and the second modality data includes gene sequencing data.

[0009] According to some embodiments, H&E image data includes digitized images of patient tissue samples stained with hematoxylin and eosin dyes, and gene sequencing data includes mRNA gene expression extracted from the patient's tumor tissue.

[0010] According to some embodiments, receiving first modality data and second modality data includes receiving first modality data by a first model of the prediction system and receiving second modality data by a second model of the prediction system.

[0011] According to some embodiments, generating a first intermediate feature vector and a second intermediate feature vector includes generating a first intermediate feature vector based on first modality data using a first model, and generating a second intermediate feature vector based on second modality data using a second model.

[0012] According to some embodiments, the first model includes an attention-based multi-instance learning (AMIL) model, and the second model includes a feedforward neural network (FNN) model.

[0013] According to some embodiments, determining a first attention score and a second attention score includes combining the first and second intermediate feature vectors to generate a combined feature, processing the combined feature by one or more convolutional and activation layers to generate first and second attention values ​​corresponding to the first and second intermediate feature vectors, and converting the first and second attention values ​​into first and second attention scores by a softmax function layer.

[0014] According to some embodiments, the concatenation of a first intermediate feature vector and a second intermediate feature vector includes stacking the first and second intermediate feature vectors onto a two-dimensional array which is the concatenated feature, and one or more convolutional layers are configured to perform a two-dimensional convolution on the two-dimensional array.

[0015] According to some embodiments, the activation layer includes a hyperbolic tangent (Tanh) function layer or a normalized linear unit (ReLU) function layer, where the sum of the first and second attention scores is equal to 1.

[0016] According to some embodiments, the method further includes normalizing the values ​​of the first and second intermediate feature vectors before combining them.

[0017] According to some embodiments, determining a first attention score and a second attention score includes processing a first intermediate feature vector with one or more first convolutional layers and a first activation layer to generate a first attention value corresponding to the first intermediate feature vector, processing a second intermediate feature vector with one or more second convolutional layers and a second activation layer to generate a second attention value corresponding to the second intermediate feature vector, and converting the first and second attention values ​​into first and second attention scores using a softmax function layer.

[0018] According to some embodiments, each of the first and second activation layers includes a hyperbolic tangent (Tanh) function layer or a normalized linear unit (ReLU) function layer, and the sum of the attention scores of the first and second layers is equal to 1.

[0019] According to some embodiments, one or more first convolutional layers include kernel values ​​that are not the same as those of one or more second convolutional layers.

[0020] According to some embodiments, generating an aggregated feature vector comprises scaling a first intermediate feature vector based on a first attention score to generate a first scaled intermediate feature vector, scaling a second intermediate feature vector based on a second attention score to generate a second scaled intermediate feature vector, and aggregating the first and second scaled intermediate feature vectors to generate the aggregated feature vector.

[0021] According to some embodiments, the aggregating of the first and second scaled intermediate feature vectors comprises concatenating the first and second scaled intermediate feature vectors to generate the aggregated feature vector, wherein the aggregated feature vector has a length equal to the sum of the lengths of the first and second scaled intermediate feature vectors.

[0022] According to some embodiments, generating a survival likelihood prediction comprises generating, by a classifier of a prediction system, the survival likelihood prediction corresponding to a patient based on the aggregated feature vector, wherein the survival likelihood prediction corresponds to the overall survival likelihood of the patient.

[0023] According to some embodiments, the survival likelihood prediction comprises a range of values from a plurality of consecutive value ranges.

[0024] According to some embodiments, the method further comprises transmitting the survival likelihood prediction to a display device for presentation to a user.

[0025] According to some embodiments, in a prediction system for predicting the overall survival possibility of a patient based on machine learning, the prediction system comprises: a first model configured to receive first modality data corresponding to the patient and generate a first intermediate feature vector based on the first modality data; a second model configured to receive second modality data corresponding to the patient and generate a second intermediate feature vector based on the second modality data; and an attention-based multimodal fusion circuit configured to determine a first attention score and a second attention score based on the first and second intermediate feature vectors, and generate an aggregated feature vector based on the first and second intermediate feature vectors and the first and second attention scores.

[0026] According to some embodiments, the prediction system further comprises a classifier configured to receive the aggregated feature vector and generate a survival possibility prediction corresponding to the patient based on the aggregated feature vector. [Brief Description of the Drawings]

[0027] Non-limiting and non-exhaustive embodiments according to the present disclosure are described with reference to the following drawings, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.

[0028] [Figure 1] FIG. 2 is a block diagram illustrating a prediction system according to some embodiments of the present disclosure.

[0029] [Figure 2A] FIG. 3 is a block diagram illustrating a first model of a prediction system according to some embodiments of the present disclosure.

[0030] [Figure 2B] FIG. 4 is a block diagram illustrating a second model of a prediction system according to some embodiments of the present disclosure.

[0031] [Figure 3A]The following are block diagrams of attention-based multimodal fusion circuits employing a non-shared attention-based multimodal fusion approach according to some embodiments of the present disclosure.

[0032] [Figure 3B] The following are block diagrams of attention-based multimodal fusion circuits employing a shared attention-based multimodal fusion approach, according to some other embodiments of the present disclosure.

[0033] [Figure 4A] Tables are provided comparing the performance of prediction systems across various modalities under various types of loss supervision, according to several embodiments of this disclosure, with that of various systems of the relevant technology.

[0034] [Figure 4B] A table comparing the performance of prediction systems using shared and non-shared kernel layers and different activation functions in several embodiments of this disclosure is provided.

[0035] [Figure 5] This flowchart illustrates a process by which a predictive system predicts the overall survival rate of a patient, according to some embodiments of the present disclosure. [Modes for carrying out the invention]

[0036] The following describes some exemplary embodiments with reference to the accompanying drawings, where similar reference numerals refer to similar elements throughout. However, the present invention may be embodied in various different forms and should not be construed as being limited only to the embodiments illustrated herein. Rather, these embodiments are provided as examples so that this disclosure is thorough and complete and adequately conveys the aspects and features of the invention to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary for those skilled in the art to fully understand the aspects and features of the invention may not be described. Unless otherwise noted, similar reference numerals throughout the accompanying drawings and specification refer to similar elements, and therefore their descriptions are not repeated. In the drawings, the relative sizes of elements, layers, and areas may be exaggerated for clarity.

[0037] Some embodiments of this disclosure enable the prediction of survival outcomes in non-small cell lung cancer (NSCLC) patients using a combination of histopathology and genomics with attention-based deep learning approaches. In particular, some embodiments utilize a cross-modality attention-based multimodal fusion (CM-MMF) approach that integrates image and RNA sequence modalities to achieve superior patient survival prediction. In some embodiments, an attention-based fusion circuit generates attention values ​​that highlight the importance of each modality during fusion for clinical diagnosis. Attention-based fusion architectures may be capable of achieving relatively improved multimodal patient survival prediction, e.g., achieving a C index of approximately 0.6587, and demonstrating the ability to inject complementary knowledge from different modalities.

[0038] Predicting cancer prognosis and survival outcomes can enable treatment response prediction and stratification of patients into different treatment groups. Integrating various data modalities into survival prediction models can improve their predictive power and benefit both clinical research and practice. Several embodiments allow for the prediction of survival outcomes in NSCLC patients using a combination of histopathology and genomics with attention-based deep learning approaches.

[0039] Figure 1 is a block diagram showing a prediction system 100 according to several embodiments of the present disclosure.

[0040] According to some embodiments, the prediction system 100 is a multimodality deep learning framework that integrates various modality data and uses an attention mechanism to predict patient outcomes. In some embodiments, the prediction system 100 is configured to receive first modality data 102 and second modality data 104 related to a patient and to generate a patient survival prediction (e.g., survival score) 142 based on modalities 102 and 104. The attention mechanism utilized by the prediction system 100 improves the prediction performance of the system 100 by focusing the system's attention on a particular modality based on different weights assigned to different modalities.

[0041] The modalities used by the prediction system 200 may be of different types. For example, the first modality 102 may include histological hematoxylin and eosin (H&E) image data. The H&E data 102 may include one or more digitized images of a patient's tissue sample (e.g., a tumor tissue sample) stained with hematoxylin and eosin dyes. The H&E dyes stain the cell nucleus, extracellular matrix, and cytoplasm, as well as other cellular structures, in different colors, thus enabling the pathologist and the prediction system 100 to distinguish between different cellular structures. The overall pattern of coloration from the staining also indicates the general arrangement and distribution of cells, providing a view of the structure of the tissue sample. In some examples, the H&E image data 102 may include one or more image tiles extracted (e.g., randomly selected and extracted) from a viable tumor area of ​​the stained tissue sample.

[0042] The second modality 104 may include DNA information of tumor mutations and / or gene sequencing data, such as mRNA gene expression, extracted from the patient's tumor tissue. Each tumor cell may have hundreds or thousands of tumor mutation genes. The second modality 104 may include some of the gene mutations found in the tissue sample. In some cases, only the expression of those most relevant to the patient's viability may be included in the second modality 104.

[0043] According to some embodiments, the prediction system 100 includes an attention-based fusion architecture for injecting first and second modalities (e.g., imaging and RNA-seq modalities) to achieve improved patient survival prediction. In some embodiments, the prediction system 100 includes a first model 110 configured to receive first modality data 102 and generate a first intermediate feature vector (e.g., pathology / histology feature vector) 112, and a second model 120 configured to receive second modality data 104 and generate a second intermediate feature vector (e.g., omic / sequencing feature vector) 114. In some embodiments, the prediction system 100 further includes an attention-based multimodal fusion circuit (e.g., a cross-modality attention-based multimodal fusion circuit (CM-MMF)) 130 which fuses first and second intermediate feature vectors 112 and 114 to generate first and second weights (e.g., attention scores or scaling factors) for scaling the first and second intermediate feature vectors 112 and 114 based on their relative importance in diagnosing a disease, and generates an aggregated feature vector 132 based on the scaled first and second intermediate feature vectors. The prediction system 100 also includes a classifier 140 which generates a patient survival probability prediction 142 based on the aggregated feature vector 132.

[0044] In some embodiments, the classifier 140 includes a neural network having several layers, each capable of performing convolution operations on the aggregated feature vector 132 through the application of kernels / filters. The neural network may be a convolutional neural network (ConvNet / CNN), a recurrent neural network (RNN), a multilayer perceptron (MLP), and the like, according to some examples. However, embodiments of the present disclosure are not limited thereto, and the classifier 140 may include, for example, a single layer.

[0045] The viability prediction 142 generated by the classifier 140 may correspond to the patient's overall viability. In some examples, a continuous timescale of total patient survival in days or months may be divided into multiple non-overlapping bins (e.g., four non-overlapping bins, bin 1 (1–200 days), bin 2 (201–400 days), bin 3 (401–600 days)), and the output of the classifier 140 may be a specific bin within the multiple non-overlapping bins (e.g., a specific range of viable days from a range of multiple consecutive values). However, in some examples, the output of the classifier 140 may be a raw viability score indicating the patient's survival in days or months.

[0046] Once the prediction system 100 generates a viability prediction 142, the prediction may be sent to a server (e.g., a remote server or cloud server) 150 for further processing and / or to a display device 160 for display to a user.

[0047] While the above description illustrates two modalities as examples of input modalities 102 and 104 to the prediction system 100, embodiments of the present disclosure are not limited thereto, and any suitable type of modality may be employed by the prediction system 100 to generate the viability prediction 142. For example, the prediction system 100 may utilize one or more derivatives of modalities 102 and 104.

[0048] Figure 2A shows a block diagram of a first model 110 according to some embodiments of the present disclosure.

[0049] In some examples, pathology data 102 includes digitized whole slide images (WSI, e.g., digitized images) 202a of tissue sample slices from a patient containing tumor or cancer cells (e.g., lung cancer cells). The tissue sample may be stained with hematoxylin and eosin (H&E), which produce a coloring pattern that reveals the general layout and distribution of cells, distinguishes different types of tissue, and gives a general overview of the structure of the tissue sample. However, embodiments of the present disclosure are not limited thereto, and tissue sections may be stained in any suitable way so that WSI 202a identifies malignant (cancer) cells, non-squamous epithelial cells of NSCLC, etc. WSI 202a may be too large to process as a whole (e.g., it may be several gigapixels in size) and therefore may be divided into smaller regions / sections, referred to herein as tiles (or patches) 202b, which are easier or more manageable to process. For example, an image classifier 109, such as a pre-trained model using the U-Net architecture, may classify regions of WSI202a as tumor and stroma, and tiles 202b (e.g., 512×512 pixel tiles) may be extracted from the classified regions of WSI202a. The first model 110 may receive tiles 202b, or it may receive WSI202a and subdivide it into multiple tiles 202b.

[0050] According to some embodiments, the first model 110 utilizes an attention-based multi-instance learning (AMIL) pipeline that extracts features 212 from each tile 204 of the WSI202a and embeds the individual tile feature vectors 212 into a first intermediate feature vector 112. The first model 110 includes a tile encoder 210 and an attention-based aggregator 220.

[0051] The tile encoder 210 encodes each tile into a tile feature vector 212 (for example, a 1024-channel feature vector). In some examples, the tile encoder 210 encodes each tile using a ResNet50-based image encoding architecture. However, embodiments of the present disclosure are not limited thereto, and any suitable image encoder, such as a convolutional neural network (CNN), may be used.

[0052] The attention-based aggregator 220 receives the tile feature vector 212, determines the tile weight for each tile based on its perceived relevance to patient-level prognosis prediction, highlights important regions when aggregating the tiles into a WSI feature representation, and identifies pivot tiles. That is, regions assigned higher tile weights contribute more to the patient-level feature representation than regions assigned lower tile weights. The tile weights are then used to create a slide-level representation feature vector, i.e., a first intermediate feature vector 112. This can be achieved by an attention pooling operation that aggregates information from all regions of the patient's WSI. In some examples, the first intermediate feature vector 112 may be a 1024-channel feature vector (e.g., a 1 × 1024 matrix) representing the entire input WSI 202a. However, embodiments of the present disclosure are not limited thereto, and in some examples, the first model 110 may output a 128-channel feature vector (e.g., a 128 × 1 matrix) representing the entire input WSI 202a.

[0053] Figure 2B shows a block diagram of a second Model 120 according to some embodiments of the present disclosure.

[0054] In some examples, the genetic data 104 includes bulk sequencing RNA data 204a from which pre-selected features 204b can be extracted. In some examples, the pre-selected features 204b may include 154 features from thousands of bulk sequencing RNA data 204a. However, embodiments of the present disclosure are not limited thereto, and any appropriate number of pre-selected features 204b may be used. For example, the number of pre-selected features may vary depending on the location of different tissues (e.g., breast, lung, etc.) and the type of cancer being analyzed. Furthermore, the second modality 104 may include other molecular profiling data such as mutation status and copy number variation.

[0055] In some embodiments, the second model 120 includes a self-normalizing neural network (SNN) 230 or a feedforward neural network (FNN) for embedding the extracted RNA sequence information 204b into a feature vector for omic feature representation, i.e., a second intermediate feature vector 114. In some examples, the second intermediate feature vector 114 may be a 128-channel feature vector. However, embodiments of the present disclosure are not limited thereto, and the feature vector 114 may contain any suitable number of channels. Furthermore, embodiments of the present disclosure are not limited to using an FNN network, and sequencing data may be processed using any suitable network, such as a self-normalizing network (SNN).

[0056] Figure 3A shows a block diagram of an attention-based multimodal fusion circuit 130 employing a non-shared attention-based multimodal fusion approach according to several embodiments of the present disclosure.

[0057] In some embodiments, the attention-based multimodal fusion (AMMF) circuit 130 includes a separate modality attention path 300 that considers the importance of each modality data 102 and 104 for survival prediction. Each modality attention path 300 receives and processes the corresponding first and second intermediate feature vectors 112 and 114. Each modality attention path 300 includes two convolutional layers 302 and 306 and an activation function layer 304 between them, which introduces nonlinearity into the model and improves how well the AMMF circuit 130 is trained (for example, the activation function may produce a zero-centered output that supports the backpropagation process during training). The convolutional layers 302 and 306, with decreasing channel numbers, compress features into a more compact representation based on certain knowledge for future purposes. In some examples, the fully convolutional layers 302 and 306 may be convolutional neural networks (CNNs) with a 1x1 convolution kernel / filter size, and the activation layer 304 may be a normalized linear unit (ReLU) function layer. However, embodiments of the present disclosure are not limited thereto, and the kernels of the convolutional layers 302 and 306 may have any suitable size, and any suitable nonlinear activation function (such as a hyperbolic tangent (Tanh) function layer) may be used instead of the ReLU function layer.

[0058] In the embodiment of Figure 3A, since the intermediate feature vectors 112 and 114 are processed by different modality attention paths 300, the intermediate feature vectors 112 and 114 do not have to be homogeneous, that is, they may have different numerical ranges and / or different dimensions (e.g., different vector lengths). However, in other examples, the intermediate feature vectors 112 and 114 may be normalized to be homogeneous, that is, to have the same numerical range (e.g., 0 to 1) and / or have the same dimensions (e.g., the same vector length).

[0059] As shown in Figure 3A, the kernel weights of convolutional layers 302 and 306 may not be shareable between different modalities, which would lead to the AMMF circuit 130 learning correction information. In other words, the AMMF in Figure 3A may employ a non-kernel-sharing approach. For example, the kernel value of the modality attention path 300 corresponding to the first modality data 102 may differ from the kernel value of the modality attention path 300 corresponding to the second modality data 104.

[0060] The attention-based multimodal fusion circuit 130 may also include a softmax layer 310, which assigns a fractional probability to the outputs of two separate modality attention paths 300. Thus, the softmax layer 310 outputs two attention scores a1 and a2, which sum to 1, representing the relative weights / importance of the two input modalities 102 and 104 with respect to the viability prediction 142 of the prediction system 100.

[0061] In some embodiments, the cross-modality attention score (a m ) can be expressed according to the following equation (1).

number

[0062] In the formula, m is an integer exponent (modality), M is an integer representing the number of input modalities 10² / 10⁴ (e.g., 2), and W m ∈R L×1 and V m ∈R L×N is a matrix of trainable parameters for the AMMF circuit 130 representing the weights of two convolutional layers, L represents the size (e.g., length) of the unimodal embedding output fm (i.e., the size of the first / second intermediate feature vectors 112 / 114), N is the number of output channels of the first layer of the AMMF circuit 130 (i.e., the size of the output of the first convolutional layer 302), T represents the matrix transpose operation, and ReLU(·) represents the normalized linear unit activation function.

[0063] In some embodiments, the attention-based multimodal fusion circuit 130 also includes a feature aggregator (e.g., a feature aggregation circuit) 312 that applies attention scores a1 and a2 to the corresponding first and second intermediate feature vectors 112 and 114, and combines the results to generate an aggregated feature vector 132. In some examples, the feature aggregator 312 may scale the first intermediate feature vector 112 based on the first attention score a1 to generate a first scaled intermediate feature vector, scale the second intermediate feature vector 114 based on the second attention score a2 to generate a second scaled intermediate feature vector, and aggregate (e.g., combine) the first and second scaled intermediate feature vectors to generate an aggregated feature vector 132. The scaling operation performed by the feature aggregator 312 may include multiplying the attention scores a1 / a2 by the corresponding intermediate feature vectors 112 / 114. In some examples, the aggregation process may include concatenating the first and second scaled intermediate feature vectors to generate the aggregated feature vector 132. However, embodiments of the present disclosure are not limited thereto.

[0064] According to some embodiments, the aggregation operation of the feature aggregator 312 is performed on an integrated cross-modality representation. To generate TIFF2026533947000003.tif5170 (i.e., aggregated feature vector 132), cross-modality attention score (a m This involves multiplying by the corresponding modality intermediate feature vector 112 / 114, which can be expressed by equation (2).

number

[0065] For the final prediction, classifier 140 (e.g., a single-layer classifier) ​​performs cross-modality embedding ( It is implemented to facilitate patient-specific survival predictions using TIFF2026533947000005.tif5170.

[0066] In the embodiment shown in Figure 3A, the intermediate feature vectors 112 and 114 of the input modalities 102 and 104 are processed separately via two independent modality attention paths 300, which may have convolutional layers with different kernel values ​​for different modalities. However, embodiments of the present disclosure are not limited to this non-shared attention-based multimodal fusion architecture. For example, the intermediate feature vectors corresponding to the input modalities may be processed via a shared attention-based multimodal fusion architecture.

[0067] Figure 3B shows a block diagram of an attention-based multimodal fusion circuit 130-1 that employs a shared attention-based multimodal fusion approach according to some other embodiments of the present disclosure.

[0068] According to some embodiments, an attention-based multimodal fusion (AMMF) circuit 130-1 determines the relative importance of two input intermediate feature vectors 112 and 114 via a shared / common attention-based multimodal fusion (AMMF) circuit 130-1. Here, the intermediate feature vectors 112 and 114 (each of which may be, for example, a 128-channel feature vector) are the outputs of two different models 110 and 120, and therefore do not have to be homogeneous, i.e., may have different numerical ranges. In such an example, the AMMF circuit 130-1 includes a feature merging circuit 301. This normalizes (e.g., scales) the input intermediate feature vectors 112 and 114 to be homogeneous, i.e., to have the same numerical range (e.g., 0 to 1), merges the normalized features into a combined feature 116 (e.g., a 128 × 2 feature vector), and is then processed by a single modality attention path 300-1. The normalization performed by the feature merging circuit 301 may be a scaling function or any suitable normalization technique that transforms the input features 112 and 144 into a common domain before they are merged. The feature merging circuit 301 is shown in Figure 3B as part of the AMMF circuit 130-1, but embodiments of the present disclosure are not limited thereto, and the operation of the feature merging circuit 301 may be performed outside of the AMMF circuit 130-1.

[0069] The combined feature 116 may be a concatenation of the normalized input features 112 and 114 into a one-dimensional vector having a length equal to the sum of the lengths of their normalized feature vectors, or it may be a stack of the normalized input features 112 and 114 into a two-dimensional array where each row / column contains corresponding elements of two normalized input features (as shown in Figure 3B). In the example in Figure 3B, the input intermediate feature vectors 112 and 114 may have the same dimensions (e.g., they may have the same vector length).

[0070] According to some embodiments, the common modality attention path 300-1 includes two convolutional layers 302-1 and 306-1, which may be two-dimensional convolutional layers, and an activation function layer 304-1 between them, which may be a hyperbolic tangent (Tanh) function layer. In some examples, the full convolutional layers 302-1 and 306-1 may have a 1×1 convolution kernel / filter applied simultaneously (e.g., at the same time) to both dimensions of the combined feature 116. However, embodiments of the present disclosure are not limited thereto, and the kernels of the convolutional layers 302 and 306 may have any suitable size. Furthermore, the activation layer 304-1 is not limited to a Tanh function layer, but may be any suitable nonlinear activation function (such as a normalized linear unit (ReLU) function layer).

[0071] As shown in Figure 3B, the AMMF circuit 130-1 in Figure 3B may employ a kernel-sharing approach in which the kernel weights of the convolutional layers 302-1 and 306-1 are shareable across different modalities. That is, intermediate feature vectors 112 and 114 from different modalities 102 and 104 are convolved by the same kernel values. This helps to facilitate the overall learning of the importance of modality-specific knowledge through cross-modality relationships.

[0072] The AMMF circuit 130-1 may also include a softmax layer 308-1, which may be the same as or substantially the same as the softmax layer 308 in Figure 3A, and is configured to output attention scores (e.g., cross-modality attention score / weights) a1 and a2, which total 1 and represent the relative weights / importance of the two input modalities 102 and 104 with respect to the viability prediction 142.

[0073] According to some embodiments, cross-modality attention (am) can be expressed according to the following formula (3).

number

[0074] wherein m is an integer exponent, M is an integer representing the number of input modalities 102 / 104 (e.g., 2), and W m ∈R L×1 and V m ∈R L×N are matrices of trainable parameters of the AMMF circuit 130-1 representing the weights of the two convolutional layers, L represents the size (e.g., length) of the unimodal embedding output fm (i.e., the size of the first / second intermediate feature vector 112 / 114), N is the number of output channels of the first layer of the AMMF circuit 130-1 (i.e., the output size of the first convolutional layer 302-1), T represents a matrix transpose operation, and Tanh(·) represents a tangent-wise element-wise nonlinear activation function.

[0075] The AMMF circuit 130-1 further includes a feature aggregator 312, which applies attention scores a1 and a2 to the corresponding ones of the first and second intermediate feature vectors 112, 114, and combines the results to generate an aggregated feature vector 132. The operations of the feature aggregator 312 and the resulting aggregated feature vector 132 have been described above with reference to FIG. 3A, so their descriptions will not be repeated here for the sake of brevity.

[0076] According to some embodiments, when training the AMMF circuit 130 / 13-1, loss functions such as a survival loss function or a Cox loss function can be used to supervise the results from the fusion architecture. In the example of a survival loss function, a continuous timescale of total patient survival times in days or months can be divided into multiple non-overlapping bins (e.g., four non-overlapping bins). A negative log-likelihood (NLL) survival loss can be used as a classification task to supervise training using both censoring conditions and bin interval labels. The survival loss can be flexible for data of varying sizes with a batch size of 1. In the example of a Cox loss function, the order of survival times within a sample group is supervised using survival times and censoring conditions as a regression task. An observation is said to be censored if the event of interest (e.g., death, recurrence, failure) did not occur or be observed by the end of the study period. The Cox loss can utilize relatively large batch sizes (e.g., 32, 64, etc.) to achieve better performance. Thus, the size of the data can be made uniform before being loaded into the model.

[0077] In some cases, the first and second modalities may be based on data from patients treated with atezolizumab + carboplatin + paclitaxel, derived from a phase III clinical trial evaluating the efficacy of adding programmed cell death ligand 1 (PD-L1) targeted therapy to current standard treatment in non-small cell lung cancer. In some cases, a multimodal framework may be based on anonymized histopathology images (e.g., from 270 patients) along with bulk RNA sequence data.

[0078] In some examples, the first modality data (e.g., pathology data) 102 may include tissue image data. For example, H&E-stained whole slide image (WSI) data may be scanned (e.g., 20 × (0.5 microns / pixel)). In some examples, the slides may include manual annotation of stromal and tumor regions. However, in some embodiments, a pre-trained model using a U-Net architecture is utilized to classify the regions into tumor and stromal. Pixel tiles (e.g., 512 × 512) may be taken from annotated / classified WSI to form the first modality data 102. In some examples, each image tile may be embedded in a first intermediate feature vector 112 via a first model 110 with pre-trained model weights with a ResNet-50 backbone, where this vector may be a 1024 channel feature vector (e.g., a vector length of 1024).

[0079] In some examples, the second modality data (e.g., genetic data)104 includes RNA sequence data containing gene expression values ​​along with ensemble gene identifiers. According to some examples, the RNA sequence data may be preprocessed by (1) mapping all ensemble gene IDs to gene symbols, (2) normalizing the RNA sequence data to generate transcript permillion (TPM) expression data, (3) calculating the Z-score of the TPM data, and (4) selecting (e.g., manually selecting) several genes (e.g., 154 genes) most strongly associated with lung cancer from the curated data.

[0080] Such input modality data can be used to test the performance of the predictive system 100 regarding unimodal and fused systems of related technologies.

[0081] Figure 4A shows a table comparing the performance of prediction system 100 across various modalities under various types of loss supervision, according to several embodiments of the present disclosure, with that of various systems of the relevant technology.

[0082] Table 1 shows that the survival prediction results are evaluated using the fit index (c-index), which measures the proportion of pairs of all possible observations in which the model's predictions accurately predict the actual order of survival times.

[0083] As shown in Figure 4A, most fusion designs using multimodal learning achieved better performance than unimodal learning, demonstrating the ability to inject modality-specific knowledge from different modalities. However, prediction system 100 (last row of the table) using cross-modality attention-based multimodal fusion achieved a maximum c-index of approximately 0.6587 compared to other systems of related technologies. In the table in Figure 4A, "raw fusion" may refer to fusion designs of related technologies in which RNA-seq features are directly fused to image features without passing through a feedforward neural network (FNN).

[0084] To improve training robustness, Gaussian noise was added to image and RNA-seq features before loading them into the AMMF circuit 130-1. All models in the table in Figure 4A were trained over 55 epochs using the ADAM optimizer with a learning rate of 0.01 and a batch size of 1. Standardization was performed for the RNA-seq modality, and the normalization was expanded to rearrange the feature vectors between 0 and 1 for all modalities before implementing the fusion architecture.

[0085] Figure 4B shows a table comparing the performance of prediction systems 100 using shared and non-shared kernel layers and different activation functions according to several embodiments of the present disclosure.

[0086] The table in Figure 4B shows that various attention mechanism designs using different activation functions (e.g., ReLU and Tanh) were evaluated using the same dataset as in Figure 4A. Attention-based fusion approaches in several embodiments are shown in Figure 4B as being divided into two strategies, distinguished by whether or not kernel weights are shared (see, for example, the kernel-shared design of AMMF circuit 130-1 and the non-shared kernel design of AMMF circuit 130) while learning embedding features from multiple modalities. As shown by the survival prediction performance in Table 3, in some examples, better performance can be achieved with a higher mean c-exponent by sharing kernel weights in the attention-based fusion approach with the Tanh activation function (e.g., as shown in Figure 2). However, this is only an example, and the performance of each design in Figure 4B may vary depending on the modality data used.

[0087] Figure 5 is a flowchart illustrating a process 500 in which a predictive system 100 predicts the overall viability of a patient, according to several embodiments of the present disclosure.

[0088] In some embodiments, the prediction system 100 (e.g., first and second models 110 and 120) receives first modality data 102 and second modality data 104 corresponding to a patient (S502), and generates first intermediate feature vectors 112 and second intermediate feature vectors 114 based on the first and second modality data 102 and 104 (S504).

[0089] At that time, the prediction system 100 (for example, the AMMF circuit 130) determines a first attention score a1 and a second attention score a2 based on the first and second intermediate feature vectors 112 and 114 (S506), and generates an aggregated feature vector 132 based on the first and second intermediate feature vectors 112 and 114 and the first and second attention scores a1 and a2 (S508).

[0090] In some embodiments, determining first and second attention scores a1 and a2 includes combining the first and second intermediate feature vectors 112 and 114 to generate a combined feature 116, processing the combined feature 116 by one or more convolutional layers 302-1 and 306-1 and an activation layer 304-1 to generate first and second attention values ​​corresponding to the first and second intermediate feature vectors 112 and 114, and converting the first and second attention values ​​into first and second attention scores a1 and a2 by a softmax function layer 308-1. In some examples, combining the first and second intermediate feature vectors 112 and 114 includes stacking the first and second intermediate feature vectors 112 and 114 onto a two-dimensional array which is the combined feature 116. In such examples, one or more convolutional layers 302-1 and 306-1 are configured to perform two-dimensional convolutions on the two-dimensional array 116.

[0091] In some other embodiments, determining first and second attention scores a1 and a2 includes processing a first intermediate feature vector 112 with one or more first convolutional layers 302 and 306 and a first activation layer 304 to generate a first attention value corresponding to the first intermediate feature vector 112; processing a second intermediate feature vector 114 with one or more second convolutional layers 302 and 306 and a second activation layer 304 to generate a second attention value corresponding to the second intermediate feature vector 114; and converting the first and second attention values ​​into first and second attention scores a1 and a2 by a softmax function layer 308.

[0092] Next, the prediction system 100 (e.g., classifier 140) generates a survival probability prediction 142 corresponding to the patient based on the aggregated feature vector 132 (S510).

[0093] As described above, some embodiments include attention-based multimodal fusion architectures for injecting knowledge from pathological and genetic data to achieve improved lung cancer survival prediction. Attention-based multimodal fusion methods can achieve superior lung cancer survival prediction with higher C-index compared to other fusion designs and unimodal learning methods. Attention scores from the fusion layer can demonstrate the importance of each modality for diagnosis, while instances of attention from multiple instance learning (AMIL) can indicate the contribution of each image tile. According to some embodiments, cross-modality attention-based multimodal fusion methods can outperform other fusion designs and unimodal learning methods. This highlights their ability to integrate modality-specific knowledge from various sources and emphasizes the functionality of multimodal fusion in taking cross-modality relationships into account.

[0094] According to various embodiments of the present disclosure, the prediction system 100 is implemented using one or more processing circuits or electronic circuits configured to perform the various operations described above. Types of electronic circuits may include central processing units (CPUs), graphics processing units (GPUs), artificial intelligence (AI) accelerators (e.g., vector processors, which may include vector arithmetic logic units configured to efficiently perform operations common to neural networks, such as dot products and softmax), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), and the like. For example, in some situations, aspects of the embodiments of the present disclosure are implemented by program instructions stored in non-volatile computer-readable memory that perform the operations described when executed by electronic circuits (e.g., CPUs, GPUs, AI accelerators, or combinations thereof). The operations performed by the prediction system 200 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, etc.) or may be assigned among multiple electronic circuits (e.g., multiple GPUs, or CPUs in combination with GPUs). Multiple electronic circuits may be local to one another (e.g., located on the same die, in the same package, or in the same embedded device or computer system), and / or remote to one another (e.g., in communication over a network such as a local personal area network such as Bluetooth®, communication over a local area network such as a local wired and / or wireless network, and / or communication over a wide area network such as the Internet, where some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits operating to implement the prediction system 200 may be referred to herein as a computer or computer system, which may include memory that stores instructions that, when executed by one or more electronic circuits, implement the systems and methods described herein.

[0095] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms “a” and “an” as used herein are intended to include the plural form unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and “including,” as used herein, identify the presence of a described feature, integer, step, action, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. The term “and / or” as used herein includes any one or any combination of the associated enumerated items. Expressions such as “at least one of ~” modify the entire list of elements, when preceding a list of elements, and do not modify any individual elements of the list.

[0096] Terms such as “first,” “second,” and “third” may be used herein to describe various elements, components, regions, layers, and / or parts, but it will be understood that these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or part from other elements, components, regions, layers, or parts. Accordingly, the first element, component, region, layer, or part described below may be called the second element, component, region, layer, or part without departing from the spirit and scope of the invention.

[0097] As used herein, the terms “substantially,” “about,” and similar terms are used as approximations rather than terms of degree, and are intended to describe the inherent deviations of measured or calculated values ​​that would be recognized by those skilled in the art. Furthermore, the use of “may” when describing embodiments of the invention refers to “one or more embodiments of the invention.” As used herein, the terms “use,” “by means of,” and “to be used” may be considered synonymous with the terms “to utilize,” “by utilizing,” and “to be utilized,” respectively.

[0098] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meanings in the relevant art and / or in the context of this specification, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0099] While some exemplary embodiments of systems and methods for quantifying pathology slides using cell-based scoring systems are described and illustrated herein, various modifications and variations can be made without departing from the spirit and scope of the embodiments of this disclosure, as will be understood by those skilled in the art. Therefore, it should be understood that pathology slide manufacturing systems and methods according to the principles of this disclosure may be embodiments other than those specifically described herein. This disclosure is also defined in the following claims and equivalents.

Claims

1. A method for predicting a patient's overall survival rate using a machine learning-based prediction system, wherein the method is The prediction system receives first modality data and second modality data corresponding to the patient. The prediction system generates a first intermediate feature vector and a second intermediate feature vector based on the first and second modality data. The prediction system determines a first attention score and a second attention score based on the first and second intermediate feature vectors. The prediction system generates aggregated feature vectors based on the first and second intermediate feature vectors and the first and second attention scores, and The prediction system generates a survival probability prediction corresponding to the patient based on the aggregated feature vector. Methods that include...

2. The method according to claim 1, wherein the first modality data includes histological hematoxylin and eosin (H&E) image data, and the second modality data includes gene sequencing data.

3. The H&E image data includes digitized images of the patient's tissue sample stained with hematoxylin and eosin dyes. The gene sequencing data includes mRNA gene expression extracted from the patient's tumor tissue. The method according to claim 2.

4. Receiving the first modality data and the second modality data means that The first model of the prediction system receives the first modality data, and The second model of the prediction system receives the second modality data. The method according to claim 1, including the method described in claim 1.

5. The generation of the first intermediate feature vector and the second intermediate feature vector is The first model generates the first intermediate feature vector based on the first modality data, and The second model generates the second intermediate feature vector based on the second modality data. The method according to claim 4, including the method described in claim 4.

6. The method according to claim 4, wherein the first model includes an attention-based multi-instance learning (AMIL) model, and the second model includes a feedforward neural network (FNN) model.

7. Determining the first attention score and the second attention score is The first and second intermediate feature vectors are combined to generate a combined feature. To generate first and second attention values ​​corresponding to the first and second intermediate feature vectors, the coupled features are processed by one or more convolutional layers and activation layers, and Converting the first and second attention values ​​into the first and second attention scores using a softmax function layer, The method according to claim 1, including the method described in claim 1.

8. The combination of the first and second intermediate feature vectors is This includes stacking the first and second intermediate feature vectors onto the combined feature two-dimensional array, and The one or more convolutional layers are configured to perform a two-dimensional convolution in the two-dimensional array. The method according to claim 7.

9. The activation layer includes a hyperbolic tangent (Tanh) function layer or a normalized linear unit (ReLU) function layer, and The sum of the first and second attention scores is equal to 1. The method according to claim 7.

10. The method according to claim 7, further comprising normalizing the values ​​of the first and second intermediate feature vectors before combining them.

11. Determining the first attention score and the second attention score is Processing the first intermediate feature vector with one or more first convolutional layers and first activation layers to generate a first attention value corresponding to the first intermediate feature vector, Processing the second intermediate feature vector with one or more second convolutional layers and second activation layers to generate a second attention value corresponding to the second intermediate feature vector, and Converting the first and second attention values ​​into the first and second attention scores using a softmax function layer, The method according to claim 1, including the method described in claim 1.

12. Each of the first and second activation layers includes a hyperbolic tangent (Tanh) function layer or a normalized linear unit (ReLU) function layer, The sum of the first and second attention scores is equal to 1. The method according to claim 11.

13. The method according to claim 11, wherein the one or more first convolutional layers include kernel values ​​that are not the same as those of the one or more second convolutional layers.

14. The generation of the aggregated feature vector is The first intermediate feature vector is scaled based on the first attention score to generate the first scaled intermediate feature vector. The process involves scaling the second intermediate feature vector based on the second attention score to generate a second scaled intermediate feature vector, and The first and second scaled intermediate feature vectors are aggregated to generate the aggregated feature vector. The method according to claim 1, including the method described in claim 1.

15. The aggregation of the first and second scaled intermediate feature vectors is The method involves concatenating the first and second scaled intermediate feature vectors to generate the aggregated feature vector, wherein the aggregated feature vector has a length equal to the sum of the lengths of the first and second scaled intermediate feature vectors. The method according to claim 14, including the method described in claim 14.

16. The generation of the survival probability prediction is The classifier of the prediction system generates the survival probability prediction corresponding to the patient based on the aggregated feature vector, The survival probability prediction corresponds to the overall survival probability of the patient. The method according to claim 1.

17. The method according to claim 16, wherein the survival probability prediction includes a range of values ​​from a plurality of consecutive ranges of values.

18. The method according to claim 1, further comprising transmitting the survival probability prediction to a display device for display to a user.

19. A predictive system for predicting a patient's overall survival probability based on machine learning, wherein the predictive system A first model configured to receive first modality data corresponding to the patient and to generate a first intermediate feature vector based on the first modality data, A second model configured to receive second modality data corresponding to the patient and to generate a second intermediate feature vector based on the second modality data, and An attention-based multimodal fusion circuit configured to determine a first attention score and a second attention score based on the first and second intermediate feature vectors, and to generate an aggregated feature vector based on the first and second intermediate feature vectors and the first and second attention scores, A prediction system that includes this.

20. The prediction system according to claim 19, further comprising a classifier configured to receive the aggregated feature vector and generate a survival probability prediction corresponding to the patient based on the aggregated feature vector.

Images