Cancer typing method and system based on cross-modal attention and variation representation learning
By employing cross-modal attention and variational representation learning, we have addressed the heterogeneity and modality loss issues of multimodal data in real-world clinical settings, achieving highly robust and accurate cancer subtyping and providing clinical interpretability of microscopic features.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-03-17
- Publication Date
- 2026-05-15
AI Technical Summary
Existing cancer classification systems face challenges when processing multimodal data in real-world clinical settings. These challenges include heterogeneity, high noise levels, modality bias, missing modalities, and high-noise sparse data, leading to decreased predictive performance and a lack of clinical interpretability.
We employ a cross-modal attention and variational representation learning approach, using independent mapping and dynamic weighting mechanisms to handle heterogeneous modalities. We introduce modality missing masking and random masking reconstruction strategies, and combine MMD distribution regularization loss and DeepLIFT algorithm to achieve high robustness and micro-feature attribution explanation for multimodal data.
It significantly improves the accuracy and resilience of cancer classification systems, maintains high-precision prediction under partial modality loss conditions, and provides clinical interpretability of microscopic biomarkers.
Smart Images
Figure CN122045897A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of bioinformatics processing technology, and particularly relates to a cancer subtyping method and system based on cross-modal attention and variational representation learning. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] The occurrence and development of complex diseases (such as head and neck cancer, liver cancer, etc.) are often accompanied by the evolution of multi-level, multimodal data. In real clinical settings, doctors rely on highly heterogeneous multimodal data for diagnosis, prognostic assessment, and subtyping. Currently, although deep learning has made significant progress in medical image analysis and natural language processing, the multimodal data in real clinical settings in current cancer subtyping systems exhibits extremely "non-ideal characteristics" (i.e., highly heterogeneous, easily incomplete, and extremely noisy). Existing multi-omics or multimodal fusion subtyping methods, such as models based on graph neural networks or simple attention splicing, are often based on the ideal assumption of "homogeneous and complete data," making it difficult to cope with real clinical challenges. Specifically, firstly, clinical forms, texts, and images cross a huge semantic gap, and traditional splicing is easily dominated by a single modality (such as images) with high information content or high noise, resulting in severe "modal bias"; secondly, in real medical scenarios, due to limitations in examination costs and patient compliance, very few patients have full-modal data. Existing models rely heavily on joint distribution alignment, and their predictive performance will catastrophically degrade once they encounter random missing modalities. Third, when dealing with noisy and sparse medical data, traditional variational autoencoders (VAEs) are prone to "posterior collapse" (potential loss of discriminative power of representations), and the "black box nature" of deep networks cannot provide microscopic biomarker traceability, resulting in a lack of clinically credible predictive results. Summary of the Invention
[0004] To address at least one of the technical problems in the background art, the present invention provides a cancer typing system based on cross-modal attention and variational representation learning, which can maintain strong robustness even under partial modality loss conditions, and provides an intelligent multimodal typing system that can accurately fuse heterogeneous modalities and has the ability to explain micro-feature attribution.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: A first aspect of the present invention provides a cancer subtyping method based on cross-modal attention and variational representation learning, comprising the following steps: Acquire patients' multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data; The acquired multimodal medical data and the constructed cancer subtyping model are used for training to obtain the trained cancer subtyping model; the construction process of the cancer subtyping model includes: Multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; A multi-head attention mechanism is introduced. For each attention head in the multi-head attention mechanism, an independent mapping space is constructed for each modality. This includes: generating a corresponding query matrix, key matrix, and value matrix based on the feature representation of each modality, and concatenating the features along the sequence dimension to obtain a global multi-modal query matrix, key matrix, and value matrix; and weighting the global value matrix with the attention weights obtained from the global multi-modal query matrix and key matrix to obtain the output of each attention head. The outputs of multiple attention heads are concatenated along the feature dimension and then linearly transformed to obtain a multimodal fusion representation that aggregates cross-modal complementary information. Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation, and sample to obtain latent variables; Based on latent variables and classifiers, cancer subtyping results are obtained through Softmax. The cancer classification results are obtained by predicting the medical data of the target object based on the trained cancer classification model.
[0006] Furthermore, the process of encoding the multimodal medical data and mapping it to a unified-dimensional modality-specific representation space to obtain the feature representation of the corresponding modality includes: The clinical phenotypic data is encoded, including Z-score standardization of continuous variables and one-hot encoding of discrete variables. The data is then fed into an MLP network with three hidden layers to output clinical feature representations. Encoding pathological text data includes: converting pathological text data into a sequence using WordPiece tokenization, inputting the sequence into BioBERT, extracting the output of the [CLS] tag with global context semantics at the last layer as the initial text feature, and then reducing the dimensionality of the initial text feature through a linear mapping layer to obtain the text feature representation; The tissue image data is encoded by: using a pre-trained ResNet50 as the backbone network, removing the original fully connected classification head of ResNet50, retaining the feature map output by the last convolutional layer, then applying global average pooling to extract spatially invariant features, and finally aligning the dimensions through linear mapping to output the image feature representation.
[0007] Furthermore, the attention weights obtained through the global multimodal query matrix and the key matrix are used to weight the global value matrix to obtain the output of each attention head, including: The global query matrix is multiplied by the transposed global key matrix and divided by a scaling factor to introduce a modality missing mask matrix. Attention weights are obtained by Softmax normalization. Finally, the global value matrix is weighted to obtain the output of each attention head.
[0008] Furthermore, the statistical parameters representing the latent posterior distribution in the multimodal fusion include the mean vector. With log variance , represented as: , , in, The vector representing the mean of the latent posterior distribution; Represents the log-variance vector of the latent posterior distribution; This represents a one-dimensional multimodal fusion feature vector that, after aggregation by the cross-modal attention module, is input to the variational bottleneck. and These represent the vectors used to infer the mean. The weight matrix and bias terms of the fully connected layer; and These represent the vectors used to infer the logarithmic variance. The weight matrix and bias terms of the fully connected layer.
[0009] Furthermore, when training the cancer subtyping model using a multi-task joint optimization method, a random modality mask is introduced, and the model is trained together using MMD distribution regularization loss, reconstruction consistency loss, and downstream classification cross-entropy loss. The MMD distribution regularization loss is constructed by calculating the distance between the latent distribution sample set and the sample set sampled from the standard normal prior. The reconstruction consistency loss works by randomly masking one or more input modalities according to a Bernoulli distribution in each training batch, and then using latent variables to inversely map the original data through a symmetric decoder to reconstruct the original data structure. The classification cross-entropy loss is constructed based on latent variables and classifiers, and the cancer typing results are obtained through Softmax.
[0010] Furthermore, the loss function for training the cancer subtyping model is: , in, Represents the overall objective loss function. Represents the classification cross-entropy loss. For hyperparameters, This indicates that only the reconstruction loss of the unmasked modes is calculated. This represents the regularity loss of the MMD distribution.
[0011] Furthermore, the method also includes using a deep differential propagation algorithm to calculate the contribution of each input feature in multimodal medical data relative to a reference baseline, thereby achieving attribution analysis from macro-modal to micro-biomarker.
[0012] A second aspect of the present invention provides a cancer subtyping system based on cross-modal attention and variational representation learning, comprising: The data acquisition module is used to acquire patients' multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data. The model building module is used to train the cancer subtyping model based on the acquired multimodal medical data and the constructed cancer subtyping model, resulting in a trained cancer subtyping model. The construction process of the cancer subtyping model includes: Multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; A multi-head attention mechanism is introduced. For each attention head in the multi-head attention mechanism, an independent mapping space is constructed for each modality. This includes: generating a corresponding query matrix, key matrix, and value matrix based on the feature representation of each modality, and concatenating the features along the sequence dimension to obtain a global multimodal query matrix, key matrix, and value matrix; weighting the global value matrix with the attention weights obtained from the global multimodal query matrix and key matrix to obtain the output of each attention head; concatenating the outputs of multiple attention heads along the feature dimension, and performing a linear transformation to obtain a multimodal fusion representation that aggregates cross-modal complementary information. Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation, and sample to obtain latent variables; Based on latent variables and classifiers, cancer subtyping results are obtained through Softmax. The cancer typing prediction module is used to predict cancer typing results based on the medical data of the target object using the trained cancer typing model.
[0013] A third aspect of the present invention provides a computer-readable storage medium.
[0014] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the cancer subtyping method based on cross-modal attention and variational representation learning as described above.
[0015] A fourth aspect of the present invention provides a computer device.
[0016] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the cancer subtyping method based on cross-modal attention and variational representation learning as described above.
[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention overcomes the limitations of existing technologies that can only process "idealized, homogeneous, complete data," and provides a highly robust variational cross-modal fusion and micro-attribution mechanism for real, non-ideal clinical data, significantly improving the accuracy, resilience, and clinical interpretability of cancer classification systems. Specifically: (1) The "independent mapping + dynamic weighting" mechanism is adopted to overcome the modality bias problem caused by heterogeneous data. In response to the cross-modal semantic gap of real image and text data, this invention constructs an independent feature projection space for each modality and dynamically and adaptively weights the value matrix through global query and Softmax calculation of the key matrix. This mechanism can intelligently evaluate the effective signal-to-noise ratio of each modality, automatically suppress the weight of high-noise modalities, and completely avoid the bias caused by the dominance of a single modality in traditional splicing methods.
[0018] (2) This invention pioneers a joint strategy of "forward masking + random masking reconstruction," endowing the model with extremely strong robustness against modality loss. Addressing the modality loss problem that is highly likely to occur in real clinical settings, this invention introduces a mask matrix (minimizing missing positions) in the forward cross-modal attention calculation, ensuring that the model can still function normally under incomplete input. Simultaneously, in multi-task joint training, a consistency loss based on Bernoulli distribution for random modality masking reconstruction is introduced, forcing the model to extract complementary features from the remaining modalities. This enables the system to maintain high-precision classification prediction even under extreme modality loss conditions.
[0019] (3) Integrating "MMD distribution regularization and DeepLIFT algorithm" to break the posterior collapse and black box dilemma of deep generation. Addressing the pain point of medical data easily causing model collapse, this invention abandons the traditional KL divergence and innovatively adopts MMD distribution regularization loss based on regenerative kernel Hilbert space (RKHS) to smoothly align prior distributions and effectively maintain the discriminative power of latent variables. Furthermore, by introducing the deep difference propagation algorithm, the activation difference of the top-level network is multiplied from top to bottom, accurately quantifying and tracing back to microscopic biomarkers (such as specific genes, pathological terms, or imaging regions), providing clinicians with evidence-based auxiliary explanations that break the black box.
[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0022] Figure 1 This is a schematic diagram of the overall framework of the cancer subtyping method based on cross-modal attention and variational representation learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the modality-specific feature encoder structure for pathological text and tissue images provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-modal feature interaction and semantic fusion mechanism based on multi-head attention provided in this embodiment of the invention; Figure 4 This is a bar chart comparing the classification performance of the method of the present invention and existing baseline models on real medical cohorts, as provided in the embodiments of the present invention. Figure 5 This is a schematic diagram of global attribution and core biomarker visualization of multimodal features based on the DeepLIFT algorithm provided in the embodiments of the present invention, wherein (a) is the overall modality attribution diagram, (b) is the image attribution, and (c) is the clinical features of the first 15 seconds. Detailed Implementation
[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0026] Example 1 like Figure 1 As shown, this embodiment provides a cancer subtyping method based on cross-modal attention and variational representation learning, including the following steps: Step 1: Obtain the patient's multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data; In this embodiment, the target patient dataset is defined as follows: ,in, For the total number of patients, Indicates the first i Structured clinical data from individual patients, Indicates the first i Individual patient pathology reports Indicates the first i mIHC and other tissue imaging data of individual patients Indicates cancer subtype labels (such as hypopharyngeal, laryngeal, oropharyngeal subtypes of head and neck cancer).
[0027] Step 2: Train the acquired multimodal medical data and the constructed cancer subtyping model to obtain the trained cancer subtyping model; Cancer subtyping models specifically include the following steps: Step 201: Encode the multimodal medical data and map it to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; like Figure 2 As shown, multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality, specifically including: Step 2011: Encode the clinical phenotype data to obtain clinical feature representations; Specifically, for clinical data that includes continuous variables (such as age and tumor volume) and discrete variables (such as gender and TNM stage) First, continuous variables are Z-score standardized, and discrete variables are one-hot encoded. Then, these are input into an MLP network with three hidden layers, which outputs clinical feature representations. , represented as: , , , in, This represents the output feature of the first hidden layer; This represents the output feature of the second hidden layer; This represents the preprocessed input clinical feature vector; This represents the final output of clinical features; , , These represent the weight matrices of the first, second, and third fully connected layers, respectively. , , These represent the bias terms for the first, second, and third fully connected layers, respectively. Represents the linear rectification activation function; This indicates batch normalization operation.
[0028] Step 2012: Encode the pathological text data to obtain text feature representations; Specifically, the BioBERT model, pre-trained in the medical field, is used. This involves processing pathology reports. WordPiece tokenization into a sequence After inputting the sequence into BioBERT, the output of the last layer, which contains [CLS] tags with global contextual semantics, is extracted as the initial text features. Subsequently, the initial text features are reduced in dimensionality using a linear mapping layer to obtain the text feature representation. , represented as: , in, This represents the final text feature representation after dimensionality reduction mapping. This represents the weight matrix of the linear mapping layer used to reduce the dimensionality of the initial features of the text; This represents the initial text features extracted from the BioBERT model; [CLS] represents the bias term of the linear mapping layer; [CLS] represents the global classification token added to the beginning of the input text sequence, and its corresponding output vector is used to represent the global contextual semantics of the entire pathological text. [SEP] represents the first and second sub-word tokens obtained sequentially after WordPiece segmentation of the pathology report text; [SEP] represents a special separator token added to the end of the text sequence to mark the end of the sequence.
[0029] Step 2013: Encode the tissue image data to obtain image feature representation; Specifically, for mIHC or pathological slide images In this embodiment, a pre-trained ResNet50 is used as the backbone network. The original fully connected classification head of ResNet50 is removed, and the feature map output by the last convolutional layer is retained. Subsequently, a global average pooling (GAP) operation is applied to extract the space not... Variable characteristics : , Finally, the dimensions are aligned using a linear mapping:
[0030] Output image feature representation ; in, This indicates the input raw mIHC or pathological slide image data; This indicates the number of channels in the input image data. This indicates the height of the input image data, i.e., the number of pixel rows; This indicates the width of the input image data, i.e., the number of pixel columns; This represents the feature map output by the last convolutional layer of the pre-trained network; This indicates the feature map Height; This indicates the feature map The width; Representation of feature map In spatial coordinates The feature vector along all channel dimensions; This represents the spatially invariant feature vector extracted after applying the Global Average Pooling (GAP) operation.
[0031] Step 202: Introduce a multi-head attention mechanism. For each attention head in the multi-head attention mechanism, construct an independent mapping space for each modality. Concatenate the outputs of multiple attention heads along the feature dimension and obtain a multimodal fusion representation that aggregates cross-modal complementary information through linear transformation. like Figure 3 As shown, this module aims to break down the barriers of traditional splicing and fusion, allowing the model to adaptively learn the weight allocation between different modalities. In order to fully preserve the unique heterogeneous semantics of the three modalities, this invention does not simply splice the original features, but instead constructs an independent mapping space for each modality.
[0032] For the Each attention head introduces an independent, learnable parameter matrix for clinical, text, and image representations. Text representation Image representation Multiply by their respective mapping matrices to generate the query (Q), key (K), and value (V) matrices for the corresponding modality: Clinical modality: , Text modality: , Image modality: , Subsequently, the query, key, and value matrices corresponding to these three modalities are concatenated along the sequence dimension to construct a global multimodal query matrix. Global key matrix and global value matrix .
[0033] like Figure 3 As shown in the middle module, the concatenated global query matrix will be... With the transposed global key matrix Perform matrix multiplication (MatMul) and divide by the scaling factor. (This is to prevent the softmax gradient from saturating due to excessively large inner product values). A mode absence mask matrix is then introduced. Attention weights are obtained through Softmax normalization, and finally, the global value matrix is processed. Weighting is applied.
[0034] Calculate the first Output of each attention head: , When a certain modality of data is missing, the mask matrix... Set the corresponding position to Set to when data is complete .
[0035] Will The outputs of each independent attention head are concatenated again along the feature dimension, and then output through a linear mapping layer to form a multimodal fusion representation matrix that aggregates cross-modal complementary information. :
[0036] Finally, Mean pooling or flattening along the modal dimension yields the final input to the one-dimensional fusion vector of the variational bottleneck. .
[0037] in, Indicates the index number of the attention head currently being computed; These represent the encoded clinical feature representation, text feature representation, and image feature representation, respectively. These represent the learnable mapping weight parameters specific to the clinical modality used to generate the query matrix, key matrix, and value matrix, respectively. These represent the learnable mapping weight parameters specific to the text modality used to generate the query matrix, key matrix, and value matrix, respectively. These represent the learnable mapping weight parameters specific to the image modality used to generate the query matrix, key matrix, and value matrix, respectively. These represent the query matrix, key matrix, and value matrix generated from clinical features, respectively. These represent the query matrix, key matrix, and value matrix generated from text features, respectively. These represent the query matrix, key matrix, and value matrix generated from image features, respectively. These represent the global query matrix, global key matrix, and global value matrix obtained by concatenating the corresponding matrices of the three modalities along the sequence dimension, respectively. This represents the transpose of the global key matrix. The dimension of the key matrix is represented by its size. It is used as a scaling factor to prevent gradient saturation of the Softmax function; Represents the modality missing mask matrix; Indicates the first Local feature output after weighted calculation of each attention head; This indicates that when a mode is missing, a negative infinity value is assigned to the mask matrix to mask the mode weights in the Softmax operation; The default mask value when the data is complete.
[0038] Step 203: Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation and sample to obtain latent variables; In this embodiment, the latent posterior distribution is inferred through two parallel fully connected layers. mean vector With log variance , represented as: , , in, Indicates that given input data Under the condition of , the inferred latent posterior probability distribution; The vector representing the mean of the latent posterior distribution; Represents the log-variance vector of the latent posterior distribution; This represents a one-dimensional multimodal fusion feature vector that, after aggregation by the cross-modal attention module, is input to the variational bottleneck. and These represent the vectors used to infer the mean. The weight matrix and bias terms of the fully connected layer; and These represent the vectors used to infer the logarithmic variance. The weight matrix and bias terms of the fully connected layer; To ensure the differentiability of the model parameters during backpropagation, auxiliary noise is sampled from a standard normal distribution. Generate latent variables : , in, This represents the continuous latent variables finally generated after reparameterized sampling; Indicates the distribution from the standard normal distribution An auxiliary noise vector is used in the sampling process to ensure the differentiability of the sampling process; Represented by natural constant An exponential function with base 0. Used to calculate standard deviation ; The Hadamard product is an element-wise multiplication operation of a matrix or vector.
[0039] Step 204: Based on the latent variables and classifier, obtain the cancer subtyping results using Softmax; In this embodiment, a fully connected network is used as the classifier. Specifically, the classifier consists of one or more densely connected layers to classify the dimensionality-reduced latent variables. The data is mapped to a dimensional space with the same number of cancer subtypes, and then passed through a Softmax activation function to output the predicted probability that the patient belongs to each cancer subtype.
[0040] In this embodiment, a multi-task joint optimization method is used to train the cancer subtyping model, specifically including: This embodiment abandons the KL divergence, which is prone to "posterior collapse," and innovatively adopts the MMD function based on the reproducing kernel Hilbert space (RKHS). MMD can measure the inferred latent distribution sample set. Compared with standard normal prior Sample set of samples from the middle Distance between ( (for batch size), by minimizing MMD loss Forced latent variables The distribution is smoothly aligned with the prior distribution; The MMD loss function is expressed as: , in, This represents the calculated maximum mean difference loss value; This indicates the batch size during network training, which is the number of samples used in each computation. This represents the set of latent variable samples drawn from the inferred latent posterior distribution; Indicates the standard normal prior distribution The prior sample set of the sampled data; Representing the latent variable sample sets respectively The first in The and the first One sample; They represent the prior sample sets respectively. The first in The and the first One sample; The kernel function in the reproducing kernel Hilbert space (RKHS) is used to calculate the inner product (similarity) of two sample vectors in the feature space. In this embodiment, the multi-scale Gaussian radial basis (RBF) kernel function is specifically used.
[0041] To enhance the model's robustness to "modal loss" in real-world clinical scenarios, the system introduces a masked reconstruction mechanism during the training phase. In each training batch, according to the Bernoulli distribution Randomly select one or more input modalities (such as...) or ) Mask (set to zero).
[0042] Utilizing latent variables The original data is attempted to be recovered by reverse mapping using symmetric decoders. , Only calculate the reconstruction loss of the unmasked (effective observation) modes. (The mean square error (MSE) is used for continuous data, and the cross-entropy (CE) is used for discrete data.)
[0043] latent variables Input to the downstream classifier, and output the predicted probability via Softmax. Calculate the classification cross-entropy loss .
[0044] Therefore, the overall objective function is: , in, This is a hyperparameter that balances the weights of the discrimination and generation tasks. The model uses this joint loss function to achieve end-to-end backpropagation parameter updates.
[0045] Step 3: Based on the trained cancer subtyping model, predict the cancer subtyping results from the medical data of the target object.
[0046] Step 4: Using the deep differential propagation algorithm, calculate the contribution of each input feature in the multimodal medical data relative to the reference baseline, and realize the attribution analysis from macro-modality to micro-biomarkers.
[0047] Specifically, the steps include the following: Set a reference baseline: Select a neutral input that does not have a predictive bias. ; Differential signal forward computation: Calculate the actual patient input Compared with baseline The resulting final network activation difference : ,in, Represents real patient data The actual activation value obtained at the target output neuron after being input into the cancer subtyping model; Indicates reference baseline data The baseline activation value obtained at the target output neuron after being input into the cancer subtyping model; Network activation difference The total differential signal, which serves as the feature attribution, is distributed losslessly from top to bottom to each layer of the network in the subsequent multiplier backpropagation step, according to the Summation-to-Delta theorem, until it is finally distributed to each input feature, thereby calculating the contribution score of each input feature to the final prediction result.
[0048] Among them, multiplier backpropagation: according to the chain rule of DeepLIFT, the interlayer neurons are defined. Target neuron Multipliers: , in, This represents the input features (or interlayer neurons). For the target output neuron Contribution score; Representation of features The difference between the actual activation value and its reference baseline activation value; The calculated multipliers The "gradient substitution" used as a feature attribution factor is employed for backpropagation. Through the chain rule, this multiplier can distribute the confidence of the activation difference of the top layer of the network layer by layer down until the micro-contribution of each specific feature (such as the expression level of a specific gene, image pixels, etc.) in the original input data is calculated, thereby achieving highly interpretable biomarker attribution.
[0049] Figure 4This is a bar chart comparing the classification performance of the method of this invention with existing baseline models on real medical cohorts; it provides a comprehensive performance comparison with current mainstream baseline models (including early / late fusion methods, machine learning classifiers, and ensemble learning methods). Evaluation metrics include accuracy, balanced accuracy (Balanced ACC), precision, recall, macro-average F1 score (F1), area under the receiver operating characteristic curve (AUC.ROC), and area under the precision-recall curve (AUC.PR).
[0050] Contribution quantification (using the Hancock head and neck cancer cohort as an example): By calculating the mean absolute attribution value of all samples, the model outputs the macroscopic contribution of each modality (e.g., Figure 5 (As shown in the pie chart). Furthermore, at the microscopic level, the system accurately calculates the attribution scores for specific features at the input. For example, in subtype classifications such as hypopharyngeal carcinoma, the system can pinpoint the relative contribution of specific TNM staging parameters, high-frequency risk terms in pathology reports (such as "invasive" and "keratosis"), or specific highlight areas in tissue images to the classification, thus providing intuitive auxiliary interpretations.
[0051] Example 2 This embodiment provides a cancer subtyping system based on cross-modal attention and variational representation learning, including: The data acquisition module is used to acquire patients' multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data. The model building module is used to train the cancer subtyping model based on the acquired multimodal medical data and the constructed cancer subtyping model, resulting in a trained cancer subtyping model. The construction process of the cancer subtyping model includes: Multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; A multi-head attention mechanism is introduced. For each attention head in the multi-head attention mechanism, an independent mapping space is constructed for each modality. This includes: generating a corresponding query matrix, key matrix, and value matrix based on the feature representation of each modality, and concatenating the features along the sequence dimension to obtain a global multimodal query matrix, key matrix, and value matrix; weighting the global value matrix with the attention weights obtained from the global multimodal query matrix and key matrix to obtain the output of each attention head; concatenating the outputs of multiple attention heads along the feature dimension, and performing a linear transformation to obtain a multimodal fusion representation that aggregates cross-modal complementary information. Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation, and sample to obtain latent variables; Based on latent variables and classifiers, cancer subtyping results are obtained through Softmax. The cancer typing prediction module is used to predict cancer typing results based on the medical data of the target object using the trained cancer typing model.
[0052] It should be noted that the specific implementation of the cancer subtyping system based on cross-modal attention and variational representation learning in this embodiment of the invention is similar to the specific implementation of the cancer subtyping method based on cross-modal attention and variational representation learning in this embodiment of the invention. For details, please refer to the description in the method section. To reduce redundancy, it will not be repeated here.
[0053] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the cancer subtyping method based on cross-modal attention and variational representation learning as described above.
[0054] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the cancer subtyping method based on cross-modal attention and variational representation learning as described above.
[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0059] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cancer subtyping method based on cross-modal attention and variational representation learning, characterized in that, Includes the following steps: Acquire patients' multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data; The acquired multimodal medical data and the constructed cancer subtyping model are used for training to obtain the trained cancer subtyping model; the construction process of the cancer subtyping model includes: Multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; A multi-head attention mechanism is introduced. For each attention head in the multi-head attention mechanism, an independent mapping space is constructed for each modality. This includes: generating a corresponding query matrix, key matrix, and value matrix based on the feature representation of each modality, and concatenating the features along the sequence dimension to obtain a global multi-modal query matrix, key matrix, and value matrix; and weighting the global value matrix with the attention weights obtained from the global multi-modal query matrix and key matrix to obtain the output of each attention head. The outputs of multiple attention heads are concatenated along the feature dimension and then linearly transformed to obtain a multimodal fusion representation that aggregates cross-modal complementary information. Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation, and sample to obtain latent variables; Based on latent variables and classifiers, cancer subtyping results are obtained through Softmax. The cancer classification results are obtained by predicting the medical data of the target object based on the trained cancer classification model.
2. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, The process of encoding multimodal medical data and mapping it to a unified-dimensional modality-specific representation space to obtain the feature representation of the corresponding modality includes: The clinical phenotypic data is encoded, including Z-score standardization of continuous variables and one-hot encoding of discrete variables. The data is then fed into an MLP network with three hidden layers to output clinical feature representations. Encoding pathological text data includes: converting pathological text data into a sequence using WordPiece tokenization, inputting the sequence into BioBERT, extracting the output of the [CLS] tag with global context semantics at the last layer as the initial text feature, and then reducing the dimensionality of the initial text feature through a linear mapping layer to obtain the text feature representation; The tissue image data is encoded by: using a pre-trained ResNet50 as the backbone network, removing the original fully connected classification head of ResNet50, retaining the feature map output by the last convolutional layer, then applying global average pooling to extract spatially invariant features, and finally aligning the dimensions through linear mapping to output the image feature representation.
3. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, The attention weights obtained through the global multimodal query matrix and key matrix are used to weight the global value matrix to obtain the output of each attention head, including: The global query matrix is multiplied by the transposed global key matrix and divided by a scaling factor to introduce a modality missing mask matrix. Attention weights are obtained by Softmax normalization. Finally, the global value matrix is weighted to obtain the output of each attention head.
4. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, The statistical parameters representing the latent posterior distribution in the multimodal fusion include the mean vector. With log variance , represented as: ， ， in, The vector representing the mean of the latent posterior distribution; Represents the log-variance vector of the latent posterior distribution; This represents a one-dimensional multimodal fusion feature vector that, after aggregation by the cross-modal attention module, is input to the variational bottleneck. and These represent the vectors used to infer the mean. The weight matrix and bias terms of the fully connected layer; and These represent the vectors used to infer the logarithmic variance. The weight matrix and bias terms of the fully connected layer.
5. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, When training a cancer subtyping model using a multi-task joint optimization method, a random modality mask is introduced, and the model is trained together using MMD distribution regularization loss, reconstruction consistency loss, and downstream classification cross-entropy loss. The MMD distribution regularization loss is constructed by calculating the distance between the latent distribution sample set and the sample set sampled from the standard normal prior. The reconstruction consistency loss works by randomly masking one or more input modalities according to a Bernoulli distribution in each training batch, and then using latent variables to inversely map the original data through a symmetric decoder to reconstruct the original data structure. The classification cross-entropy loss is constructed based on latent variables and classifiers, and the cancer typing results are obtained through Softmax.
6. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, The loss function for training the cancer subtyping model is: ， in, Represents the overall objective loss function. Represents the classification cross-entropy loss. For hyperparameters, This indicates that only the reconstruction loss of the unmasked modes is calculated. This represents the regularity loss of the MMD distribution.
7. The cancer subtyping method based on cross-modal attention and variational representation learning as described in claim 1, characterized in that, The method also includes using a deep differential propagation algorithm to calculate the contribution of each input feature in multimodal medical data relative to a reference baseline, thereby achieving attribution analysis from macro-modal to micro-biomarker.
8. A cancer subtyping system based on cross-modal attention and variational representation learning, characterized in that, include: The data acquisition module is used to acquire patients' multimodal medical data, which includes at least clinical phenotype data, pathological text data, and tissue imaging data. The model building module is used to train the cancer subtyping model based on the acquired multimodal medical data and the constructed cancer subtyping model, resulting in a trained cancer subtyping model. The construction process of the cancer subtyping model includes: Multimodal medical data is encoded and mapped to a modality-specific representation space of a unified dimension to obtain the feature representation of the corresponding modality; A multi-head attention mechanism is introduced. For each attention head in the multi-head attention mechanism, an independent mapping space is constructed for each modality. This includes: generating a corresponding query matrix, key matrix, and value matrix based on the feature representation of each modality, and concatenating the features along the sequence dimension to obtain a global multimodal query matrix, key matrix, and value matrix; weighting the global value matrix with the attention weights obtained from the global multimodal query matrix and key matrix to obtain the output of each attention head; concatenating the outputs of multiple attention heads along the feature dimension, and performing a linear transformation to obtain a multimodal fusion representation that aggregates cross-modal complementary information. Infer the statistical parameters of the latent posterior distribution of the multimodal fusion representation, and sample to obtain latent variables; Based on latent variables and classifiers, cancer subtyping results are obtained through Softmax. The cancer typing prediction module is used to predict cancer typing results based on the medical data of the target object using the trained cancer typing model.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the cancer subtyping method based on cross-modal attention and variational representation learning as described in any one of claims 1-7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the cancer subtyping method based on cross-modal attention and variational representation learning as described in any one of claims 1-7.