Lightweight multi-mode tumor prognosis evaluation method based on digital pathology technology

Abstract

The invention provides a lightweight multi-modal tumor prognosis evaluation method based on a digital pathology technology, and relates to the technical field of simulation or simulation of medical disordered diseases, and the method comprises the following steps: executing feature determination based on an SDL module, and forming a stable dense layer by using linear transformation and Dropout regularization; performing synchronous processing on the WSI and the genome features to form a multi-information transmission module; generating two linear layers, and respectively executing affine transformation on the WSI and genome data to obtain a double-affine mapping module; constructing a multi-modal mutual information mixer based on a stable dense layer, a multi-information transmission module and a double affine mapping module; on the basis of the multi-modal mutual information mixer, lightweight multi-modal tumor prognosis evaluation is performed. The problem that in the prior art, redundant parameters are too many, efficiency is low, and the clinical environment where the method can be applied is seriously limited is solved.

Description

Technical Field

[0001] The present application relates to the technical field of simulating or emulating medical disordered diseases, and in particular to a lightweight multimodal tumor prognosis assessment method based on digital pathology technology. Background Art

[0002] Survival analysis is a cornerstone of clinical prognosis, providing critical insights for predicting life-threatening events such as death, disease recurrence, or adverse treatment responses. It is a key task in prognostic therapy, enabling clinicians to tailor treatment strategies and optimize patient outcomes. Traditional approaches, relying on short-term clinical indicators and time-consuming longitudinal follow-up, face inherent limitations in scalability and practical application. The advent of deep learning has spurred a paradigm shift, driving the development of multimodal integration, particularly the fusion of whole-slide images (WSIs) and genomic data to leverage complementary prognostic signals. Despite the diagnostic value of each modality, integrating these modalities presents significant challenges: their heterogeneity spans spatial scale (gigapixel WSIs versus high-dimensional genomic vectors), biological resolution (tissue-level phenotypic versus molecular signatures), and clinical interpretation. Further complicating matters, the high computational demands of processing and aligning these diverse data streams often hinder real-time clinical application.

[0003] Existing multimodal survival models primarily rely on attention mechanisms to bridge these gaps. However, existing models prioritize accuracy over practicality and embed cascaded attention modules, resulting in millions of redundant parameters. This computational expansion makes them inefficient and unsuitable for resource-constrained clinical settings.

[0004] Therefore, there is an urgent need for a lightweight, accurate model that can adapt to more clinical scenarios. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a lightweight multimodal tumor prognosis assessment method based on digital pathology technology, aiming to solve the problem of excessive redundant parameters and low efficiency in the existing technology, which leads to severe limitations in the clinical environment in which it can be applied.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] A lightweight multimodal tumor prognostic assessment method based on digital pathology technology, including:

[0008] Perform feature determination based on the SDL module and use linear transformation and Dropout regularization to form a stable dense layer;

[0009] Perform synchronous processing on WSI and genomic features to form a multi-information transmission module;

[0010] Generate two linear layers to perform affine transformation on WSI and genomic data respectively to obtain a dual affine mapping module;

[0011] Based on the stable dense layer, multi-information transmission module, and dual affine mapping module, a multimodal mutual information mixer is constructed;

[0012] Based on multimodal mutual information mixer to perform lightweight multimodal tumor prognosis assessment.

[0013] In some embodiments, the method of using linear transformation and Dropout regularization further includes: aligning the extracted representations of genomic features and WSI features and performing the transformation.

[0014] In some embodiments, the stable dense layer is further configured with at least one linear layer and at least one extended exponential linear unit activation function.

[0015] In some embodiments, the stable dense layer performs self-normalization via Alpha Dropout and SELU.

[0016] In some embodiments, the extraction of the aligned genomic features and WSI represents the execution of a transformation, specifically: based on the φ transformation, copying and splitting the genomic data, passing through a linear layer, a GELU activation function, and a dropout layer, performing matrix alignment on the genomic data and the WSI features.

[0017] In some embodiments, the synchronous processing of WSI and genomic features to form a multi-information transmission module specifically includes:

[0018] Based on the global channel pooling submodule, it captures the global representation;

[0019] Perform integration based on linear layers with Sigmoid activation and generate modality-specific weights;

[0020] Perform element-wise multiplication based on modality-specific weights to perform cross-modality calibration and optimization;

[0021] The features of the optimized modality are transformed based on a linear layer deployed with GELU activation.

[0022] In some embodiments, generating two linear layers to perform affine transformation on the WSI and genomic data respectively to obtain a dual affine mapping module specifically includes:

[0023] Perform feature fusion of two modalities through element-by-element Boolean operations;

[0024] Divide the feature matrix into head and tail;

[0025] Perform an affine transformation.

[0026] In some embodiments, the multimodal mutual information mixer is used to perform lightweight multimodal tumor prognosis assessment, specifically comprising:

[0027] Perform lightweight feature extraction based on stable dense layers;

[0028] Based on the multi-information transmission module, cross-modal interaction is performed through global channel pooling, normalization and transformation capture;

[0029] Based on the dual affine mapping module, feature optimization, feature alignment, and feature fusion are performed based on affine transformation;

[0030] The fused features are input into the classifier to perform risk level prediction.

[0031] According to the above technical features, the beneficial effects of the present application are as follows: the present application optimizes the integration and interaction of multimodal data by providing a multimodal mutual mixer (MMM) to ensure actual efficiency. A multi-information transfer module (MiT) is introduced, and by dynamically generating feature weights from the global information of the modality, cross-modal alignment is made more effective without unnecessary complexity. In order to further enhance the feature interaction between WSIs and genomic data, the present application also combines a dual affine mapping module (DAM), which uses a dual affine transformation to extract modality-specific features, decomposes each modality into two complementary components (i.e., head and tail), and uses parameters from another modality to adjust the features of one modality. The lightweight multimodal tumor prognosis evaluation method based on digital pathology technology provided in the present application can achieve the current most advanced (SOTA) performance in terms of overall scoring, while the required number of parameters is reduced by 12 times compared with the SOTA of the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 The process for this application;

[0034] Figure 2 This is a schematic diagram of the principles of this application;

[0035] Figure 3 This is a schematic table of C-index results of different models on three TCGA datasets in the examples of this application;

[0036] Figure 4 This is a table comparing the total parameters, training time, and inference time per epoch for different models and datasets in the embodiments of this application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] Example

[0039] Please refer to Figures 1-4 , an embodiment of the present application provides a lightweight multimodal tumor prognosis assessment method based on digital pathology technology, the method comprising:

[0040] Perform feature determination based on the SDL module and use linear transformation and Dropout regularization to form a stable dense layer;

[0041] Perform synchronous processing on WSI and genomic features to form a multi-information transmission module;

[0042] Generate two linear layers to perform affine transformation on WSI and genomic data respectively to obtain a dual affine mapping module;

[0043] Based on the stable dense layer, multi-information transmission module, and dual affine mapping module, a multimodal mutual information mixer is constructed;

[0044] Based on multimodal mutual information mixer to perform lightweight multimodal tumor prognosis assessment.

[0045] It is worth noting that there are problems of heterogeneity and computational inefficiency in the prior art. The present application provides a multimodal mutual mixer (i.e., Multi-modal Mutual Mixer, MMM for short), which provides a new framework for redefining cross-modal interaction, and optimizes and improves the performance of traditional models through efficient fusion of layers and parameters. The multimodal mutual mixer (MMM) integrates whole slice images (WSI) and genomic data through two parallel neural network branches. In order to achieve robust multimodal representation learning, the multimodal mutual mixer (MMM) includes a stable dense layer (i.e., Stable Dense Layer, SDL for short), a multi-information transmission module (i.e., Multi-information Transformation, MiT for short), and a dual affine mapping module (i.e., Dual Affine Mapper, DAM for short). Specifically, the Multi-Information Transfer (MiT) module implements a multi-information transfer mechanism. By contextualizing cross-modal dependencies at the spatial and biological levels, MiT dynamically recalibrates modality-specific feature weights. Unlike traditional attention mechanisms that compute rigid global alignment, MiT employs adaptive gating to suppress redundant activations while enhancing collaborative signals, thereby ensuring precise feature alignment without sacrificing computational efficiency. Building on MiT, the Dual Affine Mapping (DAM) module provides a novel feature fusion approach through a two-stream design. Each modality is decomposed into complementary "head" and "tail" components, which interact through element-wise affine transformations. This two-stream architecture enables bidirectional regulation, where one modality repeatedly optimizes the features and parameters of the other. The multimodal mutual mixer (MMM) constructed in this embodiment achieves a balance between computational efficiency and prognostic accuracy, overcomes the trade-offs faced by existing models, and provides a real-time multimodal integration solution that is applicable to various resource environments and clinically feasible. It solves the problem in existing technologies of excessive redundant parameters and low efficiency, which severely limits the clinical environments in which they can be applied.

[0046] In some embodiments, the method of using linear transformation and Dropout regularization further includes: aligning the extracted representations of genomic features and WSI features and performing the transformation.

[0047] In some embodiments, the stable dense layer is further configured with at least one linear layer and at least one extended exponential linear unit activation function. Specifically, the stable dense layer (SDL) is configured with at least one lightweight feature extractor. The extended exponential linear unit activation function can be SELU, which is defined as:

[0048] SELU(x)=scale*(max(0,x)+min(0,α*(exp(x)-1)))

[0049] In some embodiments, the stable dense layer performs self-normalization through Alpha Dropout and SELU. In detail, in this embodiment, Alpha Dropout and SELU are used together to maintain the self-normalization property. These modules project the two modalities to the same characteristic shape, and the formula used by the stable dense layer (SDL) is:

[0050]

[0051] Wherein, Wwsi and Womic are learnable weights. The stable dense layer (SDL) configured in this embodiment can effectively stabilize the mean and variance, thereby reducing the risk of overfitting and enhancing the robustness of the representation.

[0052] In some embodiments, the extraction of the aligned genomic features and WSI represents the execution of a transformation, specifically: based on the φ transformation, copying and splitting the genomic data, passing through a linear layer, a GELU activation function, and a dropout layer, performing matrix alignment on the genomic data and the WSI features.

[0053] In this example, to address the imbalance of genomic data, a φ transformation is introduced and aligned with the WSI feature matrix. This involves replicating and splitting the genomic data to match the WSI dimension, followed by a linear layer, a GELU activation function, and a dropout layer. The transformation φ is defined as follows:

[0054]

[0055] Here, S represents the number of features. Conventional methods used in the prior art typically concatenate or pad genomic data, using it as auxiliary information in the network. In contrast, the φ used in this embodiment leverages repetition to maintain information integrity while enhancing permutation invariance, ensuring that it is represented at the same level as the WSI. It can be understood that φ is used to expand genomic information.

[0056] In some embodiments, the synchronous processing of WSI and genomic features to form a multi-information transmission module specifically includes:

[0057] Based on the global channel pooling submodule, it captures the global representation;

[0058] Perform integration based on linear layers with Sigmoid activation and generate modality-specific weights;

[0059] Perform element-wise multiplication based on modality-specific weights to perform cross-modality calibration and optimization;

[0060] The features of the optimized modality are transformed based on a linear layer deployed with GELU activation.

[0061] In detail,

[0062] In order to overcome the limitations of the framework in capturing global information, the embodiment of the present application is configured with a multi-information transmission module (MiT), which is a module for global interaction between two modalities. This module processes WSI and genomic features in parallel, first through a global channel pooling submodule (Global channel pooling, GCP), which captures global representations, integrates them through a linear layer with Sigmoid activation, and generates weights for a specific modality. These weights are used in a cross-modal refinement mechanism, which recalibrates one modality with another by element-by-element multiplication. The refined features are then transformed through a linear layer with GELU activation. Among them, the residual connection ensures that the model retains important low-level information. The multi-information transmission module (MiT) can be formally represented as follows:

[0063]

[0064] Among them, the weight is generated by GCP, Representation connection operation. This embodiment enables the Multi-Information Transfer module (MiT) to effectively integrate information from multiple sources, allowing the model to selectively retain the most critical features. The mechanism provided by this embodiment ensures that modal fusion is both efficient and effective, optimizing robust and task-relevant representation learning.

[0065] In some embodiments, generating two linear layers to perform affine transformation on the WSI and genomic data respectively to obtain a dual affine mapping module specifically includes:

[0066] Perform feature fusion of two modalities through element-by-element Boolean operations;

[0067] Divide the feature matrix into head and tail;

[0068] Perform an affine transformation.

[0069] It should be noted that this embodiment configures a dual affine mapping module (DAM), which uses two independent linear layers to learn modality-specific affine transformations for WSI and genomic data, respectively. After applying these transformations, the module combines the features of the two modalities by element-by-element multiplication and addition, effectively fusing complementary information. This approach creates a rich feature representation and improves the performance of the model in multimodal tasks. The independent affine transformation of each modality helps the model capture modality-specific features and interaction patterns. The dual affine mapping module (DAM) can be described as follows:

[0070]

[0071] Mix wsi =(h wsi *t omic )+(h omic +t wsi )

[0072] Mix omic =(h omic *t wsi )+(h wsi +tomic)

[0073] The split(·,2) operation divides the feature matrix into two parts (i.e., head and tail). After affine transformation, these features are input into the Multi-Information Transfer module (MiT), which enables deeper cross-modal learning by integrating modality-specific transformations.

[0074] In some embodiments, the multimodal mutual information mixer is used to perform lightweight multimodal tumor prognosis assessment, specifically comprising:

[0075] Perform lightweight feature extraction based on stable dense layers;

[0076] Based on the multi-information transmission module, cross-modal interaction is performed through global channel pooling, normalization and transformation capture;

[0077] Based on the dual affine mapping module, feature optimization, feature alignment, and feature fusion are performed based on affine transformation;

[0078] The fused features are input into the classifier to perform risk level prediction.

[0079] It should be noted that the lightweight multimodal tumor prognosis assessment method based on digital pathology technology provided in this embodiment constructs a multimodal mutual assistance mixer (MMM). The architecture of the multimodal mutual assistance mixer (MMM) is a lightweight and efficient multimodal survival prediction architecture that systematically integrates feature extraction, cross-modal fusion and predictive modeling, and aims at deployment efficiency, providing a simplified and robust model architecture for clinical decision-making. It is also a risk stratification framework that is more suitable for clinical hardware in hospital environments, thereby enabling it to become a feasible and scalable solution in real applications. In addition, the multimodal mutual assistance mixer (MMM) also overcomes the limitations of non-Transformer architectures in multimodal tasks. In addition, the lightweight multimodal tumor prognosis assessment method based on digital pathology technology provided in this embodiment has been evaluated on three datasets (LUAD, BLCA, UCEC), and the evaluation results obtained can reach the level of the most advanced technology (SOTA).

[0080] In detail, this embodiment provides a multimodal mutual mixer (MMM) and its framework for survival analysis, which uses WSI and genomic data as input. In order to enhance the interaction of contextual information, a multi-information transmission module (MiT) and its mechanism are configured, and in order to improve the exchange of information between the two modalities, a dual affine mapping module (DAM) is configured, which applies the scaling and offset parameters of one modality to the other modality. The multimodal mutual mixer (MMM) and its framework provided by this embodiment not only improves the accuracy, but also significantly reduces the total number of parameters compared with other models, and can effectively shorten the inference time, with greater practical application prospects.

[0081] It should be noted that Whole Slide Images (WSIs) are usually too large to be directly processed by standard computing systems. To solve this problem, WSIs are usually divided into tiles of uniform size and features are extracted from these tiles using pre-trained neural networks. These tiles are represented as where n p is the number of tiles, and d is the dimension of the features extracted from WSIs. Genomic data, including various genetic information such as age, gender, RNA, DNA, CNV, etc., are processed through a fully connected layer to generate feature vectors. These genomic features are represented as where n g =6 represents the number of genomic features. In this embodiment, each input data pair is represented as X i =(P i ,G i ,c i ,t i ),in:

[0082] Pi :WSI tiles, representing features extracted from whole-slice images;

[0083] Gi: genomic feature vector, capturing gene expression or mutation data;

[0084] c i ∈[0,1]: right censoring state, where 1 represents uncensored data and 0 represents censored data;

[0085] t i ∈R + : Overall survival time (in months).

[0086] Specifically, given an input pair, we estimate the hazard function h(t) = h(T = t | T ≥ t, (P, G)) ∈ [0, 1], which represents the probability that the patient dies immediately after time point $t$. Then, by negating and integrating the hazard function, we obtain an ordinal value s(t | (P, G)) = Πu = 1 t (1-h(u)).

[0087] The specific experiments of this embodiment are:

[0088] Based on The Cancer Genome Atlas (TCGA), this example provides comprehensive patient data including whole slide images (WSI), genomic data, and actual survival time. The dataset used in this experiment includes 373 cases of bladder urothelial carcinoma (BLCA), 480 cases of endometrial carcinoma (UCEC), and 453 cases of lung adenocarcinoma (LUAD). Figure 3 As shown, the red bold font indicates the highest score.

[0089] Furthermore, in terms of hardware and software configuration, the Multimodal Mutual Mixer (MMM) is implemented on PyTorch2.1.2+cu118 and Python 3.10.15, the training environment is Ubuntu 22.04.4LTS, and four RTX 4090 GPUs are used.

[0090] Specifically, in this experiment, the Adamw optimizer was used with a learning rate set to 5×$10$^{-4} and weight decay configured to 1×$10$^{-4} and 1×$10$^{-5}, respectively. The WSI was segmented into 512×512 pixel patches and magnified by 40x. For feature extraction, a ResNet-50 pre-trained on ImageNet with an output dimension of 1024 was used.

[0091] Furthermore, the concordance index (c-index) is a commonly used metric to evaluate the performance of survival analysis models. It quantifies the model's ability to accurately rank pairs of individuals based on their predicted survival times. The mathematical definition of the c-index is as follows:

[0092]

[0093] Where n represents the number of cases, T i and T j Represents the survival time of the i-th patient and the j-th patient respectively. I(*) is an indicator function, which takes the value 1 when the condition is true and 0 otherwise. c j is the right-censored state. The c-index ranges from 0 to 1, with higher values, closer to 1, indicating a more accurate model.

[0094] Furthermore, this experiment analyzed Figure 4 The table shows the efficiency and scalability of the models, where M stands for million and s stands for seconds. Bold green indicates the smallest value, and bold blue indicates the second smallest value. Important information is the parameters, training time, and inference time per epoch for the UCEC, LUAD, and BLCA datasets. The Multimodal Mutual Mixer (MMM) achieves the highest efficiency, requiring 12.5 times fewer parameters than MoME and training 2.9 times faster, reducing training time from 95.985 seconds to 33.145 seconds (UCEC). Inference speed is also improved by 1.9 times, reducing inference time from 13.437 seconds to 6.864 seconds. While MoME and CMTA may perform better in some situations, their high computational cost limits scalability. In contrast, the Multimodal Mutual Mixer (MMM) balances efficiency and scalability, making it an ideal choice for resource-limited environments.

[0095] It is worth noting that this embodiment provides a multimodal framework that surpasses the existing state-of-the-art methods on the BLCA and LUAD datasets while using significantly fewer parameters. The results show the excellent performance of the architecture provided by this embodiment in terms of overall scores, especially in the BLCA dataset, where the architecture improves the overall performance of the c-index measurement by 3.9% and 0.8%, respectively. Moreover, thanks to its simple architecture, the Multimodal Mutual Mixer (MMM) can be easily adapted to add or replace other modalities. These results highlight the powerful generalization ability of the Multimodal Mutual Mixer (MMM) in multimodal survival prediction tasks. Importantly, the execution process of this architecture is efficient and can be deployed on hardware with limited GPU memory. Compared with existing technologies, it has stronger practicality and applicability in real-world applications.

[0096] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and range of equivalents of the claims are included in the present application. Any figure mark in the claims should not be construed as limiting the claim to which it relates.

[0097] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A lightweight multimodal tumor prognosis assessment method based on digital pathology technology, characterized in that: include: Perform feature determination based on the SDL module and use linear transformation and Dropout regularization to form a stable dense layer; Perform synchronous processing on WSI and genomic features to form a multi-information transmission module; Generate two linear layers to perform affine transformation on WSI and genomic data respectively to obtain a dual affine mapping module; Based on the stable dense layer, multi-information transmission module, and dual affine mapping module, a multimodal mutual information mixer is constructed; Based on multimodal mutual information mixer to perform lightweight multimodal tumor prognosis assessment.

2. The evaluation method according to claim 1, wherein: The use of linear transformation and Dropout regularization also includes: aligning the extracted representation of genomic features and WSI features to perform transformation.

3. The evaluation method according to claim 2, wherein: The stable dense layer is further configured with at least one linear layer and at least one extended exponential linear unit activation function.

4. The evaluation method according to claim 3, wherein: The stable dense layer performs self-normalization through Alpha Dropout and SELU.

5. The evaluation method according to claim 3, wherein: The extraction of the aligned genomic features and WSI represents the execution of a transformation, specifically: based on the φ transformation, the genomic data is copied and segmented, and the matrix alignment of the genomic data and the WSI features is performed through a linear layer, a GELU activation function, and a dropout layer.

6. The evaluation method according to claim 5, characterized in that The synchronous processing of WSI and genomic features to form a multi-information transmission module specifically includes: Based on the global channel pooling submodule, it captures the global representation; Perform integration based on linear layers with Sigmoid activation and generate modality-specific weights; Perform element-wise multiplication based on modality-specific weights to perform cross-modality calibration and optimization; The features of the optimized modality are transformed based on a linear layer deployed with GELU activation.

7. The evaluation method according to claim 6, characterized in that The two linear layers are generated to perform affine transformation on the WSI and genomic data respectively to obtain a dual affine mapping module, which specifically includes: Perform feature fusion of two modalities through element-by-element Boolean operations; Divide the feature matrix into head and tail; Perform an affine transformation.

8. The evaluation method according to claim 7, characterized in that The multimodal mutual information mixer is used to perform lightweight multimodal tumor prognosis assessment, specifically including: Perform lightweight feature extraction based on stable dense layers; Based on the multi-information transmission module, cross-modal interaction is performed through global channel pooling, normalization and transformation capture; Based on the dual affine mapping module, feature optimization, feature alignment, and feature fusion are performed based on affine transformation; The fused features are input into the classifier to perform risk level prediction.

Images