Cancer prognosis method based on biological pathway and histopathological feature staging fusion strategy

Abstract

The cancer prognosis method based on biological pathway and histopathological feature stage fusion strategy relates to the field of image processing. The method solves the problems of feature redundancy, insufficient modality collaboration and high computational complexity in existing multi-modal integration methods. The method includes: calculating the participation index and gene expression significance index in the gene-pathway correlation matrix according to the gene pathway selection algorithm, and using dynamic weight ordering to screen the key genes closely related to cancer survival; mapping the key genes closely related to cancer survival to the corresponding pathways to form a pathway matrix; constructing a two-stage fusion strategy, including: the first stage of gene pathway and pathological image cross-reconstruction fusion, and the second stage of gene pathway and pathological image bilinear collaborative fusion; dynamically adjusting the weight of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimizing the feature integration process through a nonlinear activation function. The present application significantly improves the fusion efficiency and prediction performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features. Background Technology

[0002] Cancer survival prediction is a crucial area of ​​cancer research, with profound implications for personalized treatment planning and accurate patient prognosis assessment. The complexity and heterogeneity of cancer make accurate prediction and personalized treatment core challenges in cancer management. Traditional clinical assessment methods can provide guidance for patient prognosis to some extent, but with continuous technological advancements, new bioinformatics tools and the integration of multimodal data have significantly improved the accuracy and operability of cancer survival prediction.

[0003] Currently, whole-slide images (WSIs) and genomic data play a crucial role in cancer research, providing key information from both tissue and molecular levels. WSIs, by detailing the histological structure, morphological features, and changes in the tumor microenvironment, help clinicians assess tumor malignancy, grade, and potential metastatic risk. These images are irreplaceable in diagnosis, especially in early tumor detection and prognostic assessment. However, WSIs, by focusing solely on morphological observation, cannot reveal the genetic and molecular mechanisms of tumors, limiting their application in complex biological research.

[0004] In contrast, genomic data, by providing information on gene expression, gene mutations, and copy number variations, offers a deeper understanding of cancer biology research and precision medicine. This data can reveal the molecular mechanisms of tumor development, progression, and metastasis, providing effective molecular targets for clinical treatment. However, genomic data is typically high-dimensional, sparse, and exhibits significant redundancy among features. This makes the processing and analysis of genomic data challenging; efficiently extracting meaningful information, avoiding redundancy, and improving predictive performance are key issues.

[0005] While single-modal data (such as individual WSIs or genomic data) can provide some support for tumor diagnosis and treatment, the multidimensional nature of tumors makes it difficult for single-modal methods to fully reflect their complexity. Therefore, the integration of multimodal data has become an important means to improve the accuracy of cancer prediction. Existing multimodal integration methods typically combine pathological images and genetic data features through simple splicing or shallow interaction. However, these methods often fail to fully capture the deep correlation between the two, resulting in limited improvement in predictive performance. Simple splicing or shallow fusion can only enhance the model's expressive power to a certain extent, but cannot deeply understand the potentially complex relationship between genetic data and tissue images.

[0006] Furthermore, the high dimensionality and sparsity of genomic data often result in a large number of redundant features, which not only increases computational complexity but may also introduce noise, affecting model accuracy. Against this backdrop, effectively selecting features, reducing dimensionality, and integrating information to eliminate redundant features has become a significant challenge in multimodal fusion. The massive volume and high resolution of WSI data make processing these images extremely computationally expensive, especially in practical applications. Ensuring efficiency without losing crucial information is a pressing issue that needs to be addressed. Summary of the Invention

[0007] This invention addresses the problems of feature redundancy, insufficient modal synergy, and high computational complexity in existing multimodal integration methods. It proposes a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features, the method comprising:

[0008] Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight ranking to screen key genes closely related to cancer survival;

[0009] Step S2: Map key genes closely related to cancer survival to their corresponding pathways to form a pathway matrix;

[0010] Step S3: Construct a two-stage fusion strategy, including: the first stage of cross-reconstruction fusion of gene pathways and pathological images, and the second stage of bilinear synergistic fusion of gene pathways and pathological images;

[0011] Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through a nonlinear activation function.

[0012] Furthermore, a preferred embodiment is proposed, wherein step S1 includes:

[0013] Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the degree of participation and activity of the gene in the pathway by summing all elements in the gene row;

[0014] Gene expression significance (ES) was calculated using the DESeq2 method.

[0015] The weighted score of each gene is calculated based on the gene participation index (PI) and expression significance (ES).

[0016] Generate a sorted list W by sorting the genes in descending order of their weighted scores. sorted ;

[0017] Genes in the top p% based on scores are selected, a dynamic threshold is calculated, and the final gene list SL is filtered by the threshold, with the p% value set to 25%.

[0018] Furthermore, a preferred embodiment is proposed, wherein step S2 includes:

[0019] Key genes closely related to cancer survival are mapped to corresponding pathways to form feature representations;

[0020] Low-dimensional pathway embeddings are generated by processing feature representations using a self-normalized neural network.

[0021] The low-dimensional path embedding is processed using the Alpha Dropout technique to obtain the path matrix.

[0022] Furthermore, a preferred embodiment is proposed, wherein the first stage of gene pathway and pathological image cross-reconstruction fusion in step S3 includes: simulating the complex interaction between pathological image features and gene pathway features using a cross-modal feature recombination module, wherein the cross-modal feature recombination module includes:

[0023] Computational pathway-pathology and pathway-pathway feature interactions;

[0024] The interactions between pathway-pathology and pathway-pathology features are processed using a normalized exponential function and then spliced ​​together.

[0025] The concatenated interaction features are multiplied by the value vectors of pathways and pathology to generate a comprehensive gene feature R. F1 ;

[0026] Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features;

[0027] The pathology-pathway interaction features and pathway-pathology interaction features are subjected to dot product and concatenation operations with the value vectors, respectively, to obtain the second fusion feature R. F2 .

[0028] Furthermore, a preferred method is proposed, wherein the integrated gene feature R F1 for:

[0029] R F1 =Cat(softmax(R) P-H ), softmax(R) P-P ))·Cat(V P V H )

[0030] Among them, R P-H As a characteristic of pathway-pathology interaction, R P-P V represents the characteristics of pathway-to-pathway interactions. P V is the value vector of the path. H This is a pathological value vector.

[0031] Furthermore, a preferred method is proposed, wherein the second stage of step S3 involves bilinear synergistic fusion of gene pathways and pathological images, including:

[0032] R is calculated using the cross-modal feature recombination module. F1 Interaction features between pathological features generate new interaction features R H-F1 and R F1-H ;

[0033] R H-F1 and R F1-H respectively with value vector V F1 and V H Perform a dot product operation, followed by a concatenation operation to obtain a new cross-modal fusion feature R. F3 .

[0034] Furthermore, a preferred embodiment is proposed, wherein step S4 includes:

[0035] The new cross-modal fusion feature R F3 Fusion features R generated in the first stage F2 After concatenation, intermediate feature Z is generated through linear transformation. cat ;

[0036] intermediate feature Z cat Nonlinear representations are generated using Tanh and Sigmoid activation functions respectively, and then the final fused feature R is obtained by multiplying them element-wise and performing a linear transformation. Final .

[0037] Furthermore, a preferred method is proposed, wherein the final fusion feature R Final for:

[0038] R Final =W(τ(Z) cat )⊙σ(Zcat ))+b

[0039] Where W is the weight matrix of the linear transformation, b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function.

[0040] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in any of the preceding claims.

[0041] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features as described in any of the above-mentioned embodiments.

[0042] The advantages of this invention are:

[0043] This invention proposes a cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features to address the problems of feature redundancy, insufficient modal synergy, and high computational complexity in existing multimodal integration methods. First, a gene pathway selection algorithm is employed, using a gene-pathway association matrix combined with gene expression significance index (ES), and dynamic weighting to screen key genes closely related to cancer survival. This extracts highly biologically interpretable, low-dimensional pathway features from high-dimensional, sparse gene data, significantly improving data usability and model generalization ability. Based on this, a two-stage fusion strategy is designed to achieve more efficient modal integration by deeply mining the interaction between gene pathway features and histopathological image features. In the first stage, a cross-modal feature reconstruction fusion strategy is proposed. Through a cross-modal feature recombination module, a multi-head self-attention mechanism is used to capture the deep interaction between gene pathway features and pathological features, generating preliminary fusion features to lay the foundation for subsequent processing. In the second stage, a bilinear synergistic fusion strategy of gene pathways and histopathology was adopted. A bilinear gating mechanism was introduced to dynamically adjust the weights of cross-modal features, and a nonlinear activation function was used to further optimize the feature integration process, significantly improving fusion efficiency and predictive performance. Experimental results show that this method outperforms state-of-the-art methods in survival prediction on four cancer datasets, with a significant improvement in C-index. The model also demonstrates clear advantages in robustness and biological interpretability. Attached Figure Description

[0044] Figure 1This is a flowchart of the cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features as described in Implementation Method 1. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0046] Implementation Method 1, see Figure 1 This embodiment describes a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features. The method includes:

[0047] Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight ranking to screen key genes closely related to cancer survival;

[0048] Step S2: Map key genes closely related to cancer survival to their corresponding pathways to form a pathway matrix;

[0049] Step S3: Construct a two-stage fusion strategy, including: the first stage of cross-reconstruction fusion of gene pathways and pathological images, and the second stage of bilinear synergistic fusion of gene pathways and pathological images;

[0050] Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through a nonlinear activation function.

[0051] In this embodiment, a gene pathway selection algorithm is employed. This method first calculates the gene-pathway association matrix, screening out key genes closely related to cancer survival, effectively reducing the dimensionality of input features and mitigating the impact of redundant features. Furthermore, a two-stage fusion strategy is used to achieve cross-reconstruction fusion of gene pathways and pathological images, as well as bilinear synergistic fusion, in two stages. In the first stage, the cross-reconstruction fusion of gene pathways and pathological images constructs a more accurate feature space based on the correlation between gene pathway information and pathological images. In the second stage, the bilinear synergistic fusion of gene pathways and pathological images further enhances the interaction and synergistic effect of cross-modal features. This two-stage fusion strengthens the synergistic relationship between different modalities and improves overall prediction accuracy. Simultaneously, this embodiment optimizes the weights of different modal features during the fusion process by introducing dynamic weight ranking and a bilinear gating mechanism, effectively reducing computational complexity. In particular, the use of a nonlinear activation function to optimize the feature integration process ensures both the accuracy of feature selection and improved model computational efficiency.

[0052] Implementation Method Two: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method One. Step S1 includes:

[0053] Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the degree of participation and activity of the gene in the pathway by summing all elements in the gene row;

[0054] Gene expression significance (ES) was calculated using the DESeq2 method.

[0055] The weighted score of each gene is calculated based on the gene participation index (PI) and expression significance (ES).

[0056] Generate a sorted list W by sorting the genes in descending order of their weighted scores. sorted ;

[0057] Genes in the top p% based on scores are selected, a dynamic threshold is calculated, and the final gene list SL is filtered by the threshold, with the p% value set to 25%.

[0058] This implementation considers both the gene's role in the pathway (PI) and the significance of gene expression (ES) simultaneously. This approach comprehensively assesses gene function, rather than relying solely on single gene expression levels, further uncovering key signals closely related to cancer survival. A dynamic threshold (set at 25%) is used to screen genes, adaptively adjusting the screening criteria based on the dataset characteristics. This makes the screening process more flexible, avoiding biases and omissions that may arise from using fixed thresholds. It helps select the most predictive genes from a large pool, improving the model's prediction accuracy. Furthermore, sorting genes in descending order of weighted scores prioritizes the screening of the most important genes. This not only improves the effectiveness of gene selection but also helps researchers focus on key biomarkers, enhancing model performance.

[0059] The weighted scoring method in step S1 of this implementation, which comprehensively considers both gene expression significance and pathway participation, can more accurately screen gene biomarkers closely related to cancer survival, thereby improving the accuracy and reliability of cancer survival prediction models. By effectively fusing gene expression data and biological pathway information, this method can significantly improve the shortcomings of traditional single-gene expression data in predicting cancer survival. The screening method using dynamic thresholds and weighted scores can effectively reduce the interference of irrelevant genes and improve the generalization ability of the model. By limiting the number of genes screened, the genes ultimately retained all have strong biological relevance and predictive value.

[0060] Implementation Method 3: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method 1. Step S2 includes:

[0061] Key genes closely related to cancer survival are mapped to corresponding pathways to form feature representations;

[0062] Low-dimensional pathway embeddings are generated by processing feature representations using a self-normalized neural network.

[0063] The low-dimensional path embedding is processed using the Alpha Dropout technique to obtain the path matrix.

[0064] This implementation effectively captures systemic changes in gene expression, rather than just the effects of individual genes, by mapping key genes closely related to cancer survival to corresponding biological pathways. This approach, considering gene interactions and their roles in biological pathways, helps to provide a more comprehensive understanding of tumor biology and offers more accurate predictions. Furthermore, processing feature representations using a self-normalizing neural network effectively reduces data dimensionality, decreases computational resource consumption, and retains as much biological information as possible. Through these steps, the model not only effectively extracts key features from gene data but also improves the accuracy of cancer survival predictions through pathway-level integration and data self-normalization. In addition, the inclusion of Alpha Dropout enhances the model's generalization ability, reduces the risk of overfitting, and allows the model to adapt to different cancer datasets, exhibiting better universality.

[0065] In this implementation, the network relationships of biological pathways and genes are fully utilized when combining genomic and histopathological information, which enhances the model's understanding of tumor biological mechanisms.

[0066] Implementation Method Four: This implementation method further defines the cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features described in Implementation Method One. The first phase of gene pathway and pathological image cross-reconstruction fusion in step S3 includes: simulating the complex interaction between pathological image features and gene pathway features using a cross-modal feature recombination module. The cross-modal feature recombination module includes:

[0067] Computational pathway-pathology and pathway-pathway feature interactions;

[0068] The interactions between pathway-pathology and pathway-pathology features are processed using a normalized exponential function and then spliced ​​together.

[0069] The concatenated interaction features are multiplied by the value vectors of pathways and pathology to generate a comprehensive gene feature R. F1 ;

[0070] Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features;

[0071] The pathology-pathway interaction features and pathway-pathology interaction features are subjected to dot product and concatenation operations with the value vectors, respectively, to obtain the second fusion feature R. F2 .

[0072] Implementation Method Five: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method Four. The integrated gene feature R... F1 for:

[0073] R F1 =Cat(softmax(R) P-H ), softmax(R) P-P ))·Cat(V P V H )

[0074] Among them, R P-H As a characteristic of pathway-pathology interaction, R P-P V represents the characteristics of pathway-to-pathway interactions. P V is the value vector of the path. H This is a pathological value vector.

[0075] Implementation Method Six: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method Four. The bilinear synergistic fusion of gene pathways and pathological images in the second stage of step S3 includes:

[0076] R is calculated using the cross-modal feature recombination module. F1 Interaction features between pathological features generate new interaction features R H-F1 and R F1-H ;

[0077] R H-F1 and R F1-H respectively with value vector V F1 and V H Perform a dot product operation, followed by a concatenation operation to obtain a new cross-modal fusion feature R. F3 .

[0078] This embodiment is described in conjunction with Embodiments Four and Five. In this embodiment, by combining gene pathway information and pathological image features, information can be extracted from data of two different modalities. This fusion method can comprehensively capture the biological and histological characteristics of tumors, thereby improving the accuracy of prediction results. In the first stage, a cross-modal feature recombination module is used to perform complex interactions between pathway and pathological image features, allowing their information to complement and enhance each other, improving the model's learning ability. By calculating pathway-pathology and pathway-pathology interaction features, the deep-level correlation between gene pathways and pathological images can be effectively captured. After processing with a normalized exponential function and then splicing the interaction features, the expressive and discriminative capabilities of the features can be strengthened, thereby improving the accuracy of prediction. Comprehensive gene features are generated through dot product operations and further fused with the value vectors of pathology and pathways. Combining dot product and splicing operations allows for the organic combination of different types of features, better expressing their potential information. In the second stage, bilinear collaborative fusion is used to further enhance the interaction effect of features. By calculating the interaction between pathological features and pathway features and performing dot product and splicing operations on them, new cross-modal fusion features can be effectively generated. This process enhances the model's ability to collaboratively perceive features across different modalities, leading to more accurate predictions. Through multi-level feature fusion and interaction, the model obtains richer feature representations, enabling it to better capture complex factors influencing survival when predicting cancer survival. The relationship between gene pathway data and pathological images is complex, and traditional single-feature extraction methods often struggle to effectively capture these associations. The two-stage fusion method, through refined feature interaction processing, effectively addresses this issue and improves prediction accuracy.

[0079] Implementation Method Seven: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method Five. Step S4 includes:

[0080] The new cross-modal fusion feature R F3 Fusion features R generated in the first stage F2 After concatenation, intermediate feature Z is generated through linear transformation. cat ;

[0081] intermediate feature Z cat Nonlinear representations are generated using Tanh and Sigmoid activation functions respectively, and then the final fused feature R is obtained by multiplying them element-wise and performing a linear transformation. Final .

[0082] In this implementation, by splicing new cross-modal fusion features with the fusion features generated in the first stage, the model can simultaneously integrate information from genetic and pathological data, capturing valuable features from different data sources. This multi-data source fusion strategy can comprehensively reflect the biological characteristics of cancer patients, contributing to the construction of more accurate survival prediction models. Intermediate features are generated through linear transformation, and then nonlinear representations are generated through Tanh and Sigmoid activation functions. Tanh and Sigmoid activation functions can extract different nonlinear relationships respectively. Tanh is typically used to handle features with strong symmetry, while Sigmoid can compress data to between 0 and 1, facilitating the handling of probabilistic problems. Through this combination, the model can capture complex relationships between features, improving the model's expressive power. Multiplying the features after Tanh and Sigmoid activation element-wise generates richer interaction information, representing the synergistic effect between features. In the process of interaction between multiple features, element-wise multiplication can effectively fuse interactive features between different modalities, which is very useful in processing complex biomedical data, especially the nonlinear relationships that may exist between genetic and histopathological information.

[0083] Implementation Method Eight: This implementation method further defines the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features described in Implementation Method Seven. The final fusion feature R... Final for:

[0084] R Final =W(τ(Z) cat )⊙σ(Z cat ))+b

[0085] Where W is the weight matrix of the linear transformation, b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function.

[0086] Implementation Method Nine: A computer device according to this implementation method includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in any one of Implementation Methods One to Eight.

[0087] Implementation Method 10: A computer-readable storage medium according to this implementation method, wherein a computer program is stored on the computer program, and the computer program is executed by a processor to perform the steps of the cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features as described in any one of Implementation Methods 1 to 8.

[0088] Implementation Method Eleven: This implementation method provides a specific example of the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in Implementation Method One, and also serves to explain Implementation Methods Two to Eight. Specifically:

[0089] (1) In this embodiment, multiple cancer datasets including GC_HYD and TCGA are used, and their survival prediction performance is evaluated by 5-fold cross-validation. The model is implemented using Python 3.11.7 and PyTorch framework, with training set to 20 epochs and RAdam optimizer (learning rate 5e-5, weight decay of 1e-3). All training is performed on NVIDIA RTX4070Ti.

[0090] (2) This embodiment proposes a pathological feature extraction method, using a pre-trained UNI (Universal Self-Supervised Model) to efficiently extract biologically meaningful low-dimensional embedding features from complex panoramic histopathological images (WSIs). Traditional methods encounter difficulties when processing large-scale data, especially when the image resolution is high, the number of image patches is large, and the feature dimension is large. To solve this problem, this embodiment uses a pre-trained UNI model to capture multi-level semantic features from large-scale pathological datasets. Specifically, the tissue region in each WSI is segmented, irrelevant background parts are excluded, and it is divided into multiple non-overlapping image patches. Each image patch hi is processed by the UNI model to extract deep features and generate a low-dimensional feature vector H. i =f(h i ), where f(·) is the feature extraction function. Finally, the features of all image patches are integrated into a single feature matrix. Local information is captured, where d is the feature dimension. The UNI model improves the robustness and adaptability of feature extraction, enabling it to better adapt to different pathological datasets.

[0091] (3) This embodiment also proposes an innovative gene selection algorithm (GSA), which combines biological pathway information to guide gene selection. This algorithm not only removes redundant genes but also retains those genes crucial to biological processes, thereby improving the quality of gene data. First, the algorithm calculates the participation index (PI) of each gene in the gene-pathway association matrix, reflecting the gene's participation and activity in the pathway by summing all elements in the gene row. Then, the DESeq2 method is used to calculate gene expression significance (ES), which assesses the expression differences of genes under different sample conditions. The gene participation index (PI) and expression significance (ES) are combined to calculate the weighted score W[g] for each gene. i =PI*ES, considering both pathway involvement and expression significance. After calculating the scores of all genes, they are sorted in descending order of score to generate a sorted list W. sorted Next, the top p% of genes are selected based on their scores, a dynamic threshold (T) is calculated, and this threshold is used to filter the final gene list SL, with the p% value set to 25%. Subsequently, the algorithm maps biologically relevant genes to their corresponding pathways to provide input for survival prediction. These gene-pathway associations form feature representations. By using a self-normalized neural network (SNN), we generate low-dimensional pathway embeddings and reduce overfitting using Alpha Dropout. The final generated pathway matrix... Where NP represents the number of pathways and d represents the embedding dimension. This pathway matrix retains biological interpretability and can be effectively integrated with neural networks, thereby providing accurate and biologically meaningful feature representations for survival prediction tasks.

[0092] (4) The multimodal fusion strategy in this embodiment aims to efficiently integrate data from different sources to achieve more accurate survival prediction. To achieve this goal, an innovative two-stage fusion strategy is proposed. The first stage is the cross-reconstruction fusion of gene pathways and pathological images, and the second stage is the bilinear synergistic fusion of gene pathways and pathological images.

[0093] In the first stage, the main goal is to improve fusion performance by deeply modeling the interaction between pathological image features and gene pathway features. To this end, a novel module called the cross-modal feature recombination module is designed. This module uses a multi-head self-attention (MHSA) mechanism to model the complex interactions between pathological image features and gene pathway features. The unique feature of this module is its ability to optimize intermodal interactions by rearranging feature representations, ensuring effective fusion of pathological image features and gene pathway features. In this stage, we first calculate the pathway-pathology (R...P-H ) and pathway-to-pathway (R) P-P The interaction of features is represented by the following formulas:

[0094] R P-H =CMFR(Q P K H ),

[0095] R P-P =CMFR(Q P K P ).

[0096] Among them, Q P It is a query vector for gene pathways, K H and K P These are the key vectors for pathological image features and gene pathway features, respectively, with CMFR representing feature recombination operations.

[0097] Then, R P-H and R P-P These two interaction features are concatenated after being processed by a normalized exponential function (softmax), and then combined with the value vectors of pathways and pathology (V). P V H Perform a dot product operation to generate the final comprehensive gene feature R. F1 :

[0098] R F1 =Cat(softmax(R) P-H ), softmax(R) P-P ))·Cat(V P V H ).

[0099] This process can highlight the deep interactions between gene pathways and pathological image features, and capture their potential associations. In generating R... F1 Then, the interaction between pathological features and pathway features is further calculated to generate: R H-P =CMFR(Q H K P ), for R H-P and R P-H The sum of the vectors (V) P V H Perform a dot product operation, then concatenate the results to obtain the second fused feature:

[0100] R F2 =Cat(softmax(R) P-H )·V H softmax(R) H-P )·V P ).

[0101] R F2 The generation process aims to capture the relationship between gene pathway features and pathological image features at a deeper level, while balancing information between modalities.

[0102] (5) In the second stage, the goal is to further explore the pathological features (H) and the integrated genetic features (R) generated in the first stage. F1 The deep interaction relationships between them are addressed by employing the cross-modal feature recombination module again to calculate R. F1 Interaction features between pathological features generate new interaction features R H-F1 and R F1-H Among them, R H-F1 =CMFR(Q H K F1 ), R F1-H =CMFR(Q F1 K H The reason for choosing R F1 Interact, instead of R F2 It is because of R F1 Including more interactive information, it not only reflects the deep connection between pathological images and gene pathways, but also integrates the relationships between different pathways, thus providing richer contextual information. The core objective of this stage is to further improve the representation and semantic understanding of pathological image features, enabling the model to better perform subsequent tasks. Subsequently, R... H-F1 and R F1-H respectively with value vector V F1 and V H Perform a dot product operation, followed by a concatenation operation to obtain a new cross-modal fusion feature R. F3 ,Right now:

[0103] R F3 =Cat(softmax(R) H-F1 )·V F1 softmax(R) F1-H )·V H ).

[0104] (6) To address the issues of information redundancy, intermodal feature imbalance, and insufficient nonlinear interaction during multimodal feature fusion, this embodiment designs a bilinear gating mechanism to dynamically optimize the cross-modal feature fusion process. After multimodal feature concatenation, due to the significant differences in the dimensions and distributions of different modal features, direct fusion may lead to some modal information dominating, while important information from other modalities is weakened or ignored. Furthermore, simple linear concatenation struggles to capture complex nonlinear relationships between modalities, further limiting the expressive power of the fused features. To address these issues, the bilinear gating mechanism introduces Tanh and Sigmoid activation functions to dynamically weight and nonlinearly transform the concatenated features. The Tanh activation function captures complex nonlinear relationships between features and allows feature values ​​to fluctuate within the range of (-1, 1); the Sigmoid activation function restricts feature values ​​to the range of (0, 1), reflecting the importance or weight of the features. The combination of the two activation functions achieves dynamic adjustment of feature weights through element-wise multiplication, which strengthens key features and suppresses redundant or noisy information, thereby ensuring information balance between modalities.

[0105] Specifically, the cross-modal fusion feature R F3 Fusion features R generated in the first stage F2 After concatenation, intermediate feature Z is generated through linear transformation. cat =W·R cat +b, where R cat =concat(R F3 R F2 Subsequently, nonlinear representations were generated using Tanh and Sigmoid activation functions, and then multiplied element-wise before undergoing a linear transformation to obtain the final fused feature R. Final =W(τ(Z) cat )⊙σ(Z cat ))+b, where R cat Z represents the concatenated feature vector. cat Let R represent the feature vector after linear transformation. W is the weight matrix of the linear transformation, which adjusts the weights of the input features. b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function. The bilinear gating mechanism enhances the nonlinear interaction between modalities while optimizing feature fusion, effectively suppressing information redundancy and dynamically balancing feature contributions between modalities, thereby generating high-quality fused features R. Final This provides more accurate and robust inputs for survival prediction tasks.

[0106] (7) To predict a patient's survival, multimodal features R obtained through two-stage fusion can be used. Final Make predictions and calculate mortality scores based on each time interval. And calculate the estimated patient survival risk score, specifically: based on R Final A classifier is used to calculate a mortality score for each time interval. The score is then processed using the Sigmoid activation function σ(·) to obtain the probability of death for each time interval. Next, the patient's survival probability is calculated based on the mortality probabilities of all time intervals, specifically by accumulating the mortality probabilities of all previous time intervals. Get each time interval (t) j-1 , t j The patient's survival probability. Finally, the patient's survival risk score R. risk It is calculated by summing the negative logarithms of the probability of death for all time intervals. The survival risk score R risk The data is used to train the model and ultimately to assess the patient's survival probability across different time intervals. In this way, we organically combine multimodal features with survival prediction to achieve accurate survival risk prediction.

[0107] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit its protection scope. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading this disclosure, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the published pending claims.

Claims

1. A cancer prognostic method based on a staged fusion strategy of biological pathways and histopathological characteristics, characterized in that, The method includes: Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight ranking to screen key genes closely related to cancer survival; Step S2: Map key genes closely related to cancer survival to their corresponding pathways to form a pathway matrix; Step S3: Construct a two-stage fusion strategy, including: the first stage of cross-reconstruction fusion of gene pathways and pathological images, and the second stage of bilinear synergistic fusion of gene pathways and pathological images; Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through a nonlinear activation function; Step S1 includes: Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the degree of participation and activity of the gene in the pathway by summing all elements in the gene row; Gene expression significance was calculated using the DESeq2 method. ; Based on the gene participation index The weighted score for each gene was calculated using expression significance ES; Generate a sorted list by sorting the genes in descending order of their weighted scores. ; Select based on score The genes are analyzed, a dynamic threshold is calculated, and the final gene list is filtered by the threshold. , The value is set to 25%; Step S2 includes: Key genes closely related to cancer survival are mapped to corresponding pathways to form feature representations; Low-dimensional pathway embeddings are generated by processing feature representations using a self-normalized neural network. The low-dimensional path embedding is processed using the Alpha Dropout technique to obtain the path matrix; The first stage of gene pathway and pathological image cross-reconstruction fusion in step S3 includes: simulating the complex interaction between pathological image features and gene pathway features using a cross-modal feature recombination module, wherein the cross-modal feature recombination module includes: Computational pathway-pathology and pathway-pathway feature interactions; The interactions between pathway-pathology and pathway-pathology features are processed using a normalized exponential function and then spliced ​​together. The concatenated interaction features are multiplied by the value vectors of pathways and pathology to generate comprehensive gene features. ; Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features; The pathology-pathway interaction features and pathway-pathology interaction features are subjected to dot product and concatenation operations with the value vectors, respectively, to obtain the second fusion feature. ; The second stage of step S3, the bilinear synergistic fusion of gene pathways and pathological images, includes: Calculation using cross-modal feature recombination module Interaction features between pathological features and generate new interaction features and ; Will and respectively with value vector and Perform a dot product operation, followed by a concatenation operation to obtain new cross-modal fusion features. ; Step S4 includes: New cross-modal fusion features Fusion features generated in the first stage After splicing, intermediate features are generated through linear transformation. ; intermediate features Nonlinear representations are generated using Tanh and Sigmoid activation functions respectively, and then the final fused features are obtained by multiplying them element-wise and performing a linear transformation. .

2. The cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features according to claim 1, characterized in that, The comprehensive genetic characteristics for: in, This is a characteristic of pathway-pathology interaction. Features of pathway-to-pathway interactions The value vector of the path, This is a pathological value vector.

3. The cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features according to claim 1, characterized in that, The final fusion feature for: in, It is the weight matrix of the linear transformation. It is the bias term of the linear transformation. yes Activation function yes Activation function.

4. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features as described in any one of claims 1-3.

Images