Cancer prognosis method based on biological pathway and histopathologic feature staged fusion strategy

Abstract

The invention discloses a cancer prognosis method based on a biological pathway and histopathologic feature staged fusion strategy, and relates to the technical field of image processing. The problems of feature redundancy, insufficient modal collaboration, high calculation complexity and the like in the existing multi-modal integration method are solved. The method comprises the following steps: calculating a participation index and a gene expression significance index in a gene-pathway incidence matrix according to a gene pathway selection algorithm, and screening key genes closely related to cancer survival by utilizing dynamic weight sorting; key genes closely related to cancer survival are mapped to corresponding pathways to form a pathway matrix; constructing a double-stage fusion strategy, including cross reconstruction fusion of the gene pathway and the pathological image in the first stage and bilinear collaborative fusion of the gene pathway and the pathological image in the second stage; and dynamically adjusting the weight of the cross-modal feature in the dual-stage fusion strategy according to a bilinear gating mechanism, and optimizing the feature integration process through a nonlinear activation function. According to the invention, fusion efficiency and prediction performance are significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features. Background Art

[0002] Cancer survival prediction is a crucial direction in cancer research, and has far-reaching significance for the development of personalized treatments and accurate assessment of patient prognosis. The complexity and heterogeneity of cancer make accurate prediction and personalized treatment the core challenges of cancer treatment. Traditional clinical evaluation methods can provide guidance for patient prognosis to a certain extent, but with the continuous advancement of technology, the integration of new bioinformatics tools and multimodal data has significantly improved the accuracy and operability of cancer survival prediction.

[0003] Currently, histopathological images (Whole Slide Images, WSIs) and genomic data play a vital role in cancer research, providing key information of different dimensions from the tissue and molecular levels. WSIs can help clinicians assess the malignancy, grade, and possible metastasis risk of tumors by showing the histological structure, morphological characteristics, and changes in the tumor microenvironment in detail. These images play an irreplaceable role in diagnosis, especially in the early detection and prognosis assessment of tumors. However, WSIs only observe from a morphological perspective and it is difficult to reveal the genetic and molecular mechanisms of tumors, which limits its application in complex biological research.

[0004] In contrast, genomic data provides a deeper understanding of cancer biology and precision treatment by providing information such as gene expression, gene mutation, and copy number variation. These data can reveal the molecular mechanisms of tumor occurrence, development, and metastasis, and provide effective molecular targets for clinical treatment. However, genomic data are usually high-dimensional, sparse, and have strong redundancy between features. This makes the processing and analysis of genomic data a challenge. How to efficiently extract meaningful information from it, avoid redundancy, and improve prediction performance is a key issue.

[0005] Although single modality data (such as separate WSIs or genomic data) can provide certain support for the diagnosis and treatment of tumors, due to the multidimensional characteristics of tumors, methods that rely solely on a single modality are difficult to fully reflect the complexity of tumors. Therefore, the integration of multimodal data has become an important means to improve the accuracy of cancer prediction. Existing multimodal integration methods usually combine pathological images and genetic data features through simple splicing or shallow interaction. However, these methods often fail to fully capture the deep association between the two, resulting in limited improvement in prediction performance. Simple splicing or shallow fusion can only enhance the expressive power of the model to a certain extent, but cannot deeply understand the potential complex relationship between genetic data and tissue images.

[0006] In addition, due to the high dimensionality and sparsity of genomic data, these data often contain a large number of redundant features, which not only increases the complexity of calculations, but also may introduce noise and affect the accuracy of the model. In this context, how to effectively perform feature selection, dimensionality reduction and information integration, and eliminate redundant features, has become an important challenge in multimodal fusion. WSIs have a huge amount of data and high resolution, which makes the processing of these images face huge computational costs. Especially in practical applications, how to ensure high efficiency while not losing key information is an urgent problem to be solved. Summary of the invention

[0007] In view of the problems of feature redundancy, insufficient modality coordination and high computational complexity existing in existing multimodal integration methods, the present invention proposes a cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features, the method comprising:

[0008] Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight sorting to screen key genes closely related to cancer survival;

[0009] Step S2: Map the key genes closely related to cancer survival to the corresponding pathways to form a pathway matrix;

[0010] Step S3: constructing a two-stage fusion strategy, including: cross-reconstruction fusion of gene pathways and pathological images in the first stage, and bilinear collaborative fusion of gene pathways and pathological images in the second stage;

[0011] Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through nonlinear activation functions.

[0012] Furthermore, a preferred method is proposed, wherein step S1 comprises:

[0013] Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the participation degree and activity of the gene in the pathway by summing all elements in the gene row;

[0014] The DESeq2 method was used to calculate the gene expression significance ES;

[0015] The weighted score of each gene is calculated based on the gene participation index PI and expression significance ES;

[0016] According to the weighted score of each gene, the scores are arranged in descending order to generate a sorted list W sorted ;

[0017] The top p% genes were selected according to the scores, a dynamic threshold was calculated, and the final gene list SL was filtered by the threshold, with the p% value set to 25%.

[0018] Furthermore, a preferred method is proposed, wherein step S2 comprises:

[0019] Map key genes closely related to cancer survival to corresponding pathways to form feature representations;

[0020] Processing feature representations according to a self-normalizing neural network to generate low-dimensional pathway embeddings;

[0021] The low-dimensional pathway embedding is processed by Alpha Dropout technology to obtain the pathway matrix.

[0022] Furthermore, a preferred method is proposed, wherein the cross-reconstruction fusion of the gene pathway and the pathological image in the first stage in step S3 comprises: using a cross-modal feature recombination module to simulate the complex interaction between the pathological image features and the gene pathway features, and the cross-modal feature recombination module comprises:

[0023] Computing pathway-pathology and pathway-pathway feature interactions;

[0024] The pathway-pathology and pathway-pathway feature interactions were processed by normalized exponential function and then spliced;

[0025] Perform dot product operation on the spliced ​​interaction features with the value vectors of pathways and pathologies to generate comprehensive gene features R F1 ;

[0026] Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features;

[0027] Perform dot product operations and concatenation operations on the pathology-pathway interaction features and pathway-pathology interaction features with the value vector respectively to obtain the second fusion feature R F2 .

[0028] Furthermore, a preferred embodiment is proposed, wherein the comprehensive gene feature R F1 for:

[0029] R F1 = Cat(softmax(R P-H ), softmax(R P-P ))·Cat(V P , V H )

[0030] Among them, R P-H is the characteristic of pathway-pathology interaction, R P-P is the characteristic of pathway-pathway interaction, V P is the value vector of the path, V H is the pathological value vector.

[0031] Furthermore, a preferred method is proposed, in which the bilinear collaborative fusion of the gene pathway and the pathological image in the second stage in step S3 comprises:

[0032] The cross-modal feature recombination module is used to calculate R F1 The interaction features between the pathological features are used to generate new interaction features R H-F1 and R F1-H ;

[0033] R H-F1 and R F1-H Respectively with the value vector V F1 and V H Perform a dot product operation, and then perform a concatenation operation to obtain a new cross-modal fusion feature R F3 .

[0034] Furthermore, a preferred method is proposed, wherein step S4 comprises:

[0035] The new cross-modal fusion feature R F3 The fusion feature R generated in the first stage F2 After concatenation, the intermediate feature Z is generated by linear transformation cat ;

[0036] The intermediate feature Z cat The nonlinear representation is generated by Tanh and Sigmoid activation functions respectively, and then linearly transformed again after element-by-element multiplication to obtain the final fusion feature R Final .

[0037] Furthermore, a preferred method is proposed, wherein the final fusion feature R Final for:

[0038] R Final =W(τ(Z cat )⊙σ(Zcat ))+b

[0039] Among them, W is the weight matrix of the linear transformation, b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function.

[0040] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the cancer prognosis method based on the staged fusion strategy of biological pathways and tissue pathological features as described in any one of the above items.

[0041] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the cancer prognosis method based on the staged fusion strategy of biological pathways and tissue pathological characteristics as described above are executed.

[0042] The present invention is beneficial in that:

[0043] The present invention proposes a cancer prognosis method based on a phased fusion strategy of biological pathways and histopathological features to solve the problems of feature redundancy, insufficient modality coordination and high computational complexity in existing multimodal integration methods. First, a gene pathway selection algorithm is adopted. Through the gene-pathway association matrix combined with the gene expression significance index (ES), dynamic weight sorting is used to screen key genes closely related to cancer survival, and high biological interpretability and low-dimensional pathway features are extracted from high-dimensional and sparse gene data, which significantly improves the availability of data and the generalization ability of the model. On this basis, a two-stage fusion strategy is designed to achieve more efficient modality integration by deeply mining the interactive relationship between gene pathway features and histopathological image features. In the first stage, a cross-reconstruction fusion strategy of gene pathway and histopathology is proposed. Through the cross-modal feature reconstruction module, the multi-head self-attention mechanism is used to capture the deep interaction between gene pathway features and pathological features, and generate preliminary fusion features to lay the foundation for subsequent processing. In the second stage, a bilinear collaborative fusion strategy of gene pathway and tissue pathology was adopted, and a bilinear gating mechanism was introduced to dynamically adjust the weights of cross-modal features. The feature integration process was further optimized through nonlinear activation functions, significantly improving the fusion efficiency and prediction performance. Experimental results show that the survival prediction performance of this method on four cancer datasets is better than the existing state-of-the-art methods, with a significant improvement in C-index, and the model shows obvious advantages in robustness and biological interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1This is a flow chart of the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in implementation mode 1. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0046] Implementation method 1, see Figure 1 The present embodiment is described as follows. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in the present embodiment comprises:

[0047] Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight sorting to screen key genes closely related to cancer survival;

[0048] Step S2: Map the key genes closely related to cancer survival to the corresponding pathways to form a pathway matrix;

[0049] Step S3: constructing a two-stage fusion strategy, including: cross-reconstruction fusion of gene pathways and pathological images in the first stage, and bilinear collaborative fusion of gene pathways and pathological images in the second stage;

[0050] Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through nonlinear activation functions.

[0051] In this embodiment, through the gene pathway selection algorithm, the method first calculates the gene-pathway association matrix, screens out key genes closely related to cancer survival, effectively reduces the dimension of the input features, and reduces the impact of redundant features. Further, through the two-stage fusion strategy, the cross-reconstruction fusion and bilinear collaborative fusion of gene pathways and pathological images are realized in the two-stage process. In the first stage, the cross-reconstruction fusion of gene pathways and pathological images can construct a more accurate feature space through the correlation between gene pathway information and pathological images. In the second stage, the bilinear collaborative fusion of gene pathways and pathological images further enhances the interaction and synergy of cross-modal features. Through this two-stage fusion, the collaborative relationship between different modalities can be enhanced and the overall prediction accuracy can be improved. At the same time, this embodiment optimizes the weights of different modal features during the fusion process through the introduction of dynamic weight sorting and bilinear gating mechanism, effectively reducing the computational complexity. In particular, the use of nonlinear activation functions to optimize the feature integration process ensures the accuracy of feature selection and the improvement of model calculation efficiency.

[0052] Embodiment 2: This embodiment further limits the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 1, and the step S1 comprises:

[0053] Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the participation degree and activity of the gene in the pathway by summing all elements in the gene row;

[0054] The DESeq2 method was used to calculate the gene expression significance ES;

[0055] The weighted score of each gene is calculated based on the gene participation index PI and expression significance ES;

[0056] According to the weighted score of each gene, the scores are arranged in descending order to generate a sorted list W sorted ;

[0057] The top p% genes were selected according to the scores, a dynamic threshold was calculated, and the final gene list SL was filtered by the threshold, with the p% value set to 25%.

[0058] In this embodiment, by considering both the degree of gene involvement in the pathway (PI) and the significance of gene expression (ES), this method can comprehensively evaluate the role of genes, rather than relying solely on a single gene expression level, and further tap into key signals that are closely related to cancer survival. A dynamic threshold is used to screen genes (the threshold is set at 25%), and the screening criteria are adaptively adjusted according to the characteristics of the data set, making the screening process more flexible, avoiding the bias and omissions that may be caused when using a fixed threshold, and helping to select the most predictive genes from a large number of genes, thereby improving the prediction accuracy of the model. At the same time, by arranging the genes in descending order according to the weighted scores, the most important genes can be screened out first. This not only improves the effectiveness of gene selection, but also helps researchers focus on key biomarkers and improve the performance of the model.

[0059] The weighted scoring method in step S1 of this embodiment, which comprehensively considers the significance of gene expression and the degree of pathway involvement, can more accurately screen out gene markers that are closely related to cancer survival, thereby improving the accuracy and reliability of the cancer survival prediction model. By effectively integrating gene expression data and biological pathway information, this method can significantly improve the shortcomings of traditional single gene expression data in predicting cancer survival. The screening method using dynamic thresholds and weighted scores can effectively reduce the interference of irrelevant genes and improve the generalization ability of the model. By limiting the number of genes screened, the genes that are ultimately retained have strong biological relevance and predictive value.

[0060] Embodiment 3: This embodiment further limits the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 1, and the step S2 includes:

[0061] Map key genes closely related to cancer survival to corresponding pathways to form feature representations;

[0062] Processing feature representations according to a self-normalizing neural network to generate low-dimensional pathway embeddings;

[0063] The low-dimensional pathway embedding is processed by Alpha Dropout technology to obtain the pathway matrix.

[0064] In this embodiment, by mapping key genes closely related to cancer survival to corresponding biological pathways, it is possible to effectively capture systematic changes in gene expression, rather than just the effects of a single gene. This method helps to more fully understand tumor biological processes by considering the interactions between genes and their roles in biological pathways, providing more accurate information for prediction. Furthermore, by processing feature representations using a self-normalizing neural network, the dimension of the data can be effectively reduced, the consumption of computing resources can be reduced, and as much biological information as possible can be retained. Through the above steps, the model can not only effectively extract key features from genetic data, but also improve the accuracy of cancer survival prediction through the integration of pathway levels and the self-normalization of data. In addition, the addition of Alpha Dropout technology enhances the generalization ability of the model, reduces the risk of overfitting, and enables the model to adapt to different cancer data sets with better universality.

[0065] In this embodiment, when combining genomics and histopathology information, full use is made of the network relationship between biological pathways and genes, thereby enhancing the model's understanding of tumor biological mechanisms.

[0066] Embodiment 4: This embodiment further limits the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in embodiment 1. The cross-reconstruction fusion of gene pathways and pathological images in the first stage in step S3 includes: using a cross-modal feature recombination module to simulate the complex interaction between pathological image features and gene pathway features, and the cross-modal feature recombination module includes:

[0067] Computing pathway-pathology and pathway-pathway feature interactions;

[0068] The pathway-pathology and pathway-pathway feature interactions were processed by normalized exponential function and then spliced;

[0069] Perform dot product operation on the spliced ​​interaction features with the value vectors of pathways and pathologies to generate comprehensive gene features R F1 ;

[0070] Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features;

[0071] Perform dot product operations and concatenation operations on the pathology-pathway interaction features and pathway-pathology interaction features with the value vector respectively to obtain the second fusion feature R F2 .

[0072] Embodiment 5: This embodiment further limits the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 4. F1 for:

[0073] R F1 = Cat(softmax(R P-H ), softmax(R P-P ))·Cat(V P , V H )

[0074] Among them, R P-H is the characteristic of pathway-pathology interaction, R P-P is the characteristic of pathway-pathway interaction, V P is the value vector of the path, V H is the pathological value vector.

[0075] Embodiment 6: This embodiment further defines the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in embodiment 4. The bilinear collaborative fusion of the gene pathway and the pathological image in the second stage in step S3 includes:

[0076] The cross-modal feature recombination module is used to calculate R F1 The interaction features between the pathological features are used to generate new interaction features R H-F1 and R F1-H ;

[0077] R H-F1 and R F1-H Respectively with the value vector V F1 and V H Perform a dot product operation, and then perform a concatenation operation to obtain a new cross-modal fusion feature R F3 .

[0078] This embodiment is explained in combination with the fourth and fifth embodiments. In this embodiment, by combining gene pathway information and pathological image features, information can be extracted from data of two different modalities. This fusion method can fully capture the biological and histological characteristics of the tumor, thereby improving the accuracy of the prediction results. In the first stage, a cross-modal feature recombination module is used to perform complex interactions between pathway and pathological image features, so that the information of the two can complement and enhance each other, thereby improving the learning ability of the model. By calculating the interaction features of pathway-pathology and pathway-pathway, the deep association between gene pathways and pathological images can be effectively captured. By splicing the interaction features after processing with the normalized exponential function, the expression and differentiation capabilities of the features can be enhanced, thereby improving the accuracy of the prediction. The comprehensive gene features are generated by the dot product operation, and further fused with the value vectors of the pathology and pathway. Combined with the dot product and splicing operations, different types of features can be organically combined to better express their potential information. In the second stage, bilinear collaborative fusion is used to further enhance the interaction effect of the features. By calculating the interaction between pathological features and pathway features, and performing dot product and splicing operations on them, new cross-modal fusion features can be effectively generated. This process can enhance the model's ability to collaboratively perceive features of different modalities, making the prediction process more accurate. Through multi-level feature fusion and interaction, the model can obtain richer feature representations, which enables the model to better capture the complex factors that affect survival when predicting cancer survival. The relationship between gene pathway data and pathological images is relatively complex, and traditional single feature extraction methods often have difficulty effectively capturing these associations. The two-stage fusion method effectively solves this problem through sophisticated feature interaction processing and improves the accuracy of prediction.

[0079] Embodiment 7: This embodiment further limits the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 5, and the step S4 comprises:

[0080] The new cross-modal fusion feature R F3 The fusion feature R generated in the first stage F2 After concatenation, the intermediate feature Z is generated by linear transformation cat ;

[0081] The intermediate feature Z cat The nonlinear representation is generated by Tanh and Sigmoid activation functions respectively, and then linearly transformed again after element-by-element multiplication to obtain the final fusion feature R Final .

[0082] In this embodiment, by splicing the new cross-modal fusion features with the fusion features generated in the first stage, this process allows the model to integrate the information of genetic data and pathological data at the same time, and capture the valuable features in different data sources. This multi-data source fusion strategy can fully reflect the biological characteristics of cancer patients and help to build a more accurate survival prediction model. The intermediate features are generated by linear transformation, and then the nonlinear representation is generated by Tanh and Sigmoid activation functions. Tanh and Sigmoid activation functions can extract different nonlinear relationships respectively. Tanh is usually used to process features with strong symmetry, while Sigmoid can compress data between 0 and 1, which is convenient for processing probabilistic problems. Through this combination, the model can capture the complex relationship between features and improve the expression ability of the model. The features activated by Tanh and Sigmoid are multiplied element by element to generate richer interactive information, indicating the synergistic effect between features. In the process of interaction between multiple features, element-by-element multiplication can effectively fuse the interactive features between different modalities, which is very useful when processing complex biomedical data, especially the nonlinear relationship that may exist between genetic information and tissue pathology information.

[0083] Embodiment 8: This embodiment further defines the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 7. The final fusion feature R Final for:

[0084] R Final =W(τ(Z cat )⊙σ(Z cat ))+b

[0085] Among them, W is the weight matrix of the linear transformation, b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function.

[0086] Embodiment 9. A computer device described in this embodiment includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the cancer prognosis method based on the staged fusion strategy of biological pathways and tissue pathological features described in any one of embodiments 1 to 8.

[0087] Embodiment 10. A computer-readable storage medium described in this embodiment stores a computer program, and when the computer program is executed by a processor, the steps of the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in any one of embodiments 1 to 8 are executed.

[0088] Embodiment 11: This embodiment provides a specific example of the cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features described in Embodiment 1, and is also used to explain Embodiments 2 to 8. Specifically:

[0089] (1) In this embodiment, multiple cancer datasets including GC_HYD and TCGA are used, and 5-fold cross validation is used to evaluate their survival prediction performance. The model is implemented using Python 3.11.7 and PyTorch framework, the training is set to 20 cycles, and the RAdam optimizer is used (learning rate is 5e-5, weight decay is 1e-3), and all training is performed on NVIDIA RTX4070Ti.

[0090] (2) In this embodiment, a pathological feature extraction method is proposed, which uses a pre-trained UNI (Universal Self-Supervised Model) model to efficiently extract biologically meaningful low-dimensional embedded features from complex panoramic tissue pathology images (WSIs). Traditional methods encounter difficulties in processing large-scale data, especially when the image resolution is high, the number of image blocks is large, and the feature dimension is large. To solve this problem, a pre-trained UNI model is used in this embodiment to capture multi-level semantic features from a large-scale pathology dataset. Specifically, the tissue area in each WSI is segmented, irrelevant background parts are excluded, and it is divided into multiple non-overlapping image blocks. Each image block hi is processed by the UNI model to extract deep features and generate a low-dimensional feature vector H i =f(h i ), where f(·) is the feature extraction function. Finally, the features of all image blocks are integrated into a feature matrix The local information is captured, where d is the feature dimension. The UNI model improves the robustness and adaptability of the extracted features, making it better suited to different pathology datasets.

[0091] (3) This embodiment also proposes an innovative gene screening algorithm (Gene Selection Algorithm, GSA), which combines biological pathway information to guide gene screening. This algorithm not only removes redundant genes, but also retains those genes that are critical to biological processes, thereby improving the quality of genetic data. First, the algorithm calculates the participation index (PI) of each gene in the gene-pathway association matrix, and reflects the degree of participation and activity of the gene in the pathway by summing all elements in the gene row. Then, the DESeq2 method is used to calculate the gene expression significance (ES), which evaluates the expression differences of genes under different sample conditions. The gene participation index (PI) and expression significance (ES) are combined to calculate the weighted score W[g i ]=PI*ES, taking into account both pathway involvement and expression significance. After calculating the scores of all genes, they are sorted in descending order of scores to generate a sorted list W sorted . Next, the top p% genes are selected based on the score, a dynamic threshold (T) is calculated, and the final gene list SL is filtered by this threshold, with the p% value set to 25%. Subsequently, the algorithm maps biologically relevant genes to their corresponding pathways to provide input for survival prediction. These genes are associated with pathways to form feature representations. By using a self-normalizing neural network (SNN), we generate low-dimensional pathway embeddings and reduce overfitting through the Alpha Dropout technique. The final generated pathway matrix Where NP is the number of pathways and d is the embedding dimension. The pathway matrix retains biological interpretability and can be effectively integrated with neural networks to provide accurate and biologically meaningful feature representations for survival prediction tasks.

[0092] (4) The multimodal fusion strategy in this embodiment aims to efficiently integrate data from different sources to achieve more accurate survival prediction. To achieve this goal, an innovative two-stage fusion strategy is proposed. The first stage is the cross-reconstruction fusion of gene pathways and pathological images, and the second stage is the bilinear collaborative fusion of gene pathways and pathological images.

[0093] In the first stage, the main goal is to improve the fusion performance by deeply modeling the interaction between pathological image features and gene pathway features. To this end, a new module called the cross-modal feature recombination module is designed. The feature recombination module uses the multi-head self-attention mechanism (Multi-Head SelfAttention, MHSA) to model the complex interaction between pathological image features and gene pathway features. The uniqueness of this module is that it can optimize the interaction between modalities by rearranging feature representations, ensuring that pathological image features and gene pathway features can be effectively fused. In this stage, we first calculate the pathway-pathology (RP-H ) and pathway-pathway (R P-P ) features, the formulas are:

[0094] R P-H =CMFR(Q P , K H ),

[0095] R P-P =CMFR(Q P , K P ).

[0096] Among them, Q P is the query vector of the gene pathway, K H and K P are the key vectors of pathological image features and gene pathway features respectively, and CMFR is the feature recombination operation.

[0097] Then, R P-H and R P-P These two interaction features are concatenated after being processed by a normalized exponential function (softmax) and then combined with the value vectors of pathway and pathology (V P , V H ) to perform a dot product operation to generate the final comprehensive gene feature R F1 :

[0098] R F1 = Cat(softmax(R P-H ), softmax(R P-P ))·Cat(V P , V H ).

[0099] This process can highlight the deep interactions between gene pathways and pathological image features and capture the potential associations between them. F1 After that, the interaction between pathological features and pathway features is further calculated to generate: R H-P =CMFR(Q H , K P ), for R H-P and R P-H Respectively and value vector (V P , V H ) performs a dot product operation and then concatenates them to get the second fusion feature:

[0100] R F2 = Cat(softmax(R P-H )·V H , softmax(R H-P )·V P ).

[0101] R F2 The generation process aims to capture the relationship between gene pathway features and pathological image features at a deeper level, while balancing the information between modalities.

[0102] (5) In the second stage, the goal is to further explore the pathological features (H) and the comprehensive gene features (R) generated in the first stage. F1 ). To this end, the cross-modal feature recombination module is used again to calculate R F1 The interaction features between the pathological features are used to generate new interaction features R H-F1 and R F1-H Among them, R H-F1 =CMFR(Q H , K F1 ), R F1-H =CMFR(Q F1 , K H ). The reason for choosing R F1 To interact, instead of R F2 , because R F1 It contains more interactive information, which not only reflects the deep connection between pathological images and gene pathways, but also integrates the relationship between different pathways, thus providing richer contextual information. The core purpose of this stage is to further improve the representation and semantic understanding of pathological image features so that the model can better perform subsequent tasks. H-F1 and R F1-H Respectively with the value vector V F1 and V H Perform a dot product operation and then perform a concatenation operation to obtain a new cross-modal fusion feature R F3 ,Right now:

[0103] R F3 = Cat(softmax(R H-F1 )·V F1 , softmax(R F1-H )·V H ).

[0104] (6) In order to solve the problems of information redundancy, feature imbalance between modalities, and insufficient nonlinear interaction in the process of multimodal feature fusion, a bilinear gating mechanism is designed in this embodiment to dynamically optimize the fusion process of cross-modal features. After the multimodal features are spliced, due to the significant differences in the dimensions and distribution of different modal features, direct fusion may cause some modal information to dominate, while the important information of other modalities is weakened or ignored. In addition, simple linear splicing is difficult to capture the complex nonlinear relationship between modalities, further limiting the expressive power of fused features. To address the above problems, the bilinear gating mechanism dynamically weights and nonlinearly transforms the spliced ​​features by introducing the Tanh activation function and the Sigmoid activation function. The Tanh activation function is used to capture the complex nonlinear relationship between features and allows the eigenvalue to fluctuate within the range of (-1, 1); the Sigmoid activation function limits the eigenvalue to the range of (0, 1), reflecting the importance or weight of the feature. The combination of the two activation functions realizes the dynamic adjustment of feature weights by element-by-element multiplication, which can strengthen key features and suppress redundant or noise information, thereby ensuring information balance between modalities.

[0105] Specifically, the cross-modal fusion feature R F3 The fusion feature R generated in the first stage F2 After concatenation, the intermediate feature Z is generated by linear transformation cat =W·R cat +b, where R cat =concat(R F3 , R F2 ). Then, the nonlinear representation is generated by Tanh and Sigmoid activation functions respectively, and the final fusion feature R is obtained by linear transformation after element-by-element multiplication. Final =W(τ(Z cat )⊙σ(Z cat ))+b, where R cat represents the concatenated feature vector, Z cat represents the feature vector after linear transformation, W is the weight matrix of linear transformation, which is used to adjust the weight of input features, b is the bias term of linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function. The bilinear gating mechanism optimizes feature fusion while enhancing the nonlinear interaction ability between modalities, effectively suppressing information redundancy and dynamically balancing the feature contribution between modalities, thereby generating high-quality fusion features R Final , providing more accurate and robust input for survival prediction tasks.

[0106] (7) If you want to predict the patient's survival, you can use the multimodal feature R obtained based on the two-stage fusion Final Make predictions and calculate the death score for each time interval And calculate the estimated patient survival risk score, specifically: based on R Final , use the classifier to calculate the mortality score for each time interval The score is activated by the Sigmoid function σ(·) to obtain the probability of death in each time interval. Next, the patient's survival probability is calculated based on the probability of death in all time intervals, specifically by accumulating the probability of death in all previous time intervals. Get each time interval (t j-1 , t j ) survival probability. Finally, the patient's survival risk score R risk It is calculated by summing the negative logarithm of the probability of death in all time intervals. The survival risk score R risk It is used to train the model and ultimately to assess the survival probability of patients in different time intervals. In this way, we organically combine multimodal features with survival prediction to complete accurate survival risk prediction.

[0107] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features, characterized in that: The method comprises: Step S1: Calculate the participation index and gene expression significance index in the gene-pathway association matrix according to the gene pathway selection algorithm, and use dynamic weight sorting to screen key genes closely related to cancer survival; Step S2: Map the key genes closely related to cancer survival to the corresponding pathways to form a pathway matrix; Step S3: constructing a two-stage fusion strategy, including: cross-reconstruction fusion of gene pathways and pathological images in the first stage, and bilinear collaborative fusion of gene pathways and pathological images in the second stage; Step S4: Dynamically adjust the weights of cross-modal features in the two-stage fusion strategy according to the bilinear gating mechanism, and optimize the feature integration process through nonlinear activation functions.

2. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 1, characterized in that: The step S1 comprises: Calculate the participation index PI of each gene in the gene-pathway association matrix, and obtain the participation degree and activity of the gene in the pathway by summing all elements in the gene row; The DESeq2 method was used to calculate the gene expression significance ES; The weighted score of each gene is calculated based on the gene participation index PI and expression significance ES; According to the weighted score of each gene, the scores are arranged in descending order to generate a sorted list W sorted ; The top p% genes were selected according to the scores, a dynamic threshold was calculated, and the final gene list SL was filtered by the threshold, with the p% value set to 25%.

3. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 1, characterized in that: The step S2 comprises: Map key genes closely related to cancer survival to corresponding pathways to form feature representations; Processing feature representations according to a self-normalizing neural network to generate low-dimensional pathway embeddings; The low-dimensional pathway embedding is processed by Alpha Dropout technology to obtain the pathway matrix.

4. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 1, characterized in that: The cross-reconstruction fusion of gene pathway and pathological image in the first stage in step S3 includes: using a cross-modal feature recombination module to simulate the complex interaction between pathological image features and gene pathway features, and the cross-modal feature recombination module includes: Computing pathway-pathology and pathway-pathway feature interactions; The pathway-pathology and pathway-pathway feature interactions were processed by normalized exponential function and then spliced; Perform a dot product operation on the spliced ​​interaction features and the value vectors of pathways and pathologies to generate a comprehensive gene feature R F1 ; Calculate the interaction between pathological features and pathway features to obtain pathological-pathway interaction features; Perform dot product operations and concatenation operations on the pathology-pathway interaction features and pathway-pathology interaction features with the value vector respectively to obtain the second fusion feature R F2 .

5. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 4, characterized in that: The comprehensive gene signature R F1 for: R F1 =Cat(softmax(R P-H ),softmax(R P-P ))·Cat(V P ,V H ) Among them, R P-H is the characteristic of pathway-pathology interaction, R P-P is the characteristic of pathway-pathway interaction, V P is the value vector of the path, V H is the pathological value vector.

6. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 4, characterized in that: The bilinear collaborative fusion of the gene pathway and the pathological image in the second stage in step S3 includes: The cross-modal feature recombination module is used to calculate R F1 The interaction features between the pathological features are used to generate new interaction features R H-F1 and R F1-H ; R H-F1 and R F1-H Respectively with the value vector V F1 and V H Perform a dot product operation, and then perform a concatenation operation to obtain a new cross-modal fusion feature R F3 .

7. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 5, characterized in that: The step S4 comprises: The new cross-modal fusion feature R F3 The fusion feature R generated in the first stage F2 After concatenation, the intermediate feature Z is generated by linear transformation cat ; The intermediate feature Z cat The nonlinear representation is generated by Tanh and Sigmoid activation functions respectively, and then linearly transformed again after element-by-element multiplication to obtain the final fusion feature R Final .

8. The cancer prognosis method based on the staged fusion strategy of biological pathways and histopathological features according to claim 7, characterized in that: The final fusion feature R Final for: R Final =W(τ(Z cat )⊙σ(Z cat ))+b Among them, W is the weight matrix of the linear transformation, b is the bias term of the linear transformation, τ is the tanh activation function, and σ is the sigmoid activation function.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the cancer prognosis method based on the staged fusion strategy of biological pathways and tissue pathological features according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the cancer prognosis method based on a staged fusion strategy of biological pathways and histopathological features as described in any one of claims 1 to 8.

Images