A multimodal survival prediction method based on graph enhancement and bipolar decoupling

Abstract

This invention proposes a multimodal survival prediction method based on graph enhancement and bipolar decoupling. The method includes: acquiring original pathological images, genomic data, and survival outcome labels, and preprocessing them separately; constructing a local spatial graph and a global semantic graph, and using the confidence scores of the two graphs to perform weighted aggregation of graph neighborhoods; decoupling the enhanced pathological image block features and gene features into positive and negative polarities and dividing them into consistency groups and complementarity groups, calculating the attention weights within the consistency group and complementarity group respectively, and performing weighted aggregation and residual connections; inputting the data into a feature refinement network for refinement, and performing global mean pooling and concatenation to obtain aggregated features; inputting the aggregated features into a survival risk prediction classification head to obtain the conditional hazard rate of the samples. This invention, through cross-modal semantic association and feature mapping, effectively maintains the independent feature expression space of each modality while achieving deep information complementarity.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of computer vision, artificial intelligence and bioinformatics, and in particular to a multimodal survival prediction method based on graph augmentation and bipolar decoupling. Background Technology

[0002] Multimodal survival prediction has significant clinical implications in precision oncology, assessing patient prognostic risk by integrating pathological images and genomic data. Pathological images provide detailed morphological information at the cellular and tissue architecture levels, reflecting the complexity of the tumor microenvironment. Genomic data captures biological characteristics closely related to tumor development at the molecular level. By synergistically processing information from these two modalities, a comprehensive assessment of patient status can be achieved, providing crucial scientific evidence for personalized treatment planning and clinical prognostic management.

[0003] However, existing prediction methods still face many technical limitations in practical applications. First, pathological images are generally filled with background noise and interference from non-diagnostic regions, and existing models are insufficient in establishing spatial continuity and semantic correlation between local image features, making it difficult to effectively filter noise and capture complex regional interaction information. Second, due to significant differences in semantic structure and statistical distribution between different modalities, feature representations within a single modality are highly entangled, making it difficult to cope with significant feature heterogeneity between samples. Finally, most existing cross-modal interaction strategies focus on overall feature alignment, ignoring the complementary information and potential conflict signals unique to different modalities, and lack in-depth mining and decoupling modeling of complex interaction features, thus limiting the prediction accuracy of the models in complex clinical prognostic scenarios. Summary of the Invention

[0004] In view of the above situation, the main objective of this invention is to propose a multimodal survival prediction method based on graph augmentation and bipolar decoupling to solve the above-mentioned technical problems.

[0005] This invention proposes a multimodal survival prediction method based on graph augmentation and bipolar decoupling, the method comprising the following steps: Step 1: Construct a prediction model based on the confidence-aware graph enhancement module, expert-guided decoupling module, bipolar feature interaction network, feature refinement network, and survival risk prediction classification head; Step 2: Obtain the original pathological images, genomic data, and survival outcome labels, and preprocess them respectively to obtain the pathological image block feature set, gene features, and discretized survival labels; Step 3: Input the pathological image block feature set into the confidence-aware graph enhancement module. Based on the spatial coordinates of each pathological image block feature in the feature set and the semantic similarity between features, construct a local spatial graph and a global semantic graph respectively. Then, use the confidence scores of the local spatial graph and the global semantic graph to perform weighted aggregation of the graph neighborhood to obtain the enhanced pathological image block features. Step 4: Input the enhanced pathological image block features and gene features into the bipolar feature interaction network. Through the expert-guided decoupling module, the enhanced pathological image block features and gene features are decoupled into positive and negative polarities and divided into consistency groups and complementarity groups. The attention weights in the consistency group and complementarity group are calculated and weighted aggregation and residual connection are performed to obtain the interactively updated pathological image block features and interactively updated gene features. Step 5: Input the interactively updated pathological image patch features and the interactively updated gene features into the feature refinement network for refinement, and perform global mean pooling and concatenation to obtain aggregated features; input the aggregated features into the survival risk prediction classification head to obtain the conditional hazard rate of the sample; construct a negative log-likelihood loss function based on the discretized survival label and the conditional hazard rate of the sample to optimize the prediction model and obtain the optimized prediction model; use the optimized prediction model to make predictions to obtain the final prediction result.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention adopts a dual-branch parallel interactive fusion strategy, which effectively maintains the independent feature expression space of each modality while achieving deep information complementarity through cross-modal semantic association and feature mapping. 2. This invention constructs a confidence-aware graph enhancement module, which, while explicitly capturing the spatial and semantic associations of pathological image blocks, effectively suppresses the interference of background noise in pathological images through adaptive confidence assessment. 3. This invention designs a statistically perceptive hybrid expert network, which drives adaptive matching of activation paths through feature distribution information, effectively alleviating the significant feature heterogeneity problem within a single modality. 4. This invention proposes a cross-modal bipolar interaction paradigm, which realizes in-depth mining and decoupling modeling of intermodal cooperative patterns and differential conflict signals by constructing consistency groups and complementarity groups.

[0007] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0008] Figure 1 This is a flowchart of a multimodal survival prediction method based on graph augmentation and bipolar decoupling proposed in this invention; Figure 2 This is a diagram illustrating the overall framework of a multimodal survival prediction method based on graph augmentation and bipolar decoupling proposed in this invention. Figure 3 This is a schematic diagram of the confidence perception map enhancement module proposed in this invention; Figure 4 This is a schematic diagram of the structure of the expert-guided decoupling module proposed in this invention; Figure 5 This is a schematic diagram of the bipolar feature interaction network structure proposed in this invention. Detailed Implementation

[0009] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0010] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0011] Please see Figure 1 This embodiment provides a multimodal survival prediction method based on graph augmentation and bipolar decoupling, the method comprising the following steps: Step 1: Construct a prediction model based on the confidence-aware graph enhancement module, expert-guided decoupling module, bipolar feature interaction network, feature refinement network, and survival risk prediction classification head.

[0012] Step 2: Obtain the original pathological images, genomic data, and survival outcome labels, and preprocess them respectively to obtain the pathological image block feature set, gene features, and discretized survival labels.

[0013] Please see Figure 2 In step 2, the original pathological images, genomic data, and survival outcome labels are acquired and preprocessed to obtain pathological image patch feature sets, gene features, and discretized survival labels. This includes the following sub-steps: Acquire raw pathological images, genomic data, and survival outcome labels; After removing background noise from the original pathological image using the Otsu thresholding method, the image is then divided into non-overlapping pathological image blocks at a preset magnification to obtain a set of pathological image blocks. A pre-trained visual Transformer network is used to encode each pathological image patch in the pathological image patch set into a fixed-dimensional feature vector to obtain the pathological image patch feature set. ; in, Represents a feature set of pathological image blocks. Indicates the first Features of a pathological image patch Indices representing pathological image patches. Indicates the number of pathological image blocks; The genomic data was divided into six functional subgroups according to biological function. An independent self-normalized neural network was constructed as an encoder for each functional subgroup to extract features, resulting in six functional subgroup features. The features of the six functional subgroups were then stacked to obtain the gene features. The self-normalized neural network consists of a linear layer and a nonlinear activation layer. The continuous survival timescales in the survival outcome labels are discretized into a certain number of non-overlapping time intervals, and mapped to corresponding discrete time indices according to the patient's actual survival status to obtain discrete survival labels.

[0014] Step 3: Input the pathological image block feature set into the confidence-aware graph enhancement module. Based on the spatial coordinates of each pathological image block feature in the feature set and the semantic similarity between features, construct a local spatial graph and a global semantic graph respectively. Then, use the confidence scores of the local spatial graph and the global semantic graph to perform weighted aggregation of the graph neighborhood to obtain the enhanced pathological image block features.

[0015] Please see Figure 3 In step 3, the pathological image block feature set is input into the confidence-aware map enhancement module. Based on the spatial coordinates of each pathological image block feature in the feature set and the semantic similarity between features, a local spatial map and a global semantic map are constructed respectively. The confidence scores of the local spatial map and the global semantic map are used to perform weighted aggregation of the graph neighborhood to obtain the enhanced pathological image block features. The specific steps include the following: The feature set of pathological image patches is input into the confidence-aware map enhancement module, and the physical spatial coordinates of each pathological image patch in the set are recorded in the original pathological image to obtain the set of physical spatial coordinates of the pathological image patches. ; in, A set of physical space coordinates representing a patch of pathological image. Indicates the first The physical spatial coordinates of a pathological image block; Based on the set of physical spatial coordinates of pathological image blocks, the physical distance between each pathological image block is calculated; for each pathological image block, a preset number of pathological image blocks with the shortest physical distance are selected from the remaining pathological image blocks as spatial neighbors to establish adjacency relationships and construct a local spatial map. Based on the feature set of pathological image blocks, the semantic similarity between each pathological image block is calculated; for each pathological image block, a preset number of pathological image blocks with the highest semantic similarity are selected from the remaining pathological image blocks as semantic neighbors to establish adjacency relationships and construct a global semantic graph. The consistency between the features of each pathological image patch and the features of its neighbors within the graph structure is calculated, and then mapped to a confidence score using the Softmax activation function. Neighborhood aggregation features are calculated in the local spatial map and the global semantic map respectively, and the feature difference between the neighborhood aggregation features and the original features is calculated as the enhancement signal. The feature difference is weighted and scaled using the corresponding confidence scores, and the enhancement signal is accumulated onto the original features to obtain the enhanced pathological image block features.

[0016] Based on the set of physical spatial coordinates of pathological image blocks, the physical distance between each pathological image block is calculated. For each pathological image block, a predetermined number of pathological image blocks with the shortest physical distance are selected from the remaining pathological image blocks as spatial neighbors to establish adjacency relationships and construct a local spatial graph. The following relationship exists in the corresponding process: ; in, Indicates the first The pathological image block and the first Euclidean distance between pathological image blocks Represents the L2 norm. Indicates the first The physical spatial coordinates of a pathological image block; In the steps of calculating the semantic similarity between pathological image blocks based on the feature set of pathological image blocks, and selecting a predetermined number of pathological image blocks with the highest semantic similarity from the remaining pathological image blocks as semantic neighbors to establish adjacency relationships and construct a global semantic graph, the following relationship exists in the corresponding process: ; in, Indicates the first The pathological image block and the first Semantic similarity between pathological image patches Indicates the first Features of a single pathological image block; In the step of calculating the consistency between the features of each pathological image patch and the features of its neighbors within the graph structure, and mapping this consistency to a confidence score using the Softmax activation function, the following relationship exists in the corresponding process: ; in, Indicates the degree of consistency. Represents a node The neighboring nodes, Represents the set of neighboring edges. Represents an exponential function. Represents a node Pathological image patch features, Represents a node Pathological image patch features, Indicates matrix transpose. The dimension of the feature vector. Represents a node Neighborhood aggregation characteristics Represents a node The confidence score, This indicates that the process has been handled by the Sigmoid function. Represents a logarithmic function; It should be noted that a higher degree of consistency indicates that the features of the pathological image patch are more reliable, and the confidence score obtained by mapping is higher, thereby achieving explicit suppression of low-reliability noise signals.

[0017] In the steps of calculating the neighborhood aggregation features under the local spatial map and the global semantic map respectively, and calculating the feature difference between the neighborhood aggregation features and the original features as the enhancement signal; using the corresponding confidence scores to weight and scale the feature difference, and accumulating the enhancement signal onto the original features to obtain the enhanced pathological image patch features, the following relationship exists in the corresponding process: ; in, Represents a node Enhanced pathological image patch features Representing nodes in a local spatial graph The confidence score, Represents nodes in a local spatial graph Neighborhood aggregation characteristics Represents nodes in the global semantic graph The confidence score, Represents nodes in the global semantic graph The neighborhood aggregation feature.

[0018] Step 4: Input the enhanced pathological image block features and gene features into the bipolar feature interaction network. Through the expert-guided decoupling module, the enhanced pathological image block features and gene features are decoupled into positive and negative polarities and divided into consistency groups and complementarity groups. The attention weights in the consistency group and complementarity group are calculated and weighted aggregation and residual connection are performed to obtain the interactively updated pathological image block features and interactively updated gene features.

[0019] Please see Figure 4 and Figure 5 In step 4, the enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module decouples the enhanced pathological image patch features and gene features into positive and negative polarities and divides them into consistency and complementarity groups. Attention weights within the consistency and complementarity groups are calculated, and weighted aggregation and residual connections are performed to obtain the interactively updated pathological image patch features and interactively updated gene features. Specifically, the steps include the following: The enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module is used to perform bipolar decoupling on the enhanced pathological image patch features, yielding positive and negative pathological features. Similarly, the expert-guided decoupling module is used to perform bipolar decoupling on the gene features, yielding positive and negative gene features. Specifically, the bipolar decoupling of the enhanced pathological image patch features includes: using the enhanced pathological image patch features as positive input to decouple positive pathological features, and taking the negative value of the enhanced pathological image patch features as negative input to decouple negative pathological features. The bipolar decoupling of the gene features includes: using the gene features as positive input to decouple positive gene features, and taking the negative value of the gene features as negative input to decouple negative gene features. Pathological positive polarity features, pathological negative polarity features, gene positive polarity features, and gene negative polarity features are divided into two cross-modal interaction groups according to their polarity attributes. Feature pairs with consistent polarity, namely pathological positive polarity features and gene positive polarity features, and pathological negative polarity features and gene negative polarity features, are divided into a consistency group. Feature pairs with opposite polarity, namely pathological positive polarity features and gene negative polarity features, and pathological negative polarity features and gene positive polarity features, are divided into a complementarity group. The sum of the dot products of two feature pairs within the consistency group is calculated separately, and then processed by a normalized exponential function to obtain the consistency attention weight matrix; the sum of the dot products of two feature pairs within the complementarity group is calculated separately, and then processed by a normalized exponential function to obtain the complementarity attention weight matrix. The consistent attention weight matrix and the complementary attention weight matrix are added together, multiplied by the linear projection features of the gene features, and accumulated to the gene features to obtain the interactively updated gene features. The transpose of the consistent attention weight matrix and the transpose of the complementary attention weight matrix are added together, multiplied by the linear projection features of the enhanced pathological image patch features, and accumulated to the enhanced pathological image patch features to obtain the interactively updated pathological image patch features.

[0020] The enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module performs bipolar decoupling on the enhanced pathological image patch features to obtain positive and negative pathological features. Similarly, the expert-guided decoupling module performs bipolar decoupling on the gene features to obtain positive and negative gene features. The following relationship exists in this process: ; in, Indicates the positive polarity characteristic of pathology. This refers to expert-guided decoupling operations performed on pathological modalities. Indicates the features of pathological image blocks. Indicates a pathological negative polarity characteristic. Indicates the positive polarity characteristic of a gene. This indicates an expert-guided decoupling operation performed on a gene modality. Indicates gene characteristics, This indicates a negative polarity characteristic of the gene; In the steps of calculating the sum of the dot products of two feature pairs within the consistency group, and then processing the sum of the dot products of the two feature pairs within the consistency group through a normalized exponential function to obtain the consistency attention weight matrix; and in the steps of calculating the sum of the dot products of two feature pairs within the complementarity group, and then processing the sum of the dot products of the two feature pairs within the complementarity group through a normalized exponential function to obtain the complementary attention weight matrix, the following relationship exists in the corresponding process: ; in, This represents the consistency attention weight matrix. This indicates that the data has been processed using the Softmax function. This represents the complementary attention weight matrix.

[0021] The steps of adding the consistent attention weight matrix and the complementary attention weight matrix, multiplying them with the linear projection features of the gene features, and accumulating the result to obtain the interactively updated gene features; and adding the transpose of the consistent attention weight matrix and the transpose of the complementary attention weight matrix, multiplying them with the linear projection features of the enhanced pathological image patch features, and accumulating the result to obtain the interactively updated pathological image patch features, involve the following relationships: ; in, Represents pathological feature sequences, The projected features represent the original features of a pathological image obtained through linear mapping. Represents the characteristic sequence of a gene. The projected features represent the original characteristics of a gene obtained through linear mapping.

[0022] It should be noted that the expert-guided decoupling module is designed to input features and their statistical distribution information into a statistically perceptual router to match the optimal expert for that feature. Decoupling representation of the feature is achieved through nonlinear transformation. The expert-guided decoupling module contains a hybrid expert network with three parallel decoupling expert branches. Each decoupling expert consists of a specific activation function and a fully connected layer, used to provide differentiated nonlinear transformation paths for features with different distributions. The three decoupling expert networks use ReLU, ELU, and GELU activation functions, respectively. A statistically perceptual router based on a fully connected layer structure is constructed, and the original features and their statistical distribution information are concatenated and input into the statistically perceptual router to calculate the selection score of each decoupling expert branch. The statistical distribution information includes the mean and variance of the feature. The statistically perceptual router adopts a hard routing strategy, that is, based on the highest selection score, it uniquely selects a matching expert branch from the hybrid expert network to decouple the original feature. Figure 4 In this context, the decoupling feature is the output feature obtained by bipolar decoupling.

[0023] Step 5: Input the interactively updated pathological image patch features and the interactively updated gene features into the feature refinement network for refinement, and perform global mean pooling and concatenation to obtain aggregated features; input the aggregated features into the survival risk prediction classification head to obtain the conditional hazard rate of the sample; construct a negative log-likelihood loss function based on the discretized survival label and the conditional hazard rate of the sample to optimize the prediction model and obtain the optimized prediction model; use the optimized prediction model to make predictions to obtain the final prediction result.

[0024] In step 5, the interactively updated pathological image patch features and interactively updated gene features are respectively input into the feature refinement network for refinement, and global mean pooling and concatenation are performed to obtain aggregated features. The aggregated features are then input into the survival risk prediction classification head to obtain the conditional hazard rate of the sample. The specific steps include the following: The interactively updated pathological image patch features and interactively updated gene features are respectively input into the feature refinement network. The survival signal is refined through normalization, upsampling, nonlinear mapping and downsampling operations to obtain the refined global features of the pathological image and the refined global features of the gene. After performing global mean pooling on the refined global features of the pathological image and the refined global features of the gene, a splicing operation is performed to obtain the aggregated features. The aggregated features are input into a survival risk prediction classification head composed of fully connected layers for processing to obtain the conditional hazard rate of the sample in multiple preset discrete time intervals. Based on the conditional hazard rates corresponding to each interval, the cumulative survival probability of a sample remaining alive within a discrete time interval is derived.

[0025] The interactively updated gene features and pathological image patch features are respectively input into the refinement network. Through normalization, upsampling, nonlinear mapping, and downsampling operations, the survival signal is refined to obtain the refined global features of the pathological image and the refined global features of the gene. The following relationship exists in the correspondence process: ; in, This represents the global features of the refined pathological image. This indicates that the data has been processed by a downsampling linear layer. This indicates that the data has been processed by the GELU function. This indicates that the data has been processed by an upsampling linear layer. This indicates that the process has undergone a layer normalization process. This represents the global characteristics of the refined gene; After performing global mean pooling on the refined global features of the pathological image and the refined global features of the gene, a concatenation operation is performed to obtain the aggregated features. The following relationship exists in the corresponding process: ; in, Indicates aggregation features, This indicates that a splicing operation has been performed. This indicates that global mean pooling has been performed. The aggregated features are input into a survival risk prediction classification head composed of fully connected layers for processing to obtain the conditional hazard rate of the sample in multiple preset discrete time intervals. The following relationship exists in the corresponding process: ; in, Indicates the characterization of the first The sample has survived to the [number]th [number]. Given the conditions at the beginning of a discrete time interval, the conditional hazard rate of the observed event occurring within the current interval. Represents the observed survival time variable. Indicates the corresponding first A discrete time interval Indicates the first A set of pathological image patch features for each sample. Indicates the first Genetic characteristics of each sample Indicates a discrete-time interval index; Based on the conditional hazard rates corresponding to each interval, the sample is derived. In the The cumulative survival probability of surviving within a discrete time interval is related to the following formula: ; in, Indicates the first The sample at the th The cumulative survival probability of still being alive within a discrete time interval. This indicates a series of multiplication operations. Indicates the first The sample at the th Survival probability within a discrete time interval This represents the time interval index variable in a multiplication operation; A negative log-likelihood loss function is constructed based on discretized survival labels and conditional hazard rates. The following relationship exists in the corresponding process: ; in, This represents the negative log-likelihood loss function. Represents the total number of samples. Indicates the first Censorship indicator variable for each sample, Indicates the first The sample at the th Conditional hazard rate for a real discrete time interval Indicates the first The sample at the th The cumulative survival probability of surviving within a given discrete time interval. Indicates the first The sample at the th The cumulative survival probability of surviving within a given discrete time interval. Indicates the first The index of the true discrete time interval corresponding to each sample.

[0026] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0027] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0028] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multimodal survival prediction method based on graph augmentation and bipolar decoupling, characterized in that, The method includes the following steps: Step 1: Construct a prediction model based on the confidence-aware graph enhancement module, expert-guided decoupling module, bipolar feature interaction network, feature refinement network, and survival risk prediction classification head; Step 2: Obtain the original pathological images, genomic data, and survival outcome labels, and preprocess them respectively to obtain the pathological image block feature set, gene features, and discretized survival labels; Step 3: Input the pathological image block feature set into the confidence-aware graph enhancement module. Based on the spatial coordinates of each pathological image block feature in the feature set and the semantic similarity between features, construct a local spatial graph and a global semantic graph respectively. Then, use the confidence scores of the local spatial graph and the global semantic graph to perform weighted aggregation of the graph neighborhood to obtain the enhanced pathological image block features. Step 4: Input the enhanced pathological image block features and gene features into the bipolar feature interaction network. Through the expert-guided decoupling module, the enhanced pathological image block features and gene features are decoupled into positive and negative polarities and divided into consistency groups and complementarity groups. The attention weights in the consistency group and complementarity group are calculated and weighted aggregation and residual connection are performed to obtain the interactively updated pathological image block features and interactively updated gene features. Step 5: Input the interactively updated pathological image patch features and the interactively updated gene features into the feature refinement network for refinement, and perform global mean pooling and concatenation to obtain aggregated features; The aggregated features are input into the survival risk prediction classification head to obtain the conditional hazard rate of the sample; a negative log-likelihood loss function is constructed based on the discretized survival label and the conditional hazard rate of the sample to optimize the prediction model, resulting in an optimized prediction model; the optimized prediction model is then used to make predictions to obtain the final prediction result.

2. The multimodal survival prediction method based on graph augmentation and bipolar decoupling according to claim 1, characterized in that, In step 2, the original pathological images, genomic data, and survival outcome labels are acquired and preprocessed to obtain pathological image patch feature sets, gene features, and discretized survival labels. Specifically, this includes the following steps: Acquire raw pathological images, genomic data, and survival outcome labels; After removing background noise from the original pathological image using the Otsu thresholding method, the image is then divided into non-overlapping pathological image blocks at a preset magnification to obtain a set of pathological image blocks. A pre-trained visual Transformer network is used to encode each pathological image patch in the pathological image patch set into a fixed-dimensional feature vector to obtain the pathological image patch feature set. ; in, Represents a feature set of pathological image blocks. Indicates the first Features of a pathological image patch Indices representing pathological image patches. Indicates the number of pathological image blocks; The genomic data was divided into six functional subgroups according to biological function. An independent self-normalized neural network was constructed as an encoder for each functional subgroup to extract features, resulting in six functional subgroup features. The features of the six functional subgroups were then stacked to obtain the gene features. The self-normalized neural network consists of a linear layer and a nonlinear activation layer. The continuous survival timescales in the survival outcome labels are discretized into a certain number of non-overlapping time intervals, and mapped to corresponding discrete time indices according to the patient's actual survival status to obtain discrete survival labels.

3. The multimodal survival prediction method based on graph augmentation and bipolar decoupling according to claim 2, characterized in that, In step 3, the pathological image block feature set is input into the confidence-aware graph enhancement module. Based on the spatial coordinates of each pathological image block feature in the feature set and the semantic similarity between features, a local spatial graph and a global semantic graph are constructed respectively. The confidence scores of the local spatial graph and the global semantic graph are used to perform weighted aggregation of the graph neighborhood to obtain the enhanced pathological image block features. The specific steps include the following: The feature set of pathological image patches is input into the confidence-aware map enhancement module, and the physical spatial coordinates of each pathological image patch in the set are recorded in the original pathological image to obtain the set of physical spatial coordinates of the pathological image patches. ; in, A set of physical space coordinates representing a patch of pathological image. Indicates the first The physical spatial coordinates of a pathological image block; Based on the set of physical spatial coordinates of pathological image blocks, the physical distance between each pathological image block is calculated; for each pathological image block, a preset number of pathological image blocks with the shortest physical distance are selected from the remaining pathological image blocks as spatial neighbors to establish adjacency relationships and construct a local spatial map. Based on the feature set of pathological image blocks, the semantic similarity between each pathological image block is calculated; for each pathological image block, a preset number of pathological image blocks with the highest semantic similarity are selected from the remaining pathological image blocks as semantic neighbors to establish adjacency relationships and construct a global semantic graph. The consistency between the features of each pathological image patch and the features of its neighbors within the graph structure is calculated, and then mapped to a confidence score using the Softmax activation function. Neighborhood aggregation features are calculated in the local spatial map and the global semantic map respectively, and the feature difference between the neighborhood aggregation features and the original features is calculated as the enhancement signal. The feature difference is weighted and scaled using the corresponding confidence scores, and the enhancement signal is accumulated onto the original features to obtain the enhanced pathological image block features.

4. The multimodal survival prediction method based on graph augmentation and bipolar decoupling according to claim 3, characterized in that, Based on the set of physical spatial coordinates of pathological image blocks, the physical distance between each pathological image block is calculated. For each pathological image block, a predetermined number of pathological image blocks with the shortest physical distance are selected from the remaining pathological image blocks as spatial neighbors to establish adjacency relationships and construct a local spatial graph. The following relationship exists in the corresponding process: ； in, Indicates the first The pathological image block and the first Euclidean distance between pathological image blocks This indicates taking the 2-norm. Indicates the first The physical spatial coordinates of a pathological image block; Based on the feature set of pathological image blocks, the semantic similarity between each pathological image block is calculated. For each pathological image block, a predetermined number of pathological image blocks with the highest semantic similarity are selected from the remaining pathological image blocks as semantic neighbors to establish adjacency relationships and construct a global semantic graph. The following relationship exists in the corresponding process: ； in, Indicates the first The pathological image block and the first Semantic similarity between pathological image patches Indicates the first Features of a single pathological image block; The consistency between the features of each pathological image patch and the features of its neighbors within the graph structure is calculated, and then mapped to a confidence score using the Softmax activation function. The following relationship exists in the mapping process: ； in, Indicates the degree of consistency. Represents a node The neighboring nodes, Represents the set of neighboring edges. Represents an exponential function. Represents a node Pathological image patch features, Represents a node Pathological image patch features, Indicates matrix transpose. The dimension of the feature vector. Represents a node Neighborhood aggregation characteristics Represents a node The confidence score, This indicates that the process has been handled by the Sigmoid function. Represents a logarithmic function; Neighborhood aggregation features are calculated separately for the local spatial map and the global semantic map. The feature difference between the neighborhood aggregation features and the original features is then calculated as an enhancement signal. The feature difference is weighted and scaled using the corresponding confidence scores, and the enhancement signal is accumulated onto the original features to obtain the enhanced pathological image patch features. The following relationship exists in the corresponding process: ； in, Represents a node Enhanced pathological image patch features Represents nodes in a local spatial graph The confidence score, Represents nodes in a local spatial graph Neighborhood aggregation characteristics Represents nodes in the global semantic graph The confidence score, Represents nodes in the global semantic graph The neighborhood aggregation feature.

5. The multimodal survival prediction method based on graph augmentation and bipolar decoupling according to claim 4, characterized in that, In step 4, the enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module decouples the enhanced pathological image patch features and gene features into positive and negative polarities and divides them into consistency and complementarity groups. Attention weights within the consistency and complementarity groups are calculated, and weighted aggregation and residual connections are performed to obtain the interactively updated pathological image patch features and interactively updated gene features. Specifically, the steps include the following: The enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module is used to perform bipolar decoupling on the enhanced pathological image patch features, yielding positive and negative pathological features. Similarly, the expert-guided decoupling module is used to perform bipolar decoupling on the gene features, yielding positive and negative gene features. Specifically, the bipolar decoupling of the enhanced pathological image patch features includes: using the enhanced pathological image patch features as positive input to decouple positive pathological features, and taking the negative value of the enhanced pathological image patch features as negative input to decouple negative pathological features. The bipolar decoupling of the gene features includes: using the gene features as positive input to decouple positive gene features, and taking the negative value of the gene features as negative input to decouple negative gene features. Pathological positive polarity features, pathological negative polarity features, gene positive polarity features, and gene negative polarity features are divided into two cross-modal interaction groups according to their polarity attributes. Feature pairs with consistent polarity, namely pathological positive polarity features and gene positive polarity features, and pathological negative polarity features and gene negative polarity features, are divided into a consistency group. Feature pairs with opposite polarity, namely pathological positive polarity features and gene negative polarity features, and pathological negative polarity features and gene positive polarity features, are divided into a complementarity group. The sum of the dot products of two feature pairs within the consistency group is calculated separately, and then processed by a normalized exponential function to obtain the consistency attention weight matrix; the sum of the dot products of two feature pairs within the complementarity group is calculated separately, and then processed by a normalized exponential function to obtain the complementarity attention weight matrix. The consistent attention weight matrix and the complementary attention weight matrix are added together, multiplied by the linear projection features of the gene features, and accumulated to the gene features to obtain the interactively updated gene features. The transpose of the consistent attention weight matrix and the transpose of the complementary attention weight matrix are added together, multiplied by the linear projection features of the enhanced pathological image patch features, and accumulated to the enhanced pathological image patch features to obtain the interactively updated pathological image patch features.

6. The multimodal survival prediction method based on graph enhancement and bipolar decoupling according to claim 5, characterized in that, The enhanced pathological image patch features and gene features are input into a bipolar feature interaction network. An expert-guided decoupling module performs bipolar decoupling on the enhanced pathological image patch features to obtain positive and negative pathological features. Similarly, the expert-guided decoupling module performs bipolar decoupling on the gene features to obtain positive and negative gene features. The following relationship exists in this process: ； in, Indicates the positive polarity characteristic of pathology. This refers to expert-guided decoupling operations performed on pathological modalities. Indicates the features of pathological image blocks. Indicates a pathological negative polarity characteristic. Indicates the positive polarity characteristic of a gene. This indicates an expert-guided decoupling operation performed on a gene modality. Indicates gene characteristics, This indicates a negative polarity characteristic of the gene; In the steps of calculating the sum of the dot products of two feature pairs within the consistency group, and then processing the sum of the dot products of the two feature pairs within the consistency group through a normalized exponential function to obtain the consistency attention weight matrix; and in the steps of calculating the sum of the dot products of two feature pairs within the complementarity group, and then processing the sum of the dot products of the two feature pairs within the complementarity group through a normalized exponential function to obtain the complementary attention weight matrix, the following relationship exists in the corresponding process: ； in, This represents the consistency attention weight matrix. This indicates that the data has been processed using the Softmax function. This represents the complementary attention weight matrix.

7. The multimodal survival prediction method based on graph enhancement and bipolar decoupling according to claim 6, characterized in that, The steps of adding the consistent attention weight matrix and the complementary attention weight matrix, multiplying them with the linear projection features of the gene features, and accumulating the result to obtain the interactively updated gene features; and adding the transpose of the consistent attention weight matrix and the transpose of the complementary attention weight matrix, multiplying them with the linear projection features of the enhanced pathological image patch features, and accumulating the result to obtain the interactively updated pathological image patch features, involve the following relationships: ； in, Represents pathological feature sequences, The projected features represent the original features of a pathological image obtained through linear mapping. Represents the characteristic sequence of a gene. The projected features represent the original characteristics of a gene obtained through linear mapping.

8. The multimodal survival prediction method based on graph enhancement and bipolar decoupling according to claim 7, characterized in that, In step 5, the interactively updated pathological image patch features and the interactively updated gene features are respectively input into the feature refinement network for refinement, and global mean pooling and concatenation are performed to obtain aggregated features. The aggregated features are then input into the survival risk prediction classification head to obtain the conditional hazard rate of the sample. Specifically, this includes the following steps: The interactively updated pathological image patch features and interactively updated gene features are respectively input into the feature refinement network. The survival signal is refined through normalization, upsampling, nonlinear mapping and downsampling operations to obtain the refined global features of the pathological image and the refined global features of the gene. After performing global mean pooling on the refined global features of the pathological image and the refined global features of the gene, a splicing operation is performed to obtain the aggregated features. The aggregated features are input into a survival risk prediction classification head composed of fully connected layers for processing to obtain the conditional hazard rate of the sample in multiple preset discrete time intervals. Based on the conditional hazard rates corresponding to each interval, the cumulative survival probability of a sample remaining alive within a discrete time interval is derived.

9. The multimodal survival prediction method based on graph augmentation and bipolar decoupling according to claim 8, characterized in that, The interactively updated pathological image patch features and interactively updated gene features are input into a feature refinement network. Through normalization, upsampling, nonlinear mapping, and downsampling operations, the survival signal is refined to obtain the refined global features of the pathological image and the refined global features of the gene. The following relationship exists in this process: ； in, This represents the global features of the refined pathological image. This indicates that the data has been processed by a downsampling linear layer. This indicates that the data has been processed by the GELU function. This indicates that the data has been processed by an upsampling linear layer. This indicates that the process has undergone a layer normalization process. This represents the global characteristics of the refined gene; In the process of performing global mean pooling on the refined global features of the pathological image and the refined global features of the gene, and then concatenating them to obtain the aggregated features, the following relationship exists: ； in, Indicates aggregation features, This indicates that a splicing operation has been performed. This indicates that global mean pooling has been performed. In the step of inputting aggregated features into a survival risk prediction classification head composed of fully connected layers to obtain the conditional hazard rate of a sample within multiple preset discrete time intervals, the following relationship exists in the corresponding process: ； in, Indicates the characterization of the first The sample has survived to the [number]th [number]. Given the conditions at the beginning of a discrete time interval, the conditional hazard rate of the observed event occurring within the current interval. Represents probability operators, Represents the observed survival time variable. Indicates the corresponding first A discrete time interval Indicates the first A set of pathological image patch features for each sample. Indicates the first Genetic characteristics of each sample Indicates a discrete-time interval index; In the step of deriving the cumulative survival probability of a sample within a discrete time interval based on the conditional hazard rate corresponding to each interval, the following relationship exists in the corresponding process: ； in, Indicates the first The sample at the th The cumulative survival probability of still being alive within a discrete time interval. This indicates a series of multiplication operations. Indicates the first The sample at the th Survival probability within a discrete time interval This represents the time interval index variable in a multiplication operation.

10. The multimodal survival prediction method based on graph enhancement and bipolar decoupling according to claim 9, characterized in that, A negative log-likelihood loss function is constructed based on discretized survival labels and conditional hazard rates. The following relationship exists in the corresponding process: ； in, This represents the negative log-likelihood loss function. Represents the total number of samples. Indicates the first Censorship indicator variable for each sample, Indicates the first The sample at the th Conditional hazard rate for a real discrete time interval Indicates the first The sample at the th The cumulative survival probability of surviving within a given discrete time interval. Indicates the first The sample at the th The cumulative survival probability of surviving within a given discrete time interval. Indicates the first The index of the true discrete time interval corresponding to each sample.

Images