Method for constructing cancer recurrence prediction model based on multi-modal anti-noise map network
By constructing a multimodal noise-resistant graph network, the problem of insufficient integration of multimodal information in cancer recurrence prediction is solved, achieving efficient cancer recurrence prediction, improving prediction accuracy and model robustness, and enhancing interpretability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-27
AI Technical Summary
Existing cancer recurrence prediction methods lack effective integration and utilization of multimodal information, have insufficient modeling of inter-sample relationships, inadequate mining of modal interactions, and poor robustness to noise.
A multimodal noise-resistant graph network is constructed, and feature learning and fusion are performed through the multimodal graph network MMGF. Noise edges are filtered by graph convolutional network layers, feature selection-based graph fusion module and multi-head self-attention mechanism, and node representation is optimized by combining graph contrastive learning loss.
It achieves accurate modeling of complex relationships between samples, improves prediction accuracy, efficiently integrates multimodal information, has strong noise resistance and robustness, and provides a prediction framework with enhanced interpretability.
Smart Images
Figure CN121747899A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of medical artificial intelligence, and particularly relates to a construction method of a cancer recurrence prediction model based on a multi-modal anti-noise graph network. BACKGROUND
[0002] Cancer recurrence prediction is a key link in tumor treatment and follow-up management, and directly affects the selection of treatment options and prognosis evaluation of patients. Traditional methods mostly rely on clinical features or single modal data, lacking effective integration and utilization of multi-modal information. In recent years, multi-modal learning methods have gradually attracted attention in medical data analysis, but there are the following problems in cancer recurrence prediction: (1) Insufficient modeling of relationships between samples: existing methods usually process each patient sample independently, ignoring the potential similarities and differences within the patient population; (2) Insufficient modeling of modal interaction: lacking the ability to model complex dependency relationships within and between modalities; (3) Poor noise robustness: there is generally noise and missing in medical data, and existing methods lack effective noise filtering and robustness mechanisms.
[0003] There is no related report on a cancer recurrence prediction model combining multi-modal graph network, adaptive feature fusion and noise edge filtering mechanism. SUMMARY
[0004] The purpose of the embodiments of the present application is to provide a construction method of a cancer recurrence prediction model based on a multi-modal anti-noise graph network, aiming to solve the problems raised in the background.
[0005] The embodiments of the present application are implemented in the following way: the construction method of the cancer recurrence prediction model based on the multi-modal anti-noise graph network comprises the following steps: Step 1: constructing a multi-modal graph, including a PET / CT graph, a pathology graph, a gene graph and a clinical feature graph; Step 2: constructing a multi-modal graph network MMGF for feature learning and fusion of the multi-modal graph; Step 3: training the multi-modal graph network MMGF using training data to obtain a trained cancer recurrence prediction model.
[0006] Further technical solutions, the step 1 comprises the following specific steps: Step 1.1: using a pre-trained BERT model to embed the PET / CT report text into a high-dimensional vector as the node feature of the PET / CT graph; embedding the pathology examination report text into a high-dimensional vector as the node feature of the pathology graph; directly taking the gene features and clinical features as the node features of the gene graph and the clinical feature graph, respectively; Step 1.2: For PET / CT images and pathology images, cosine similarity is used to calculate the similarity between nodes; for gene images and clinical feature images, Euclidean distance is used to calculate the distance between nodes; Step 1.3: Based on the similarity or distance, connect the most similar nodes or the nearest nodes to each node to construct the edges of the graph; Step 1.4: Process the empty nodes and add self-loops, and use the Jaccard coefficient to evaluate the quality of the edges and remove noise edges.
[0007] Further technical solutions, in the step 2, the constructed multi-modal graph network MMGF includes: a graph convolution network layer for performing multi-layer graph convolution operation on each modality graph to extract node feature representation of each modality; a feature selection-based graph fusion module for fusing the feature vectors of multiple modalities extracted by the graph convolution network layer.
[0008] Further technical solutions, the fusion process of the feature selection-based graph fusion module is: In the training phase, using reparameterization technique to convert the scores of each modality feature into a probability distribution, and sampling to obtain a one-hot encoded mask : ; wherein, is a temperature parameter that controls the degree of dispersion of the distribution; is a differentiable sampling operation used to generate a one-hot mask for dimension-level modality selection; is the original prediction score without normalization; In the inference phase, the modality is selected directly by the maximum value of , that is: ; wherein, is the modality index corresponding to the maximum value, that is, each feature dimension is assigned to the modality with the highest score; According to the modality selection mask, select the features of the corresponding dimension from the feature vectors of multiple modalities, and concatenate or weight to generate the fused feature vector .
[0009] Further technical solutions, the constructed multi-modal graph network MMGF further includes a noise edge filtering graph network, which uses a multi-head self-attention mechanism to update node representation; in the first layer of the multi-head self-attention mechanism, the update process of the node representation is as follows: ;
[0010] wherein, is the node representation matrix of the th layer with size ; , and is the learnable parameter matrix of the th head; is the scaling parameter to prevent the dot product result from being too large; denotes concatenating the outputs of multiple heads together; is the linear transformation matrix after concatenation; is the output of the th attention head, wherein is the head index, which ranges from 1 to , is the total number of attention heads; is the th attention head.
[0011] After updating the node representation, the node representation is further processed by layer normalization and feed-forward network: ; ; wherein, is the output obtained by layer normalization after being processed by self-attention mechanism and feed-forward neural network in the th layer; and are the learnable parameter matrices in the feed-forward network; is the layer normalization operation; is the nonlinear activation function.
[0012] Further technical solutions, in the step 3, when training the multi-modal graph network MMGF, a graph contrastive learning loss is introduced, which is based on a multi-hop adjacency matrix and a feature similarity matrix: ; wherein, is the multi-hop adjacency matrix; is the training node index; is a minimum value for numerical stability; is the graph contrastive loss, is the feature similarity matrix.
[0013] Further technical solutions, in the step 3, in order to further optimize the graph structure, the noise density of each node is calculated, which is defined as the average value of the reliability scores of all edges of the node: ; wherein, denotes the noise density of a node ; is the reliability score of an edge ; is the number of neighbor nodes of a node ; Then, based on the node noise density and the feature similarity matrix, a pseudo-graph adjacency matrix is generated: ; wherein, is the noise density of a node , is a threshold value for the noise density, and are threshold values for the feature similarity in the case of low noise and high noise, respectively.
[0014] The method for constructing a cancer recurrence prediction model based on a multi-modal anti-noise graph network according to the embodiments of the present application has the following beneficial effects: (1) The complex relationships between samples are accurately modeled, and the prediction accuracy is significantly improved. By constructing a multi-modal graph structure to map patients as nodes and build edges according to feature similarity, and using a graph convolution network (GCN) to aggregate neighbor information, the potential associations in the patient population are deeply mined, overcoming the limitations of existing methods that isolate samples.
[0015] (2) Through an adaptive feature selection fusion mechanism, multi-modal information is efficiently integrated. A fusion module based on Gumbel-Softmax can dynamically select the optimal information source modality for each feature dimension, suppress noise or redundant features, and strengthen key discriminative information, achieving intelligent complementation and efficient utilization of multi-modal data.
[0016] (3) It has strong anti-noise and robustness. The initial noise edges are filtered by the Jaccard coefficient, and the node representation is optimized by the multi-head self-attention mechanism and graph contrastive learning loss, combined with the dynamic adjacency matrix update algorithm based on node noise density, effectively improving the generalization ability and stability of the model in real noisy medical data.
[0017] (4) It provides an end-to-end and explainability-enhanced prediction framework. The entire process is integrated into a unified deep learning model, and by analyzing the generated feature selection weights, attention scores, and optimized graph structure, it can provide potential evidence for prediction decisions, enhancing the explainability and clinical assistance value of the model. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 This is a flowchart illustrating the construction process of the multimodal graph in the method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network, as provided in an embodiment of the present invention. Figure 2 The diagram shows the framework of the multimodal graph network MMGF in the method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network provided in the embodiments of the present invention. Figure 3 The results of the algorithm comparison experiment on the DLBCL multimodal dataset are presented. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0020] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.
[0021] like Figure 1 and Figure 2 The diagram illustrates a method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network, according to an embodiment of the present invention. The method includes the following steps: Step 1: Construct a multimodal graph based on semantic extraction; First, a pre-trained BERT model is used to embed PET / CT reports and pathology reports into high-dimensional vectors, which serve as node features for PET / CT images and pathology images, respectively. Simultaneously, genetic and clinical features are directly used as node features for genetic and clinical feature images, respectively.
[0022] For each modality, nodes are initialized and assigned corresponding feature vectors. Then, based on the similarity or distance between node features, the most similar or closest nodes are selected as connection objects for each node, thus constructing the graph's edges. Specifically, PET / CT and pathology images use cosine similarity to calculate the similarity between nodes, while gene and clinical feature images use Euclidean distance to measure the distance between nodes. In this way, each modality can capture the inherent relationships between its corresponding features.
[0023] To ensure the integrity and connectivity of the graph, empty nodes were processed by connecting them to their neighbors, and self-loops were added to each node. Furthermore, to remove noisy edges, a Jaccard coefficient was introduced to evaluate edge quality, retaining only edges with high Jaccard similarity, thereby improving the quality and reliability of the graph.
[0024] Step 2: Construction of Multimodal Graph Network (MMGF); Graph Convolutional Network Layers: Each modality delves into the structured information and complex relationships between nodes within the single-modality data through graph convolution operations. In the graph convolutional layer, each node updates its feature representation by aggregating information from its neighboring nodes. Through the stacking of multiple layers of graph convolutions, the network can capture a wider range of inter-node dependencies, further enhancing the overall structural understanding of the single-modality data.
[0025] Feature-selection-based graph fusion module: used to receive feature vectors from four different modalities, namely PET / CT modalities. Pathological modalities Gene modalities as well as The shape of each feature vector is ,in It refers to the batch size. These are the feature dimensions. The goal of the module is to independently select an optimal modality for each feature dimension, thereby generating a fused feature vector. Specifically, during the training phase, use The reparameterization technique transforms the rating into a probability distribution and samples it to obtain a one-hot encoded mask. .
[0026] ; in, It is a temperature parameter that controls the degree of dispersion in the distribution; This is a differentiable sampling operation used to generate a one-hot mask for dimensional modality selection; The original, unnormalized predicted scores; In the reasoning stage, directly through The maximum value selection mode, that is: ; in, To obtain the modality index corresponding to the maximum value, each feature dimension is assigned to the modality with the highest score; Noise-side filtering graph network: Multi-head self-attention module: To address the issue of noisy edges in graph data, a multi-head self-attention mechanism is employed as the core network of the noisy edge filtering graph network. Unlike traditional local graph learning methods, the multi-head self-attention mechanism can comprehensively consider the information between nodes and capture global dependencies. In the... In the multi-head self-attention mechanism of the layer, the update process of the node representation is as follows: ; ; in, It is the first The node representation of a layer is a matrix of size ; , and is the learnable parameter matrix of the th head; is a scaling parameter to prevent the dot product result from being too large; represents concatenating the outputs of multiple heads together; is the linear transformation matrix after concatenation; is the output of the th attention head, where is the head index, taking values from 1 to , is the total number of attention heads; is the th attention head.
[0027] After updating the node representation, the node representation is further processed by layer normalization and a feedforward network: ; ; where, is the output obtained by layer normalization after processing by the self-attention mechanism and the feedforward neural network in the th layer; and are the learnable parameter matrices in the feedforward network; is the layer normalization operation; is a nonlinear activation function. Through these mechanisms, the multi-head self-attention module can effectively deal with noise problems in graph data, while capturing complex dependency relationships between nodes, improving the robustness and performance of the model.
[0028] Graph contrastive learning loss: To optimize the node representation, a graph contrastive learning loss is introduced, which is based on the multi-hop adjacency matrix and the feature similarity matrix: ; where, is the multi-hop adjacency matrix; is the index of the training node; is a small value for numerical stability; is the graph contrastive loss, is the feature similarity matrix.
[0029] Dynamic adjacency matrix update: To further optimize the graph structure, the noise density of each node is calculated, defined as the average of the reliability scores of all edges of the node: ; wherein, denotes the node noise density; is the reliability score of the edge ; and is the number of neighbor nodes of the node . Then, based on the node noise density and the feature similarity matrix, a pseudo-graph adjacency matrix is generated as follows: ; wherein, is the node noise density, is the threshold of the noise density, and are the thresholds of the feature similarity in the low noise and high noise cases, respectively.
[0030] In the embodiments of the present application, experiments and result analysis: Patients diagnosed with DLBCL (diffuse large B-cell lymphoma) in a Chinese tertiary hospital were collected, and the follow-up time was up to December 31, 2024, a total of 116 samples were collected, including PET / CT reports, pathological examination reports, gene sequencing data, clinical feature data and follow-up data.
[0031] Finally, the MMGF model was tested on the DLBCL clinical multi-modal data set by 5-fold cross-validation, and compared with five algorithms (RF, MLP, MMGL, EV-GCN and DeepKEGG). As shown in Table 1 and Figure 3 the performance results of MMGF and the comparison model are shown, and the experimental results show that MMGF is better than all comparison algorithms in terms of indicators, and exhibits excellent robustness and generalization ability.
[0032] Table 1 Algorithm comparison experimental results on the DLBCL multi-modal data set
[0033] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A method for constructing a cancer recurrence prediction model based on a multimodal robust graph network, characterized in that, Includes the following steps: Step 1: Construct a multimodal map, which includes PET / CT images, pathological images, genetic images, and clinical feature images; Step 2: Construct a Multimodal Graph Network (MMGF) for feature learning and fusion of the multimodal graph; Step 3: Use the training data to train the Multimodal Graph Network (MMGF) to obtain a trained cancer recurrence prediction model.
2. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 1, characterized in that, Step 1 includes the following specific steps: Step 1.1: Use the pre-trained BERT model to embed the PET / CT report text into a high-dimensional vector as the node feature of the PET / CT image; embed the pathology report text into a high-dimensional vector as the node feature of the pathology image; and directly use the gene features and clinical features as the node features of the gene image and clinical feature image, respectively. Step 1.2: For PET / CT images and pathology images, cosine similarity is used to calculate the similarity between nodes; for gene images and clinical feature images, Euclidean distance is used to calculate the distance between nodes. Step 1.3: Based on similarity or distance, connect each node to several of its most similar nodes or several of its closest nodes to construct the edges of the graph; Step 1.4: Process empty nodes and add self-loops, and use the Jaccard coefficient to evaluate the quality of edges and remove noisy edges.
3. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 1, characterized in that, In step 2, the constructed multimodal graph network (MMGF) includes: Graph convolutional network layers are used to perform multi-layer graph convolution operations on each modality graph to extract node feature representations for each modality; The feature selection-based graph fusion module is used to fuse feature vectors from multiple modalities extracted by graph convolutional network layers.
4. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 3, characterized in that, The fusion process of the feature selection-based graph fusion module is as follows: During the training phase, use The reparameterization technique transforms the scores of each modality feature into a probability distribution and samples them to obtain a one-hot encoded mask. : ； in, It is a temperature parameter that controls the degree of dispersion in the distribution; This is a differentiable sampling operation used to generate a one-hot mask for dimensional modality selection; The original, unnormalized predicted scores; In the reasoning stage, directly through The maximum value selection mode, that is: ； in, To obtain the modality index corresponding to the maximum value, each feature dimension is assigned to the modality with the highest score; Based on modality selection mask, features of the corresponding dimension are selected from the feature vectors of multiple modalities, and then concatenated or weighted to generate a fused feature vector. .
5. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 4, characterized in that, The constructed multimodal graph network MMGF also includes a noisy side-filtered graph network, which employs a multi-head self-attention mechanism to update node representations; in the... In the multi-head self-attention mechanism of the layer, the update process of the node representation is as follows: ； ； in, It is the first The node representation of the layer is a matrix of size . ; , and It is the first The learnable parameter matrix of the size; This is a scaling parameter used to prevent the dot product result from becoming too large; This indicates that the outputs of multiple headers are concatenated together; It is the linear transformation matrix after concatenation; For the first The output of each attention head, where This is the head index, with a value range of 1 to... , For the total number of attention heads; For the first One point of attention.
6. After updating the node representation, the node representation is further processed through layer normalization and a feedforward network: ； ； in, In the first In the layer, the output is obtained after processing by the self-attention mechanism and the feedforward neural network, and then by layer normalization. and It is the learnable parameter matrix in the feedforward network; It is a layer normalization operation; It is a non-linear activation function.
7. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 5, characterized in that, In step 3, during the training of the multimodal graph network MMGF, a graph contrastive learning loss is introduced, which is based on the multi-hop adjacency matrix and the feature similarity matrix: ； in, It is a multi-hop adjacency matrix; It is the training node index; It is a local minimum value used for numerical stability; To compare the losses, This is the feature similarity matrix.
8. The method for constructing a cancer recurrence prediction model based on a multimodal noise-resistant graph network according to claim 6, characterized in that, In step 3, to further optimize the graph structure, the noise density of each node is calculated, defined as the average of the reliability scores of all edges of that node: ； in, Represents a node The noise density; For the edge Reliability score; For nodes The number of neighboring nodes; Then, based on the node noise density and feature similarity matrix, a pseudo-graph adjacency matrix is generated. : ； in, It is a node noise density, It is the threshold of noise density. and These are the thresholds for feature similarity under low noise and high noise conditions, respectively.