A cancer survival prediction method based on graph neural network and multimodal data

Abstract

The present invention discloses a cancer survival prediction method based on graph neural network and multimodal data, comprising the following steps: S1, constructing pathological image data, gene expression data and clinical data into respective graph structure data; S2, inputting the graph structures of the three modalities into the graph neural network, using inductive graph convolution to perform information interaction within the modality, and reshaping the node lengths to 512 dimensions; S3, using attention pooling to obtain global features of 512 dimensions for each modality; S4, inputting the obtained global features of the three modalities into a hybrid multilayer perceptron to perform information interaction and fusion between the modalities; S5, inputting the global features of the three modalities after interaction and fusion into a multilayer perceptron to obtain prediction results for each modality, and taking the average value of the prediction results of each modality as the total prediction result; this method can effectively improve the survival prediction effect under multimodal data.

Description

Technical Field

[0001] The present invention relates to the field of biological technology, and in particular to a cancer survival prediction method based on graph neural networks and multimodal data. Background Art

[0002] Survival prediction refers to the use of cancer patient data or information to predict a patient's future survival risk. It can significantly assist clinical management and decision-making, enabling patients to receive more appropriate treatment and monitoring plans. In recent years, the development of pathological medical imaging and genomics has brought survival prediction research into the multimodal era. Many methods for survival prediction using multimodal data have been proposed, such as: using convolutional neural networks to extract slice features and global features of pathological images, and simply splicing them with gene features extracted using the LARS feature selection algorithm to input them into a survival prediction network composed of a multi-layer perceptron to predict the results; or using convolutional neural networks and multi-layer perceptrons to process image information and clinical and genetic information respectively, and then using maximum pooling to fuse these features; or even simply splicing gene expression data directly onto each slice feature for prediction.

[0003] The above methods fail to fully utilize multimodal medical data and have the following shortcomings: 1. They lack a unified modeling approach. Due to the significant differences in the data types of different modalities—pathology images are gigapixel-level images, clinical records are multidimensional text data, and gene expression information is tens of thousands of dimensional digital sequences—sequentially, different networks or methods (convolutional neural networks, multilayer perceptrons, LARS features) are required to process each modality, making the network overly complex and unfavorable for subsequent fusion. 2. They lack interaction between information within each modality, especially image information. The above methods all use convolutional neural networks to process image information. This is because pathology images can reach gigapixels, so images are typically cut into uniformly sized slices as input to the convolutional neural network. However, each slice actually contains different information, such as cancer cells, adipocytes, lymphocytes, etc., while the convolutional neural network uses the label of the entire image as the label for each slice, leading to a complex multi-instance problem and the fragmentation of information between slices. Furthermore, clinical and genetic features are simply spliced ​​onto the image information. Third, the fusion method is overly simplistic, crudely concatenating or pooling features from different modalities to fuse them. This fails to fully leverage the differences and complementarities between modalities to improve multimodal prediction results. Therefore, due to the aforementioned inconsistent modeling approaches, lack of information exchange within modalities, and overly crude fusion methods, it is difficult to meet user needs. Summary of the Invention

[0004] The purpose of the present invention is to provide a cancer survival prediction method based on graph neural networks and multimodal data. The cancer survival prediction method based on graph neural networks and multimodal data uses graph structures to flexibly model medical data of multiple modalities, uses graph convolution to perform information interaction within the modality, and uses a hybrid multi-layer perceptron to perform information interaction between modalities, thereby improving the survival prediction effect under multimodal data.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A cancer survival prediction method based on graph neural network and multimodal data, comprising the following steps:

[0007] S1. Construct pathological image data, gene expression data and clinical data into their own graph structure data respectively;

[0008] S2. Input the graph structures of the three modalities into the graph neural network, use inductive graph convolution to perform information interaction within the modality, and reshape the node lengths to 512 dimensions;

[0009] S3. Use attention pooling to obtain the global features of each modality with a length of 512 dimensions;

[0010] S4. Input the obtained global features of the three modalities into the hybrid multi-layer perceptron to perform information interaction and fusion between the modalities;

[0011] S5. Input the global features of the three modalities after interactive fusion into the multi-layer perceptron to obtain the prediction results of each modality, and take the average value of the prediction results of each modality as the total prediction result.

[0012] Preferably, the specific process of constructing the pathological image data into graph structure data in step S1 is:

[0013] A11. Using the RGB threshold method on the pathological image data, obtain the pathological image tissue area and background area;

[0014] A12. Cut the pathological image tissue area into non-overlapping 512*512 size slice images;

[0015] A13. Input the slice image into the KimiaNet network pre-trained by the public dataset to obtain a slice feature with a length of 1024.

[0016] A14. Use slice features to represent each graph node, and construct the edge relationship between nodes based on the eight-neighborhood spatial relationship between slices, so as to obtain a graph structure representing pathological image data.

[0017] Preferably, the specific process of constructing the gene expression data into graph structure data in step S1 is:

[0018] B11. Divide gene expression data into different gene sets based on gene function;

[0019] B12. Treat each gene set as a graph node and use full connections between the nodes to obtain a graph structure representing gene expression data.

[0020] Preferably, the gene set in step B11 includes tumor suppression, tumorigenesis, protein kinase, cell differentiation and cytokine.

[0021] Preferably, the specific process of constructing the clinical data into graph structure data in step S1 is:

[0022] C11. Numericalize clinical data and divide them into different clinical information;

[0023] C12. Treat each piece of clinical information as a graph node, and use full connections between nodes to obtain a graph structure representing clinical data.

[0024] Preferably, the clinical information in step C11 includes gender, age and BMI.

[0025] Preferably, the expression of the inductive graph convolution in step S2 is:

[0026]

[0027] Among them, x i , represents the output feature of any node i, x i represents the input feature of any node i, j∈N(i) represents all neighbor nodes of node i, x j Represents the input features of neighbor node j, W1 and W2 represent learnable weight matrices, and mean represents the average value; the inductive graph convolution guides the update of the current node features by aggregating the information of all neighbor nodes corresponding to each node. While performing information interaction within the modality, it also filters and purifies the features, thereby obtaining features of each modality with differences.

[0028] Preferably, the hybrid multi-layer perceptron in step S4 adds information fusion in the modal domain on the basis of channel domain fusion, so as to increase information interaction between the various modalities, thereby obtaining complementary features of the various modalities.

[0029] After adopting the above technical solution, the present invention has the following beneficial effects:

[0030] 1. The present invention uses graph structures to flexibly model multi-modal medical data, uses graph convolution for information interaction within the modality, and uses a hybrid multi-layer perceptron for information interaction between modalities, thereby improving the survival prediction effect under multimodal data.

[0031] 2. This invention innovatively uses graph structures to uniformly model multimodal data of different forms. It not only flexibly handles the heterogeneity of different modal data, but also brings more interpretability through the attention mechanism, which can be applied to the research of more modal data in the future.

[0032] 3. The present invention uses inductive graph convolution to perform information interaction within the modality, so that the features within each modality can interact with each other, thereby fully mining the beneficial features of each modality.

[0033] 4. The present invention uses a hybrid multi-layer perceptron to interact and fuse the features of different modalities, thereby utilizing the complementarity and differences of information from different modalities, greatly improving the prediction performance. The final prediction results are not only better than the prediction effect of using a single modality, but also better than the traditional splicing and pooling-based methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the present invention;

[0035] Figure 2 It is a flowchart of the present invention;

[0036] Figure 3 This is an example diagram of pathological image segmentation and modeling of the present invention;

[0037] Figure 4 A graph structure representing pathological image data according to the present invention;

[0038] Figure 5 A graph structure representing gene expression data according to the present invention;

[0039] Figure 6 A graph structure representing clinical data according to the present invention;

[0040] Figure 7 This is a structural diagram of the hybrid multi-layer perceptron of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Example

[0043] like Figures 1 to 7 As shown, a cancer survival prediction method based on graph neural network and multimodal data includes the following steps:

[0044] S1. Construct pathological image data, gene expression data and clinical data into their own graph structure data respectively;

[0045] The specific process of constructing the pathological image data into graph structure data in step S1 is as follows:

[0046] A11. Using the RGB threshold method on the pathological image data, obtain the pathological image tissue area and background area;

[0047] A12. Cut the pathological image tissue area into non-overlapping 512*512 size slice images;

[0048] A13. Input the slice image into the KimiaNet network pre-trained by the public dataset to obtain a slice feature with a length of 1024.

[0049] A14. Use slice features to represent each graph node, and construct edge relationships between nodes based on the eight-neighborhood spatial relationship between slices, thereby obtaining a graph structure representing pathological image data;

[0050] The specific process of converting gene expression data into graph structure data in step S1 is as follows:

[0051] B11. Divide gene expression data into different gene sets based on gene function;

[0052] The gene set in step B11 includes tumor suppression, tumorigenesis, protein kinase, cell differentiation and cytokine;

[0053] B12. Treat each gene set as a graph node and fully connect the nodes to obtain a graph structure representing gene expression data.

[0054] The specific process of building the clinical data into graph structure data in step S1 is as follows:

[0055] C11. Numericalize clinical data and divide them into different clinical information;

[0056] C12. Treat each clinical information as a graph node and use full connections between nodes to obtain a graph structure representing clinical data;

[0057] The clinical information in step C11 includes gender, age, and BMI;

[0058] S2. Input the graph structures of the three modalities into the graph neural network, use inductive graph convolution to perform information interaction within the modality, and reshape the node lengths to 512 dimensions;

[0059] The expression of the inductive graph convolution in step S2 is:

[0060]

[0061] Among them, x i , represents the output feature of any node i, x i represents the input feature of any node i, j∈N(i) represents all neighbor nodes of node i, x j represents the input features of neighbor node j, W1 and W2 represent learnable weight matrices, and mean represents the average value. The inductive graph convolution guides the update of the current node features by aggregating the information of all neighbor nodes corresponding to each node. While performing information interaction within the modality, it also filters and purifies the features, thereby obtaining features of each modality with differences.

[0062] S3. Use attention pooling to obtain the global features of each modality with a length of 512 dimensions;

[0063] S4. Input the obtained global features of the three modalities into the hybrid multi-layer perceptron to perform information interaction and fusion between the modalities;

[0064] The hybrid multilayer perceptron in step S4 adds information fusion in the modal domain on the basis of channel domain fusion, so as to increase the information interaction between the various modalities, thereby obtaining complementary features of the various modalities;

[0065] S5. Input the global features of the three modalities after interactive fusion into the multi-layer perceptron to obtain the prediction results of each modality, and take the average value of the prediction results of each modality as the total prediction result.

[0066] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A cancer survival prediction method based on graph neural network and multimodal data, characterized in that: The following steps are involved: S1. Construct pathological image data, gene expression data and clinical data into their own graph structure data respectively; S2. Input the graph structures of the three modalities into the graph neural network, use inductive graph convolution to perform information interaction within the modality, and reshape the node lengths to 512 dimensions; The expression of the inductive graph convolution in step S2 is: Among them, x i , represents the output feature of any node i, x i represents the input feature of any node i, j∈N(i) represents all neighbor nodes of node i, x j represents the input features of neighbor node j, W1 and W2 represent learnable weight matrices, and mean represents the average value. The inductive graph convolution guides the update of the current node features by aggregating the information of all neighbor nodes corresponding to each node. While performing information interaction within the modality, it also filters and purifies the features, thereby obtaining features of each modality with differences. S3. Use attention pooling to obtain the global features of each modality with a length of 512 dimensions; S4. Input the obtained global features of the three modalities into the hybrid multi-layer perceptron to perform information interaction and fusion between the modalities; The hybrid multilayer perceptron in step S4 adds information fusion in the modal domain on the basis of channel domain fusion, so as to increase the information interaction between the various modalities, thereby obtaining complementary features of the various modalities; S5. Input the global features of the three modalities after interactive fusion into the multi-layer perceptron to obtain the prediction results of each modality, and take the average value of the prediction results of each modality as the total prediction result.

2. The method for predicting cancer survival based on graph neural networks and multimodal data according to claim 1, wherein: The specific process of constructing the pathological image data into graph structure data in step S1 is as follows: A11. Using the RGB threshold method on the pathological image data, obtain the pathological image tissue area and background area; A12. Cut the pathological image tissue area into non-overlapping 512*512 size slice images; A13. Input the slice image into the KimiaNet network pre-trained by the public dataset to obtain a slice feature with a length of 1024. A14. Use slice features to represent each graph node, and construct the edge relationship between nodes based on the eight-neighborhood spatial relationship between slices, so as to obtain a graph structure representing pathological image data.

3. The method for predicting cancer survival based on graph neural networks and multimodal data according to claim 1, wherein: The specific process of converting gene expression data into graph structure data in step S1 is as follows: B11. Divide gene expression data into different gene sets based on gene function; B12. Treat each gene set as a graph node and use full connections between the nodes to obtain a graph structure representing gene expression data.

4. The method for predicting cancer survival based on graph neural networks and multimodal data according to claim 3, wherein: The gene set in step B11 includes tumor suppressor, tumorigenesis, protein kinase, cell differentiation and cytokine.

5. The method for predicting cancer survival based on graph neural networks and multimodal data according to claim 1, wherein: The specific process of building the clinical data into graph structure data in step S1 is as follows: C11. Numericalize clinical data and divide them into different clinical information; C12. Treat each piece of clinical information as a graph node, and use full connections between nodes to obtain a graph structure representing clinical data.

6. The method for predicting cancer survival based on graph neural networks and multimodal data according to claim 5, characterized in that: The clinical information in step C11 includes gender, age, and BMI.

Images