Cancer patient prognosis prediction method based on graph alignment and multi-modal alignment

Abstract

The invention aims to provide a multi-modal cancer prognosis evaluation method based on a graph. The method comprises the following steps: S1, pre-processing multi-modal data, segmenting a pathological tissue part into patches, extracting patch features, and performing surrounding type composition on the patches according to cell migration continuity; the omics data is subjected to GSEA enrichment classification, wherein five cancer development processes are included; s2, single-modal features are obtained, and modal corresponding features are obtained by using a modal enhanced and shared graph mask auto-encoder; s3, cross-modal interaction is carried out, and multi-modal data distribution mapping alignment is realized by using a cross-modal alignment module; s4, performing multi-modal fusion, and fusing the cross-modal interaction feature and each single-modal feature by using a Hyperdge-Mixer module to obtain a modal fusion feature; and S5, risk prediction: inputting the final multi-modal representation into a risk prediction module, and finally obtaining a risk prediction result. The cancer prognosis is realized by effectively fusing pathology, genes and clinical data, an end-to-end prognosis analysis framework is realized by utilizing a deep learning technology, the method has good performance and relatively high interpretability, and the clinical application practicability is enhanced.

Description

Technical Field

[0001] This application is directed to the technical field of cancer survival analysis and prognosis prediction, and in particular to a method for predicting the prognosis of cancer patients based on a multimodal model using graph alignment. Background Art

[0002] The tumor microenvironment (TME) is a critical environment for tumor survival and affects tumor progression and prognosis. Due to the heterogeneity of tumors, patients' treatment responses and prognostic effects vary, and the overall prognosis is poor. In this case, the survival prognosis assessment score is helpful in assisting doctors' judgment and is crucial for developing personalized treatment. By analyzing the patient's survival time and disease status, doctors can more accurately predict the patient's disease stage in order to choose appropriate treatment decisions, which can not only improve the treatment effect, but also more reasonably optimize the allocation of medical resources.

[0003] Tumor heterogeneity is a characteristic of cancer characterization and a problem that needs to be solved in studying the causes and development of cancer. Different triggers cause multiple tissues to induce tumors, and multiple subtypes will also appear. There are too many unknowns about the causes of the induction process. Combined with individual differences in patients, tumor heterogeneity will appear in digital pathology images and gene expression. To explore the complex structure and morphological characteristics of pathological tissues, it is necessary to clarify the relationship between cell groups in the tissue and analyze the connection between tumor heterogeneity and cellular processes. Existing work is to segment tumor tissues, focus on local high-interest areas, or use global tissue segmentation to divide into patches and extract tissue block features. The current method lacks the connection between tissue blocks in pathological images and cannot capture the pathological microenvironment in the tissue. The formation of tumors is also related to the innate determination of genes. It is necessary to diagnose and treat cancer at multiple levels and angles. Gene expression can be combined with the phenotypic information provided by tissue pathology images to link phenotypes with genes. It is possible to mine tumor triggers at the genetic level and capture interpretable analysis in the tumor microenvironment. In previous method processes, there are limitations in interpretability, which obviously cannot bring into play the potential value between modal features.

[0004] The progress of deep learning in cancer pathology has enabled different modal data to be applied to the diagnosis and prognosis prediction of various cancer types. However, the use of whole-slice images alone to analyze patient prognosis still has certain limitations, and the existing multimodal models mostly ignore the spatial position association between image blocks when extracting digital pathology image features, and cannot obtain all the information about the tumor. Although models that combine multimodal data such as genome, transcriptome, and clinical characteristics with digital pathology image features have greater potential clinical value, these multimodal models do not solve the heterogeneity and missing modality problems of multimodal data, resulting in low prediction performance. Therefore, the technical problem solved by this application is that the existing multimodal models have incomplete tumor information obtained by digital pathology image features, heterogeneity and missing modalities in multimodal data, resulting in low prediction accuracy of multimodal models.

[0005] 1) The heterogeneity problem between multiple modalities will cause dimensionality explosion or data ratio problems when fusion occurs, which requires the use of higher-performance equipment or more complex models to process, making training more difficult;

[0006] 2) In multimodal fusion, how to make the data of each modality merge naturally requires a suitable interaction module, otherwise the abrupt fusion will lead to the collapse of the model;

[0007] 3) Therefore, it is necessary to use an alignment method to add interaction between modalities to the existing method, so that the data contains part of the information of each other's modalities before fusion, which is conducive to the fusion of multimodal data. Summary of the invention

[0007] In order to solve the current technical limitations, the present invention provides a method for predicting the prognosis of cancer patients based on a graph-aligned multimodal model, which can combine digital pathological features, molecular features, and clinical information features. For the first time, the concept of graph alignment is introduced into multimodal prognostic test research to achieve the approximation of different representation distributions, to achieve semantic alignment of common representation mappings, and to combine representations with genes to improve accuracy; at the same time, when there is a modality deletion, other modality data can be used to obtain risk assessment scores, so that multimodal features can interact and complement each other. Improving the accuracy of patient survival prediction is conducive to accurate prognosis.

[0008] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present application provides a method for predicting the prognosis of cancer patients based on graph alignment multimodality, the method comprising: Multimodal data preprocessing: pathological tissue parts are divided into patches, patch features are extracted, and patches are constructed in a surrounding manner according to the continuity of cell migration. The omics data is enriched and classified by GSEA, including five cancer development processes; Single-modal features, using modality enhancement and shared image mask autoencoders to obtain modality-corresponding features; Cross-modal interaction, using the cross-modal alignment module to achieve multi-modal data distribution mapping alignment; Multimodal fusion: Use the Hyperedge-Mixer module to fuse cross-modal interaction features and single-modal features to obtain modal fusion features; Risk prediction,the final multimodal representation is input into the risk prediction module,and combined with the modal decision, the risk prediction result is finally obtained.

[0010] Further, according to the multimodal cancer prognosis assessment method based on graph alignment summarized in the claim, it is characterized in that Figure 2 Step 1 specifically includes obtaining a digital pathology image and performing preprocessing such as feature extraction to obtain pathology image data, including: Segment the tissue part of the above pathological image to obtain a 256×256 tissue patch image block and the position of the patch block in the image; Input the above segmented patches into KimiaNet for feature extraction to obtain the tensor expressing each patch; According to the idea of ​​cancer invasion, the patches around each patch with a step radius of 1 are connected to construct the pathological tissue microenvironment network and form the pathological map data;

[0011] Furthermore, the Figure 2 Step 2 specifically includes obtaining omics data and performing preprocessing such as feature extraction, which is characterized in that: obtaining molecular feature map data, including: The above sequencing data were normalized using the StandardScaler method to obtain standard sequencing data; GSEA was used to classify cancer-related functions of the above standard sequencing data, and 5 sequencing groups were obtained; According to the relationship between the functions of cancer-related genes, the grouping tensors are connected to construct the omics functional gene network and form the molecular feature map data;

[0012] according to Figure 2 Step 3 specifically includes obtaining clinical data and performing preprocessing such as feature extraction, and is characterized in that obtaining clinical image data includes: The clinical data obtained above is split into survival-related data and patient vital sign data, and the vital sign data is one-hot encoded and zero-filled to form a tensor; Since the complete individual vital signs data are closely related, we connect each tensor to build a clinical vital signs network and form clinical graph data;

[0013] The method according to claim x, characterized in that the method further comprises: Acquiring diagnostic data of various cancers; the diagnostic data includes digital pathology images, gene data, and patient characteristic data; Screening each of the diagnostic cohorts based on preset sample inclusion criteria to obtain sample data; Preprocessing the sample data to obtain training data; and converting the survival-related data into label data of survival time and status; Constructing a training model, inputting the training data and label data into the training model until the loss function of the training model reaches a stable value, thereby obtaining the trained multimodal model;

[0014] The method according to claim x is characterized in that the screening of the diagnostic data based on the preset sample inclusion criteria to obtain the sample data comprises: The diagnostic data of the same cancer are used as a diagnostic cohort, and the pathological, omics and physical image data are matched and combined with the survival-related data to form a complete heterogeneous image data cohort; The sample data is divided into a plurality of heterogeneous graph data queues according to different cancer types to obtain the sample data.

[0015] The method according to claim 3, characterized in that the screening of each of the diagnostic cohorts based on the preset sample inclusion criteria to obtain multiple sample cohorts includes: In each of the heterogeneous data queues, if the data volume is less than a threshold or there is a large amount of disturbance and the data cannot be cleaned, the queue is partially deleted; Using the remaining heterogeneous data queues as the sample queues respectively;

[0016] In the first part of the model, a graph mask mechanism is introduced to construct a graph mask autoencoder for each modality to perform feature selection and dimensionality reduction and obtain intra-modal representation, which increases the data interaction within the modality and makes the selected features more stable through mask reconstruction. A shared graph mask autoencoder is also used to extract the common data of each modality using a shared encoder, while retaining the characteristics of each modality itself using the mask reconstruction mechanism.

[0017] In the second part of the model, an alignment module is introduced to determine the distribution of multimodal data, and intentionally align the distribution of the remaining modalities to the pathological data, indirectly completing the guidance of the pathological data to other modal data, so as to facilitate the fusion of co-expression features of multimodal data in the next stage.

[0018] In the third part of the model, the modality deletion module is introduced to achieve the first mutual fusion of different modality data. It mainly focuses on the node fusion between different modalities, and uses modality masks to simulate the situation of cancer data deletion. After the initial fusion, it helps the interaction between modality data, and can also generate modal compensation for deletion. On the premise of improving the accuracy, it can also judge the survival of patients with deletion with high accuracy.

[0019] In the fourth part of the model, the MLP-Mixer module is introduced and improved to the Hyperedge-Mixer module according to the graph data, which supplements the fusion of the relationship between the modalities and once again increases the data interaction between the modalities.

[0020] In the fifth part of the model, the multimodal fusion features are used to output the votes of each modality on survival, and the final votes are fused as the final prognostic result.

[0021] In summary, compared with the prior art, the technical solution provided in the embodiments of the present application has the following beneficial effects: The embodiment of the present application provides a method for predicting the prognosis of cancer patients based on a graph alignment multimodal model. The method inputs graph structure data formed by preprocessed digital pathology images, omics sequencing and clinical record data into the constructed multimodal model, and through the graph mask autoencoder and shared graph mask autoencoder corresponding to each modality, each modal feature is stably and effectively extracted, the interaction within each modality is increased, the information transmission within a single modality is strengthened, and more attention is paid to the multimodal common features; through the alignment module, the distribution of common features found by different modalities is approximated, the common features are accurately identified, and the multimodal feature alignment is realized; through the initial The secondary fusion module is used to compensate for missing modalities and interact with nodes between modalities; the interaction between modalities is realized through the Hyperedge-Mixer module; the fusion decision is converted into a prognostic result through the COX risk module; the complete cancer data and relevant prior knowledge are used to efficiently classify the data so that more effective and accurate focus points can be obtained when multimodal features are fused, which not only improves the accuracy of predictive risk assessment, but also realizes that in the case of missing data, more accurate risk scores can be achieved through the existing modality data, thereby increasing the usability of this application in clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A flowchart of a method for predicting the prognosis of cancer patients based on graph multimodal alignment is provided for one embodiment of the present application.

[0023] Figure 2 A diagram of the training structure of a multimodal alignment model provided for one embodiment of the present application.

[0024] Figure 3 A diagram of a model structure based on graph multimodal alignment provided for one embodiment of the present application.

[0025] Figure 4 A model provided in one embodiment of the present application is used to visually display the results of cancer prognosis prediction. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the present application, rather than all examples.

[0027] All other embodiments obtained by ordinary technicians in this field based on the embodiments in this application without creative work are within the scope of protection of this application.

[0028] See also Figure 1 and Figure 2 The embodiment of the present application provides a method for predicting the prognosis of cancer patients based on a graph alignment multimodal model, the method comprising the following steps, the method specifically comprising the following steps:

[0029] Step S1, multimodal data preprocessing, dividing the pathological tissue part into patches, extracting patch features, and surrounding the patches according to the continuity of cell migration. The omics data was enriched and classified by GSEA, including five cancer development processes;

[0030] The digital pathology image refers to the pathology image of the tissue stained by hematoxylin / eosin staining. The preprocessing of the digital pathology feature image to be analyzed includes: segmenting the digital pathology image to be analyzed, dividing it into 256×256 patches according to the size of the tissue and the location information of each patch. Some pathology images will have color labeling pollution when entering data. The color threshold difference is established for filtering to eliminate most of the pollution disturbances. According to the returned segmentation visualization map, the disturbance patch is manually removed to ensure the validity of the data. The obtained patch is downsampled to obtain a sampled image patch that is magnified 40 times and the location of the patch in the complete tissue. Each sampled image patch is input into KimiaNet for feature extraction to obtain the pathology image patch feature information data.

[0031] According to the characteristics of cancer cells in metastasis and invasion and their different communication functions with other cells, a single patch block should be surrounded by a static map of the cancer cell movement process, which shows that the patches around each patch block are related. In order to construct the relationship within the modality and simulate the cell movement process in the cancer microenvironment, the tensor of the patch block is used as the node data, and the saved position information is connected in a surrounding way with a step length of 1 to form a COO sparse matrix of edge_index, which is finally converted into edge set data. The two are combined to form the complete pathology map data.

[0032] The omics data is preprocessed into a data format that can be used for multimodal cancer prognosis assessment based on graph alignment, which is characterized by specifically obtaining omics data and performing preprocessing such as feature extraction to obtain molecular feature graph data, including:

[0033] The above sequencing data were normalized using the StandardScaler method to obtain standard sequencing data;

[0034] GSEA was used to classify cancer-related functions of the above standard sequencing data, and 5 sequencing groups were obtained;

[0035] According to the relationship between the functions of cancer-related genes, the grouping tensors are connected to construct the omics functional gene network and form the molecular feature map data;

[0036] The survival analysis requires a detailed matching sample of clinical records, which is characterized by a specific description of the transformation of clinical records and follow-up data and preprocessing such as feature extraction to obtain clinical graph data, including:

[0037] The clinical data matching the digital pathology and omics data are split into survival-related follow-up data and patient vital sign record data. The vital sign data is one-hot encoded and converted into vectors of 1 and 0. Since the data volume is quite different from that of digital pathology data and omics data, zero padding is used to fill the size gap and form a tensor.

[0038] Since the complete individual sign data are closely related and can be used as edge sets, each independent sample is connected to its own tensor to construct a clinical sign network and form clinical graph data;

[0039] The method according to claim x, characterized in that the method further comprises:

[0040] Acquire diagnostic data of multiple cancers; the diagnostic data includes digital pathology images, gene data and patient characteristic data; screen each diagnostic cohort based on preset sample inclusion criteria to obtain sample data; preprocess the sample data to obtain training data; and convert survival-related data into label data of survival time and status; construct a training model, input the training data and label data into the training model until the loss function of the training model reaches a stable value, and obtain the trained multimodal model;

[0041] The data constructed above are further combined into a data set, characterized in that the diagnostic data is screened based on a preset sample inclusion standard to obtain sample data, including:

[0042] The diagnostic data of the same cancer are used as a diagnostic cohort, and the pathological, omics and physical sign map data are matched and combined with survival-related data to form a complete heterogeneous graph data cohort; the data are divided into multiple heterogeneous graph data cohorts according to different cancer types to obtain the sample data.

[0043] The above standard screening method is further used to implement cohort screening, characterized in that each of the diagnostic cohorts is screened based on the preset sample inclusion criteria to obtain multiple sample cohorts, including:

[0044] In each of the heterogeneous data queues, if the data volume is less than a threshold or there is a large amount of disturbance and the data cannot be cleaned, the queue is partially deleted; and the remaining heterogeneous data queues are respectively used as the sample queues;

[0045] A graph-based multimodal cancer patient prognosis prediction model, characterized in that the modules of the model include:

[0046] The acquisition module is used to acquire the patient's digital pathology images, omics sequencing and clinical data, and preprocess them to obtain training data;

[0047] An input module, used to divide a part of the training data into analysis data and input the data into a trained multimodal model; the multimodal model includes a modality enhancement module, a modality alignment module, a fusion module and a risk assessment module;

[0048] The modality enhancement module is used to realize intra-modality enhancement features of different modalities and inter-modality shared enhancement features through modality-specific and modality-shared graph mask autoencoders;

[0050] The graph autoencoder used in it, combined with the masking mechanism, reasonably masks the nodes or edges of the graph data, realizes the destruction of the original data, and increases the robustness of the model to cope with disturbances through reconstruction. It is also possible to extract more important related features based on the missing data of multiple modalities.

[0051] The modality alignment module is used to achieve distribution convergence of shared features of different modalities through a discriminator, achieve semantic alignment, and capture co-expression features;

[0052] A discriminator module is used to project the shared features extracted from each multimodality of the modal-shared graph mask autoencoder into a common space, and then adjust the distribution of shared features of each modality to fuse node data.

[0053] The fusion module is used to fuse pathological image features, molecular features and clinical features to obtain multimodal fusion features;

[0054] The Hyper-Mixer module, which is improved from MLP-Mixer, is used to fuse the edge sets of modal mixed features that are the combination of modal specific features and modal shared features between node relationships.

[0055] The risk assessment module is used to predict the multimodal fusion features to obtain the risk decision of each modality, and to obtain the predicted risk assessment score of the patient by combining each decision.

[0056] The predicted risk assessment score is a score that predicts the prognosis of cancer patients.

[0057] The above embodiment provides a method for predicting the prognosis of cancer patients based on graph multimodal alignment. The method processes the preprocessed patient's multiple cancer-related data, and after processing, the spatial information in the digital pathology data and the association information between the data can be processed, so that the data contains complete information in the tumor tissue; input into the constructed multimodal prediction model, and extract specific features and shared features through the modality-specific and modality-shared graph mask autoencoders in the multimodal model; through the modality sharing alignment module, the modality sharing features are distributed and aligned to solve the heterogeneity between modalities, and the modality data node fusion and the data interaction within the modality are realized; the hyperedge fusion module is used to further fuse the association information to realize the data interaction between modalities; finally, the heterogeneous data is effectively fused so that the multimodal data can be jointly analyzed for risk analysis; at the same time, in actual applications, the modality data measured by the patient is not complete, and most of the time, data deletion will occur. This example also establishes a modality masking mechanism, and through the modality deletion supplement module, the reconstruction of the masked modality is realized to cope with the situation of modality data deletion, so as to enhance the usability of this application in actual clinical applications.

[0058] In one example, the above-mentioned acquisition of the patient's digital pathological image to be analyzed and preprocessing to obtain pathological image data includes: The digital pathology image to be analyzed is segmented to obtain multiple segmentation patches.

[0059] The segmented patch blocks are input into KimiaNet for feature extraction to obtain the node features of the pathological image patch blocks and the location of the patch on the complete tissue.

[0060] Specifically, the digital pathology images to be analyzed are generally saved in SVS format. Through the Histolab library or OpenSlide library, the foreground and background of the pathology image tissue can be distinguished. The preset segmentation patch size is 256×256, and the tissue is segmented in a grid manner. The position coordinates of the upper left corner of the patch are selected for saving. KimiaNet, a model pre-trained using a variety of TCGA cancer pathology images, is used to extract 1024-dimensional feature vectors from the segmented patch blocks using the pre-trained KimiaNet model to obtain pathology image node data; the feature extraction formula is shown as follows: N path =KimiaNet(X path ) Among them, KimiaNet(*) is a model that uses a stacked DenseNet graph structure trained and fine-tuned using a large amount of TCGA pathology data. path Represents the set of segmented patches, that is, the matrix set of patches composed of pixels; N pathIt represents the feature vector set after the feature extraction of the patch block. The set size is N×1024, where N represents the number of segmented patches and 1024 represents the extracted feature dimension of 1024, which is the digital pathology image node dataset. At the same time, according to the characteristics of graph neural networks, information can be transmitted, which is conducive to the realization of global exploration of nodes, which may be helpful for simulating the spread and metastasis of cancer cells. The location information of the above-mentioned retained patch blocks is used in a coordinate system with a single patch as the origin to find several patches with a step radius of 1 patch length to construct edge sets. The digital pathology map data is constructed by combining the node set and the edge set. The formula for constructing the map data is as follows: E path =COO([[Index(coords orip ),Index(coords aroundp )]…]) Among them, COO represents the conversion of matched patch number pairs into a sparse adjacency matrix, Index represents the number index set for each patch according to the coordinate position, and coords orip Represents the position of the origin of the current patch, coords aroundp Represents the position with a radius of 1 unit patch length near the origin, coords aroundp The maximum value is 8. G path =Data(N path .E path ,Other path ) Where N path Represents the vector set extracted from the patch block, that is, the node set; E path Represents the set of connection information obtained from the location information of the patch block, that is, the edge set; Other path Represents possible attribute features about nodes and edges, etc.

[0061] In some embodiments, the above-mentioned steps of obtaining the patient's gene data to be analyzed and preprocessing the data to obtain the genome data include: The StandardScaler method was used for preprocessing to normalize the gene data to be analyzed and obtain genome data.

[0062] Among them, StandardScaler is a function of the sklearn package in the standard library in Python, and the function can be directly referenced to preprocess data.

[0063] The above embodiment pre-processes the digital pathology images, gene data and clinical feature data to be analyzed, so as to convert them into a format that can be accepted and processed by the model, and the risk assessment score can be obtained through the trained model.

[0064] For details, see Figure 2 In an embodiment, the method further comprises: Step S21, obtaining diagnostic data and historical record data of various cancers; diagnostic data includes digital pathological images and gene data; historical records include current age, patient ID, survival time, survival status, etc.

[0065] Step S22, after obtaining the sample, set up standards to filter the diagnostic data and historical record data, and match them to obtain standardized sample data.

[0066] Step S23, preprocessing the standardized sample data and distinguishing them to obtain training samples and test samples.

[0067] Among them, the purpose of screening the diagnostic data is to filter out invalid data from different batches of data. The method of preprocessing the sample data is consistent with the preprocessing method of the digital pathology image data, gene data and some historical record data to be analyzed. The digital pathology image is also patched and the features of the DenseNet model are extracted as pathology image node data, and then the pathology map data is constructed according to the pathology edge set construction method; the gene data is normalized, and then the genes are grouped into cancer process group nodes through GSEA, and then linked to each other to form gene map data; for clinical characteristics, they are first encoded to meet the preprocessing size, and then the operation process consistent with the gene data is used to realize the construction of clinical map data.

[0068] Use training samples to train the model, input the training sample set into the model pre-training until the loss function of the model during training reaches the expected value, and obtain the trained multimodal model parameters.

[0069] Specifically, the preprocessed data is passed through a data receiving and feature extraction module consisting of a graph mask autoencoder for extracting three specific features and a graph mask autoencoder for extracting shared features; the extracted features are then input into the alignment enhancement module and the modality deletion and complementation module to achieve feature alignment and interaction between modalities; the data of the two modules are then fused into a hypergraph to achieve hyperedge fusion and increase the connection between modal data; finally, the COX risk model is continuously optimized to achieve convergence and finally complete the training,

[0070] The model framework and training process are as follows:

[0071] M = {p, c, g} is the modal set of data, p represents pathological data, c represents clinical data, and g represents genetic data. p =(V p ,E p ), G g =(V g ,E g ), G c =(V c ,E c ) represents the graph structure data constructed by each modality of the data; first, the graph data is input into the data receiving and feature extraction module; Before inputting to the module, the nodes are sampled using mask processing and selected as a mask subset, where The node vector is destroyed by using [MASK] and other similar mask tags. Then the node representation after mask preprocessing is: Through the information transmission of the graph neural network, the masked part can be restored through the adjacent nodes. In the process of removing some perturbed data by using the reconstruction comparison method, the data within the modality interacts with each other. resample the subset of With input mask node set Then, the mask is chosen for each node with the same probability, regardless of whether the masked part was previously masked or not. At the same time, the expression matrix of the destroyed By Node and shared mask flag [DMASK] instead of H i Build: The constructed data passes through this module: Immediately above, the decoder can be Reconstruct the input X. This process is repeated multiple times to generate a different set of remasked nodes and the corresponding destruction expression Each random probability sample contains different information and is re-masked, and they are forced to reconstruct the function of the input node. The loss function of the first part is obtained:

[0072] The next step is to split the enhanced features into two data streams; the first data stream is the data from the shared feature module, which is input into the modality alignment module, where the discriminator is used: where g i and g j It belongs to the constructed graph structure data; i∈{p},j∈{c,g} represents that the model aligns the data of other modalities according to the data distribution of pathological images. Finally, the loss function of this part is:

[0073] The second data stream is the data from the special feature module, which is input into the modal deletion and complement module to set up a modal mask and simulate the situation of modal deletion. This can enhance the data mining capability itself and increase the interaction between data. Mask m ={[p],[c],[g],[p,c],[p,g],[c,g].[p,c,g]} Among them, M n It represents the mode that is retained after masking part of the mode, and [] represents the mask of the modal data; Among them, VisionTransformer is improved based on modality specificity; Token is to tokenize the modality that needs to be masked into the same size.

[0074] Before the final super-edge fusion, and Combine and integrate the data into mixed graph data with alignment and specific representation, the formula is as follows:

[0075] MLP-Mixer is used to adaptively adjust the graph data, and the data node feature matrix is ​​transposed twice to achieve HyperEdge mixing. The first transposition is equivalent to integrating part of the information of each node into a new feature vector, and the relationship between them has changed, from strengthening the local feature connection to the global feature connection; the second transposition transfers the data back to the original node, but the node now contains global information. After a fusion block, local interactions guided by global information can be obtained, which complements the interaction between modalities and can use data information more effectively. The last step is to combine the outputs of different modes. Input into the risk prediction model, each modality outputs a risk prediction value based on itself combined with other modalities, and finally the risk prediction values ​​are combined: in, Refers to the weight parameter of the risk prediction scoring layer output after learning fusion. is the bias parameter of the learning risk prediction layer, and sigmoid(·) is the nonlinear activation function in the risk prediction scoring module. Score hazard It is the result of the pre-training model prediction, and is used to score the predicted risk of cancer in the formal application after the pre-training is completed.

[0076] Compare the prediction results output by the pre-trained model with the survival time of the training data, and optimize the loss function of the pre-trained model according to the comparison results until it is optimized to the preset value, then the training can be ended and the pre-trained model can be used as the trained multimodal model: Among them, the preset value of the loss function L is: Among them, L surv The survival loss function is used, α, β and γ are hyperparameters, and N is the number of prediction samples.

[0077] The above embodiment provides a training process based on graph multimodal alignment, wherein the effective training of the feature-specific extraction module, shared modality extraction module, modality deletion and supplementation module, modality alignment module and fusion module composed of the graph mask autoencoder ensures the accuracy of the model in predicting risk scores.

[0078] In the above embodiment, the above data needs to be screened based on the standard samples, divided into training samples, verification samples and test samples, and finally a grouped data sample set is obtained, including: The data are classified and arranged according to the type of cancer to form a cohort marked by tumor type, and the data in each cohort are matched, that is, the digital pathology images, genetic data and historical follow-up record data of the same patient with the same cancer are matched one by one, and files with damaged files or too little modal data are eliminated.

[0079] The specific limitations of the method for predicting the prognosis of cancer patients based on graph multimodal alignment provided in this embodiment can be found in the above embodiment of a method for predicting the prognosis of cancer patients based on graph multimodal alignment, which will not be repeated here. Each module in the above-mentioned method for predicting the prognosis of cancer patients based on a multimodal model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can realize the analysis and visualization of input data on a computer, such as Figure 4 In this paper, the risk area division of pathological images is achieved after joint analysis of multimodal data.

[0080] The process, working details and technical effects of the running model of the computer device provided in this embodiment can be found in the above embodiment of a method for predicting the prognosis of cancer patients based on a multimodal model, and will not be described in detail here.

[0081] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multimodal cancer prognosis assessment method based on graph alignment, characterized in that: The following steps are involved: S1. Preprocessing of multimodal data: dividing the pathological tissue into patches, extracting patch features, and surrounding the patches according to the continuity of cell migration. The omics data was enriched and classified by GSEA, including five cancer development processes; S2, single modal features, using modality enhancement and shared image mask autoencoders to obtain modality corresponding features; S3, cross-modal interaction, using the cross-modal alignment module to achieve multi-modal data distribution mapping alignment; S4, multimodal fusion, using the Hyperedge-Mixer module to fuse cross-modal interaction features and single-modal features to obtain modal fusion features; S5. Risk prediction: the final multimodal representation is input into the risk prediction module and combined with the modal decision to finally obtain the risk prediction result.

2. The multimodal cancer prognosis assessment method based on graph alignment as summarized in claim 1, characterized in that: The step S1 specifically includes acquiring a digital pathological image and performing preprocessing such as feature extraction to obtain pathological image data, including: Segment the tissue part of the above pathological image to obtain a 256×256 tissue patch image block and the position of the patch block in the image; Input the above segmented patches into KimiaNet for feature extraction to obtain the tensor expressing each patch; According to the idea of ​​cancer invasion, patches with a step radius of 1 around each patch are connected to construct a pathological tissue microenvironment network and form pathological map data.

3. The multimodal cancer prognosis assessment method based on graph alignment as summarized in claim 1, characterized in that: The step S2 specifically includes acquiring omics data and performing preprocessing such as feature extraction to obtain molecular feature map data, including: The above sequencing data were normalized using the StandardScaler method to obtain standard sequencing data; GSEA was used to classify cancer-related functions of the above standard sequencing data, and 5 sequencing groups were obtained; According to the relationship between cancer-related gene functions, the grouping tensors are connected to construct the omics functional gene network and form the molecular feature map data.

4. The multimodal cancer prognosis assessment method based on graph alignment as summarized in claim 1, characterized in that: The step S3 specifically includes obtaining clinical data and performing preprocessing such as feature extraction to obtain clinical image data, including: The clinical data obtained above is split into survival-related data and patient vital sign data, and the vital sign data is one-hot encoded and zero-filled to form a tensor; Since the complete individual vital signs data are closely related, we connect each tensor to build a clinical vital signs network and form clinical graph data.

5. The method according to claims 2, 3 and 4, characterized in that: The method further comprises: Acquiring diagnostic data of various cancers; the diagnostic data includes digital pathology images, gene data, and patient characteristic data; Screening each of the diagnostic cohorts based on preset sample inclusion criteria to obtain sample data; Preprocessing the sample data to obtain training data; and converting the survival-related data into label data of survival time and status; A training model is constructed, and the training data and label data are input into the training model until the loss function of the training model reaches a stable value, thereby obtaining the trained multimodal model.

6. The method according to claim 5, characterized in that The screening of the diagnostic data based on the preset sample inclusion criteria to obtain sample data includes: The diagnostic data of the same cancer are used as a diagnostic cohort, and the pathological, omics and physical image data are matched and combined with the survival-related data to form a complete heterogeneous image data cohort; The sample data is divided into a plurality of heterogeneous graph data queues according to different cancer types to obtain the sample data.

7. The method according to claim 6, characterized in that The diagnostic cohorts are screened based on the preset sample inclusion criteria to obtain multiple sample cohorts, including: In each of the heterogeneous data queues, if the data volume is less than a threshold or there is a large amount of disturbance and the data cannot be cleaned, the queue is partially deleted; and the remaining heterogeneous data queues are respectively used as the sample queues.

8. A graph-based multimodal cancer patient prognosis prediction model, characterized in that: The modules of the model include: An acquisition module is used to acquire digital pathological images, omics sequencing and clinical data of patients, and preprocess them to obtain training data; an input module is used to divide a part of the training data into analysis data and input it into the trained multimodal model; the multimodal model includes a modality enhancement module, a modality alignment module, a fusion module and a risk assessment module; The modality enhancement module is used to realize intra-modality enhancement features of different modalities and inter-modality shared enhancement features through modality-specific and modality-shared graph mask autoencoders; The modality alignment module is used to achieve distribution convergence of shared features of different modalities through the discriminator module, achieve semantic alignment, and capture co-expression features; The fusion module is used to fuse pathological image features, molecular features and clinical features to obtain multimodal fusion features; The risk assessment module is used to predict the multimodal fusion features to obtain the risk decision of each modality, and to obtain the predicted risk assessment score of the patient by combining each decision.

9. A computer-readable storage method, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images