Multimodal survival prediction method and apparatus

Abstract

The application discloses a multimodal survival prediction method and device, applied to the technical field of data processing, and the method comprises the steps of obtaining pathological images and genomic data of a cancer patient; using a preset survival prediction model to perform survival prediction on the pathological images and the genomic data, and obtaining the survival rate of the cancer patient in a future time period; in the prediction model, an isolated processing structure extracts features from the pathological images and the genomic data to obtain pathological isolated feature sequences and genomic isolated feature sequences, an interaction module interacts the two isolated feature sequences to obtain pathological interaction feature sequences and genomic interaction feature sequences, a cooperative processing structure extracts features from the two interaction feature sequences to obtain pathological cooperative feature sequences and genomic cooperative feature sequences, and a fusion prediction module performs fusion type survival prediction based on the two cooperative feature sequences to obtain the survival rate of the cancer patient in the future time period. The application improves the accuracy of survival prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a multi-modal survival prediction method and device. BACKGROUND

[0002] Survival prediction aims to evaluate the relative death risk in cancer prognosis. In the related art, the survival rate of a cancer patient in a future time period is usually predicted by using pathological and genomic data of the cancer patient, so as to realize survival prediction. However, the related art often relies on an existing attention mechanism to integrate features of the pathological and genomic data, and it is difficult to effectively fuse and coordinate multi-modal omics data, thereby reducing the accuracy of survival prediction. SUMMARY

[0003] Embodiments of the present application provide a multi-modal survival prediction method and device for improving the accuracy of survival prediction.

[0004] In one aspect, the present application provides a multi-modal survival prediction method, comprising the following steps:

[0005] Obtaining pathological images and genomic data of a cancer patient;

[0006] Using a preset survival prediction model to perform survival prediction on the pathological images and the genomic data, to obtain a survival rate of the cancer patient in a future time period;

[0007] The survival prediction model comprises:

[0008] An isolation processing structure for performing feature extraction on the pathological images and the genomic data to obtain a pathological isolation feature sequence and a genomic isolation feature sequence;

[0009] An interaction module for performing feature interaction on the pathological isolation feature sequence and the genomic isolation feature sequence to obtain a pathological interaction feature sequence and a genomic interaction feature sequence;

[0010] A cooperative processing structure for performing feature extraction on the pathological interaction feature sequence and the genomic interaction feature sequence to obtain a pathological cooperative feature sequence and a genomic cooperative feature sequence;

[0011] A fusion prediction module for performing fusion-type survival prediction based on the pathological cooperative feature sequence and the genomic cooperative feature sequence to obtain the survival rate of the cancer patient in the future time period.

[0012] In another aspect, the present application provides a multi-modal survival prediction device, comprising:

[0013] acquire a pathology image and genomic data of a cancer patient;

[0014] perform survival prediction on the pathology image and the genomic data by using a preset survival prediction model to obtain a survival rate of the cancer patient in a future time period;

[0015] The survival prediction model comprises:

[0016] perform feature extraction on the pathology image and the genomic data by using an isolation processing structure to obtain a pathology isolation feature sequence and a genomic isolation feature sequence;

[0017] perform feature interaction on the pathology isolation feature sequence and the genomic isolation feature sequence by using an interaction module to obtain a pathology interaction feature sequence and a genomic interaction feature sequence;

[0018] perform feature extraction on the pathology interaction feature sequence and the genomic interaction feature sequence by using a cooperative processing structure to obtain a pathology cooperative feature sequence and a genomic cooperative feature sequence;

[0019] perform fusion-type survival prediction based on the pathology cooperative feature sequence and the genomic cooperative feature sequence by using a fusion prediction module to obtain the survival rate of the cancer patient in the future time period.

[0020] The application has the beneficial effect of providing a multi-modal survival prediction method and device, acquiring a pathology image and genomic data of a cancer patient, and performing survival prediction on the pathology image and the genomic data by using a preset survival prediction model to obtain a survival rate of the cancer patient in a future time period. In the prediction model, feature extraction is performed on the pathology image and the genomic data by using an isolation processing structure to obtain a pathology isolation feature sequence and a genomic isolation feature sequence, feature interaction is performed on the pathology isolation feature sequence and the genomic isolation feature sequence by using an interaction module to obtain a pathology interaction feature sequence and a genomic interaction feature sequence, feature extraction is performed on the pathology interaction feature sequence and the genomic interaction feature sequence by using a cooperative processing structure to obtain a pathology cooperative feature sequence and a genomic cooperative feature sequence, and fusion-type survival prediction is performed based on the pathology cooperative feature sequence and the genomic cooperative feature sequence by using a fusion prediction module to obtain the survival rate of the cancer patient in the future time period, thereby achieving survival prediction of the cancer patient. In this way, the embodiments of the application can achieve effective fusion and cooperation between the pathology genomic image and the genomic data, improve the feature extraction capability of the survival prediction model for survival prediction-related features, and thus improve the accuracy of the survival prediction model.

[0021] Other features and advantages of the present application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structure particularly pointed out in the description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flow chart of the multi-modal survival prediction method provided by the present application;

[0023] Figure 2 is a schematic diagram of the multi-modal survival prediction method provided by the present application;

[0024] Figure 3 is a schematic diagram of the progressive feature extraction provided by the present application;

[0025] Figure 4 is a schematic diagram of the optimization aggregation provided by the present application;

[0026] Figure 5 is a statistical analysis verification diagram of the multi-modal survival prediction method provided by the present application;

[0027] Figure 6 is a comparison diagram of the multi-modal survival prediction method provided by the present application and other mainstream methods in terms of overhead and performance. DETAILED DESCRIPTION

[0028] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0029] The present application is further described below in combination with the drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0030] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0032] Survival prediction aims to assess the relative risk of death in cancer prognosis, which provides key support for medical decision-making. Traditional survival prediction methods rely on single-modal data to achieve survival prediction. For example, in the case of pathology data, multi-instance learning (MIL) has been widely used in recent years to process whole slide images (WSI), which are usually gigapixel-level. In multi-instance learning, the entire pathology image is regarded as an instance bag, and each instance in the instance bag represents a local area of the pathology image. A survival prediction model can achieve survival prediction by identifying the local area that has predictive power. For example, in the case of genomic data, self-normalizing neural networks (SNNs) have been used in recent years to extract features from genomic data for survival prediction. This neural network can automatically standardize data and extract features from gene expression data that are important for survival prediction.

[0033] Although single-modal survival prediction methods show certain potential, analyzing only single-modal data cannot fully utilize the information of other modalities, making it difficult to fully leverage the advantages of multi-modal omics data, thereby affecting the comprehensiveness and accuracy of survival prediction and limiting the effectiveness of survival prediction. In this regard, multi-modal technology has gradually gained more attention in survival prediction. Multi-modal survival prediction methods can effectively integrate the advantages of each modality by combining data from different modalities, thereby improving the accuracy of survival prediction and making up for the shortcomings of single-modal survival prediction methods. For example, in related technologies, pathology data and genomic data of cancer patients are often used to predict the survival rate of cancer patients in a future time period, thereby achieving survival prediction. Although multi-modal survival prediction methods have made certain progress, related technologies often rely on existing attention mechanisms such as multi-head attention mechanisms to integrate features of pathology data and features of genomic data, which makes it difficult to effectively integrate and coordinate multi-modal omics data.

[0034] Specifically, the related art ignores the heterogeneity and sparsity of modalities, limiting the advantages of multi-modal omics data fusion. For sparsity, multi-modal survival prediction methods often need to process high-dimensional genomic data and terapixel-level pathological omics images, but only a small amount of data in these data is closely related to survival prediction. Such data has a certain intra-modality sparsity, which not only increases the computational cost, but also can lead to the capture of false associations by the model, for example, false associations between survival prediction results and non-pathological regions, which can significantly reduce the accuracy and reliability of survival prediction. The related art relies only on attention mechanisms, which is difficult to reduce the impact of intra-modality sparsity on survival prediction. In addition, due to factors such as the distribution of tumors and insufficient blood samples, it is often difficult to obtain complete multi-modal data, which leads to the absence of some modal data, and the related art still cannot effectively deal with this modality absence problem, thereby affecting the effectiveness and applicability of survival prediction. For heterogeneity, on the one hand, clinical data often has significant intra-modality heterogeneity, which is mainly caused by factors such as genetic mutations, differences in tumor microenvironments, and makes feature extraction complex and difficult. The related art relies only on attention mechanisms, which is difficult to adapt to intra-modality heterogeneity, thereby leading to problems such as low accuracy of survival prediction results and poor generalization ability of the model. On the other hand, inter-modality heterogeneity is also a major challenge. Pathological omics data and genomic data fundamentally differ in nature, which often requires professional technical personnel to process omics data of different modalities, thereby increasing the cost of survival prediction and reducing the efficiency of survival prediction.

[0035] In summary, how to effectively deal with the intra-modality sparsity problem of multi-modal data in survival prediction, reduce the impact of false associations on survival prediction and improve the efficiency of survival prediction, and how to handle the heterogeneity problem of multi-modal data, effectively fuse and coordinate omics data of different modalities, and thereby improve the accuracy of survival prediction, have become problems to be solved.

[0036] Therefore, the present application provides a multi-modal survival prediction method and device, aiming to improve the accuracy of survival prediction.

[0037] First, the specific implementation steps of the multi-modal survival prediction method provided by the present application will be described in detail below.

[0038] The multi-modal survival prediction method provided by the application can be applied to a terminal, a server, software running in the terminal or the server, etc. The terminal can be a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited thereto. The blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc.

[0039] Referring to Figure 1 and Figure 2 , Figure 1 is a flowchart of the multi-modal survival prediction method provided by the application, Figure 2 is a principle diagram of the multi-modal survival prediction method provided by the application, Figure 2 In the multi-modal survival prediction method provided by the application, “genome” refers to genomic data, “pathological map” refers to a pathological image, “grouping and extracting features”, “isolated unit” and “ATSA” collectively indicate an isolated processing structure, “cross attention mechanism” refers to an interaction module, “initialization” and “collaboration unit” collectively indicate a collaborative processing structure, “low-rank fusion” and “classification head” refer to a fusion prediction module. The multi-modal survival prediction method can include the following steps S101-S102.

[0040] S101, obtaining a pathological image and genomic data of a cancer patient.

[0041] It should be noted that the pathological image refers to a full-view digital section image generated by scanning a pathological section of a cancer patient, which is a two-dimensional optical image. In addition, the genomic data refers to gene expression data, which can be obtained by sequencing technologies such as DeoxyriboNucleic Acid (DNA) microarray, RNA sequencing (RNA-seq) technology or single-cell RNA sequencing (scRNA-seq) technology, but is not limited thereto.

[0042] In this step, the pathological image and the genomic data of the cancer patient are obtained through a preset database. It should be noted that the pathological image and the genomic data of the cancer patient are data samples pre-stored in the database, rather than data samples directly sampled from a living body.

[0043] Optionally, the type of cancer can be selected according to actual conditions, and the embodiments of the present application do not make specific limitations thereon. For example, the embodiments of the present application can be applicable to adenocarcinoma, glioma, lung squamous cell carcinoma, endometrial carcinoma, bladder cancer, breast cancer, etc., but are not limited thereto.

[0044] In S102, a survival prediction model is used to perform survival prediction on the pathological image and the genomic data, to obtain the survival rate of the cancer patient in a future time period.

[0045] The survival prediction model includes:

[0046] An isolation processing structure is configured to perform feature extraction on the pathological image and the genomic data, to obtain a pathological isolation feature sequence and a genomic isolation feature sequence.

[0047] An interaction module is configured to perform feature interaction on the pathological isolation feature sequence and the genomic isolation feature sequence, to obtain a pathological interaction feature sequence and a genomic interaction feature sequence.

[0048] A cooperative processing structure is configured to perform feature extraction on the pathological interaction feature sequence and the genomic interaction feature sequence, to obtain a pathological cooperative feature sequence and a genomic cooperative feature sequence.

[0049] A fusion prediction module is configured to perform fusion-type survival prediction based on the pathological cooperative feature sequence and the genomic cooperative feature sequence, to obtain the survival rate of the cancer patient in the future time period.

[0050] It should be noted that the survival prediction model is a pre-trained improved neural network model. During training of the above survival prediction model, the data samples of the above survival prediction model include pathological image samples and genomic data samples of multiple patients, and the label of the above survival prediction model is the actual inventory condition and the event indicator variable of each patient at a certain time. If the patient dies at a certain time, the event indicator variable is a first value, otherwise it is a second value. Optionally, the first value and the second value can be set according to actual conditions, and the embodiments of the present application do not make specific limitations thereon. For example, the first value can be one and the second value can be zero, but is not limited thereto.

[0051] It can be understood that the survival rate of the cancer patient in the future time period can include the survival rate of the cancer patient at at least one future time, and the survival rate is the survival probability.

[0052] In this step, the pathological image and the genomic data are input into the survival prediction model. In the survival prediction model, first, feature extraction is performed on the pathological image and the genomic data by an isolation processing structure, aiming to perform specific feature extraction on the data of different modalities, capture key features for survival prediction, and shield redundant features irrelevant to survival prediction, thereby obtaining a pathological isolated feature sequence and a genomic isolated feature sequence; then, feature interaction is performed on the pathological isolated feature sequence and the genomic isolated feature sequence by an interaction module, aiming to promote cross-modal complementation and fusion between the pathological omics features and the genomic features, thereby obtaining a pathological interaction feature sequence and a genomic interaction feature sequence; thereafter, feature extraction is performed on the pathological interaction feature sequence and the genomic interaction feature sequence by a cooperative processing structure, aiming to perform specific feature extraction on the fused pathological omics features and the fused genomic features, further capture key features for survival prediction from the fused omics features, thereby obtaining a pathological cooperative feature sequence and a genomic cooperative feature sequence; finally, fusion prediction is performed on the pathological cooperative feature sequence and the genomic cooperative feature sequence by a fusion prediction module, which is fusion prediction performed after feature fusion, thereby obtaining the survival rate of the cancer patient in a future time period, and realizing survival prediction of the cancer patient. In this way, the embodiments of the present application can realize effective fusion and cooperation between the pathological omics image and the genomic data, improve the feature extraction capability of the survival prediction model for features related to survival prediction, and thereby improve the accuracy of the survival prediction model.

[0053] In some embodiments, with reference to Figure 2 The above isolation processing structure can include two parallel isolation processing branches, one input of one isolation processing branch is the pathological image, one output of one isolation processing branch is the pathological isolated feature sequence, one input of the other isolation processing branch is the genomic data, and one output of the other isolation processing branch is the genomic isolated feature sequence; the isolation processing branch includes a first input layer, an optimization module, and two sequentially connected isolation units; the first input layer is used to obtain a first input sequence according to the input of the isolation processing branch and a preset classification label; the first isolation unit is used to perform feature extraction on the input of the first isolation unit to obtain the output of the first isolation unit; and the optimization module is used to optimize the output of the second isolation unit to obtain the output of the isolation processing branch.

[0054] In this embodiment, the isolation processing structure includes two parallel isolation processing branches, namely a pathological isolation processing branch and a genomic isolation processing branch. The input of the pathological isolation processing branch is the pathological image, and the output is the pathological isolated feature sequence; the input of the genomic isolation processing branch is the genomic data, and the output is the genomic isolated feature sequence, as shown in Figure 2As shown in FIG. 1, the data flow branch where the pathological PREE of the isolation unit is located is the pathological isolation processing branch, and the data flow branch where the genetic PREE of the isolation unit is located is the genetic isolation processing branch. Both of the two isolation processing branches include a first input layer, an optimization module, and two sequentially connected isolation units, wherein:

[0055] Figure 2 “Group and extract features” in the above formula indicates the first input layer, and the function of the first input layer is to obtain a first input sequence according to the input of the isolation processing branch and a preset classification token. Specifically, in the first input layer of the genetic isolation processing branch, the genomic data is first divided into six functional groups of tumor suppression, protein kinase, protein kinase, cell differentiation, transcription, cytokine and growth factor, and then the data of each functional group is extracted by a self-normalization neural network to obtain a genomic feature token of each functional group, such as shown in FIG. 2. Figure 2 As shown in FIG. 2, “genomic features” refer to genomic feature tokens, and one functional group corresponds to at least one genomic feature token. Then, the genomic feature tokens of each functional group are spliced to obtain a genomic feature sequence, and finally a first classification token corresponding to the genomic is added at the beginning of the genomic feature sequence to obtain a first input sequence corresponding to the genomic, such as shown in FIG. 2. Figure 2 As shown in FIG. 2, “global features” are classification tokens. The first input sequence corresponding to the genomic includes a first classification token and a plurality of genomic feature tokens. In the first input layer of the pathological isolation processing branch, the pathological image is first divided into a plurality of non-overlapping image patches, and then each image patch is extracted by using a pre-trained model such as a clustering-constrained attention multiple instance learning (CLAM) model to obtain a plurality of pathological feature tokens, such as shown in FIG. 3. Figure 2 As shown in FIG. 3, “pathological features” refer to pathological feature tokens. Then, each pathological feature token is spliced to obtain a pathological feature sequence, and finally a second classification token corresponding to the pathology is added at the beginning of the pathological feature sequence to obtain a first input sequence corresponding to the pathology. The first input sequence corresponding to the pathology includes a second classification token and a plurality of pathological feature tokens. It should be noted that the first classification token and the second classification token are both preset classification tokens, and different modalities have different classification tokens, that is, the first classification token and the second classification token are different.

[0056] The isolation units of the genetic isolation processing branch and the pathological isolation processing branch are roughly the same in structure and function. As shown in FIG. 1, Figure 2As shown, two sequentially connected isolation units are arranged between the first input layer and the optimization module, that is, the input end of the first isolation unit is connected with the output end of the first input layer, the input of the first isolation unit is the first input sequence, the output end of the first isolation unit is connected with the input end of the second isolation unit, and the output end of the second isolation unit is connected with the input end of the optimization module. The isolation unit is used for feature extraction on the input of the isolation unit to obtain the output of the isolation unit, aiming to perform specific feature extraction on data of different modalities to effectively cope with the problems of data diversity and missing modalities.

[0057] The optimization modules of the genetic isolation processing branch and the pathological isolation processing branch are completely identical in structure and function. Figure 2 The "ATSA" in the above formula is the optimization module, also known as the adaptive tag selection and aggregation module. The optimization module is used for optimizing the output of the second isolation unit to obtain the output of the isolation processing branch, aiming to further extract key features for survival prediction and shield redundant features irrelevant to survival prediction, especially false correlations.

[0058] In some embodiments, with reference to Figure 2 The above isolation unit can include:

[0059] A first encoder is configured to perform encoding processing on the input of the isolation unit to obtain a first encoded feature sequence.

[0060] A first expert module is configured to perform progressive feature extraction on the first encoded feature sequence to obtain the output of the isolation unit.

[0061] In the embodiment, the above isolation unit can include the first encoder and the first expert module. The first encoder is a Transformer encoder, such as Figure 2 As shown, the "Transformer block" in the "isolation unit" refers to the first encoder, "PREE" refers to the first expert module, also known as the progressive residual expert expansion module, "genetic PREE" in the "isolation unit" refers to the first expert module of the genetic isolation processing branch, and "pathological PREE" in the "isolation unit" refers to the first expert module of the pathological isolation processing branch.

[0062] In the isolation unit, firstly, the input of the isolation unit is encoded by a first encoder to obtain a first encoded feature sequence. This sequence aims to calculate the correlation between omics feature labels using a multi-head self-attention mechanism and combine it with a feedforward network to preserve the basic semantic information of the omics feature labels, thereby capturing the global dependencies within the modality. The first encoded feature sequence includes a classification label and multiple omics feature labels. Here, the classification label refers to the classification label processed by the first encoder, and the omics feature labels refer to the omics feature labels processed by the first encoder. Notably, the classification label and all omics feature labels in the input of the isolation unit participate in the multi-head self-attention mechanism simultaneously as queries, keys, and values. Then, a first expert module performs progressive feature extraction on the first encoded feature sequence output by the first encoder to obtain the output of the isolation unit. Progressive feature extraction refers to multi-scale feature extraction from shallow to deep layers, aiming to extract specific features for the current modality's data and effectively address the problems of data diversity and modality missingness.

[0063] Optionally, such as Figure 3 As shown, in the isolation unit of the pathological isolation processing branch, a preset position code can be added to the first coding feature sequence corresponding to pathomics. This can capture the spatial information of the image and the relative positional relationship between pixels, thereby better preserving local features and enhancing the survival prediction model's understanding of spatial context.

[0064] In some implementations, refer to Figure 3 To address the issues of intra-modal sparsity and intra-modal heterogeneity, the first expert module, when performing progressive feature extraction on the first encoded feature sequence to obtain the output of the isolation unit, specifically performs the following operations:

[0065] The first coded feature sequence is processed using a preset first multilayer perceptron and a preset first expert network to obtain the first residual feature sequence;

[0066] The first residual feature sequence is processed using a pre-defined second multilayer perceptron and two pre-defined second expert networks to obtain the second residual feature sequence;

[0067] The second residual feature sequence is processed using a pre-defined third multilayer perceptron and four pre-defined third expert networks to obtain the output of the isolation unit.

[0068] In this embodiment, the first expert module can be divided into a three-layer sequentially connected architecture, such as... Figure 4As shown, from bottom to top are the first layer architecture, the second layer architecture and the third layer architecture, and the "expert" refers to an expert network, the first layer architecture includes a first multi-layer perceptron and a first expert network, the second layer architecture includes a second multi-layer perceptron and two second expert networks, the second layer architecture selects the final expert network from the two second expert networks through a preset gating mechanism, and the third layer architecture includes a third multi-layer perceptron and four third expert networks, and the third layer architecture also selects the final expert network from the four third expert networks through a preset gating mechanism.

[0069] Specifically, in the first layer architecture, the first encoded feature sequence is simultaneously input into the first multi-layer perceptron and the first expert network, the first multi-layer perceptron extracts the basic features of the current modality from the first encoded feature sequence, and the first expert network extracts the more targeted features for the current modality from the first encoded feature sequence, and then the output of the first multi-layer perceptron and the output of the first expert network are connected in residual, aiming to fuse deep features and shallow features, so as to obtain the first residual feature sequence and input into the second layer architecture.

[0070] In the second layer architecture, the second multi-layer perceptron extracts the basic features of the current modality from the first residual feature sequence, and at the same time, the first residual feature sequence combines the preset gating mechanism to select the second expert network most suitable for the output of the first layer architecture from the two second expert networks as the first target network. After selecting and activating the first target network, the first target network extracts the more targeted features for the current modality from the first residual feature sequence, and then the output of the second multi-layer perceptron and the output of the first target network are connected in residual, aiming to fuse deep features and shallow features, so as to obtain the second residual feature sequence and input into the third layer architecture.

[0071] In the third layer architecture, the third multi-layer perceptron extracts the basic features of the current modality from the second residual feature sequence, and at the same time, the second residual feature sequence combines the preset gating mechanism to select the third expert network most suitable for the output of the second layer architecture from the four third expert networks as the second target network. After selecting and activating the second target network, the second target network extracts the more targeted features for the current modality from the second residual feature sequence, and then the output of the third multi-layer perceptron and the output of the second target network are connected in residual, aiming to fuse deep features and shallow features, so as to obtain the output of the isolation unit.

[0072] More specifically, the gating mechanism is a gating mechanism composed of a multi-layer perceptron and a Softmax function, which belongs to the existing gating mechanism. In simple terms, for the second layer architecture and the third layer architecture, in the gating mechanism, the features input to the current layer architecture are mapped to a score vector of multiple expert networks through a multi-layer perceptron, and the scoring of the classification task is performed through a Softmax function. For example, for the second layer architecture, it has two expert networks, which is a binary classification task, and for the third layer architecture, it has four expert networks, which is a four-class classification task. The Softmax function can normalize the score vector of each expert network into a probability distribution to obtain the score information of each expert network, and select the expert network with the highest score information as the target network of the current layer architecture. The target network will participate in feature extraction, while the other unselected expert networks do not participate in feature extraction.

[0073] As can be seen, for each layer architecture, the number of expert networks that can be selected is one, two and four, respectively, that is, gradually expanding from one expert network to two expert networks, four expert networks, progressive growth, which can match the gradually complex modal features and extract modal multi-scale features from shallow to deep. Further, the embodiment of the present application activates the expert network with specific modal characteristics through the gating mechanism. Different expert networks can be activated for different modalities and different patient characteristics. The gating mechanism can select the most suitable expert network according to the input features, complete the optimal routing based on the current input, and realize flexible and efficient feature processing. This expert selection helps to gradually extract more complex deep modal features, and then uses the selected expert network to extract features and introduces residual connection to fuse the shallow features and deep features of the specific modal, realizing progressive feature extraction. In this way, the embodiment of the present application can gradually extract modal multi-scale features from shallow to deep for specific modalities, capture more and more complex modal features, effectively deal with the data diversity problem (i.e. intra-modality heterogeneity) caused by factors such as genetic mutation, differences in tumor microenvironment, and the modal missing problem (i.e. intra-modality sparsity) caused by factors such as tumor distribution and insufficient blood samples, thereby improving the feature extraction capability of the survival prediction model and improving the accuracy of survival prediction.

[0074] Optionally, for the isolation unit of the pathological isolation processing branch, the expert network of the first expert module can be a convolutional neural network (CNN); and for the isolation unit of the genetic isolation processing branch, the expert network of the first expert module can be a self-normalized neural network, but is not limited thereto. In addition, during the training of the survival prediction model, for the first expert module of the isolation unit, the hyperparameters of all the multilayer perceptrons are frozen, i.e., the hyperparameters of all the multilayer perceptrons are not updated, and are initialized using the model parameters in the preprocessing process; and the hyperparameters of all the expert networks are trainable, i.e., the hyperparameters of all the expert networks are updated.

[0075] In some embodiments, with reference to Figure 2 , to solve the problem of intra-modality sparsity, the optimization module is specifically configured to perform the following operations when used to optimize the output of the second isolation unit to obtain the output of the isolation processing branch:

[0076] splicing the classification labels and the plurality of omics feature labels in the output of the second isolation unit to obtain a plurality of spliced labels and score information of each spliced label;

[0077] classifying the plurality of spliced labels based on the score information of each spliced label to obtain a first category group and a second category group, the first category group including a plurality of spliced labels with score information greater than or equal to a preset first threshold, and the second category group including a plurality of spliced labels with score information less than the first threshold;

[0078] selectively refining the first category group to obtain a plurality of first selected labels and a plurality of second selected labels;

[0079] performing adaptive pooling based on the plurality of second selected labels and the second category group to obtain a plurality of pooled labels;

[0080] obtaining the output of the isolation processing branch according to the plurality of first selected labels and the plurality of pooled labels.

[0081] In the present embodiment, the optimization module aims to calculate the importance of the input high-dimensional feature labels and select different score features for screening and aggregation, thereby reducing redundant information and retaining key information. Specifically, the output of the second isolation unit is a feature sequence of a specific modality, which can be represented as which can include classification labels and N omics feature labels , wherein the classification labels refer to classification labels introduced from the first input layer and processed by the two isolation units, and the omics feature labels refer to omics feature labels processed by the two isolation units. The output of the second isolation unit is input into the optimization module, in which:

[0082] First, the output of the second isolation unit is dimensionality-reduced using a multilayer perceptron, resulting in a dimensionality-reduced output. Then, each omics feature label in the dimensionality-reduced output is concatenated with a classification label in the same output, resulting in N concatenated labels. Since the classification label represents global information, concatenating the classification label with the omics feature label allows the omics feature label to incorporate global information. For ease of understanding, the first... Each splicing mark is represented by the following formula (1):

[0083] , (1);

[0084] In equation (1), Indicates the first A splicing mark; This indicates the output of the second isolation unit. Omics feature labels; This indicates the classification label in the output of the second isolation unit; This represents a multilayer perceptron.

[0085] After the marker splicing is completed, an importance score for each spliced ​​marker is calculated using a gating mechanism consisting of a multilayer perceptron and a Softmax function, thereby obtaining the score information for multiple spliced ​​markers. It is understood that the principle of this gating mechanism is the same as that in the aforementioned expert module, and will not be elaborated further. For ease of understanding, the... The score information of each splicing mark is represented by the following formula (2):

[0086] (2);

[0087] In equation (2), Indicates the first Score information for each splicing marker; This represents the Softmax function.

[0088] Then, based on the score information of each splicing tag, the N splicing tags are sorted from largest to smallest. It can be understood that the higher the score information of the splicing tag, the smaller the sort number of the splicing tag, that is, the higher the ranking, while the lower the score information of the splicing tag, the larger the sort number of the splicing tag, that is, the lower the ranking. The first category group is constructed by using K splicing tags with scores greater than or equal to the first threshold. These splicing tags with scores greater than or equal to the first threshold can be understood as the top K splicing tags, denoted as TopK tags. They are the most predictive omics feature tags. At the same time, the second category group is constructed by using NK splicing tags with scores less than the first threshold. These splicing tags with scores less than the first threshold can be understood as the bottom K+1 splicing tags, denoted as non-TopK tags. They are redundant omics feature tags. For ease of understanding, the first category group satisfies the following formula (3):

[0089] , (3);

[0090] In equation (3), Indicates the first category group; This represents the set of scores consisting of all the splicing tags; This indicates that the TopK function is used on the set. The set obtained by performing an operation, which involves sorting all splicing tags in descending order of their scores and selecting the top K splicing tags; Indicates the first category group A splicing marker, which belongs to the set. .

[0091] Optionally, the number of splicing marks included in the first category group and the second category group, as well as the first threshold, can be set according to actual conditions, and this embodiment does not impose specific limitations on this.

[0092] Next, selective refinement is performed. Specifically, firstly, the K TopK labels are divided into... First choice marker and A second-choice tag is used to achieve selective improvement. It is understandable that the first-choice tag accounts for a certain percentage of all TopK tags. The proportion of the second choice marker among all TopK markers is... .

[0093] Optionally, the selective and comprehensive division method can be set according to the actual situation, and this implementation method does not impose specific limitations on it. For example, As a preset value, the first category group is divided into groups according to a preset ratio. First choice marker and A second selection tag. For example, classifying the first category group based on the score information of each splicing tag in the first category group is the same as classifying multiple splicing tags. In this case, the top K splicing tags in the first category group are the first selection tags, and the remaining splicing tags are the second selection tags, but it is not limited to this.

[0094] After selective improvement is completed, the label dimension will be... The second selection marker and NK non-TopK markers are concatenated to obtain a concatenated sequence. This concatenated sequence is then pooled to generate supplementary information. Multiple pooling labels are used to achieve adaptive pooling. In this way, by using a portion of the most predictive omics feature labels to aggregate redundant omics feature labels, redundant features unrelated to survival prediction can be effectively masked, feature information falsely associated with survival prediction can be reduced, and the computational complexity of feature extraction can be reduced, thereby reducing the impact of intramodal sparsity on survival prediction. At the same time, it can effectively capture features related to survival prediction hidden in redundant features, which is conducive to improving feature comprehensiveness. For ease of understanding, multiple pooling labels are represented by the following formula (4):

[0095] (4);

[0096] In equation (4), express One pooling marker; Indicates pooling processing; express A second selection marker; This represents NK splicing markers in the second category group.

[0097] Optionally, the above pooling process can be flexibly configured according to the actual situation. For example, the above pooling process can be average pooling, but it is not limited to this.

[0098] Finally, First choice marker and The pooled labels are summed to obtain K refined TopK labels, which is the output of the isolated processing branch. In this way, integrating the optimized features with another set of most predictive omics features effectively preserves key predictive features and improves the comprehensiveness and accuracy of key survival prediction features. For ease of understanding, the output of the isolated processing branch is expressed as follows (5):

[0099] (5);

[0100] In equation (5), an output of the isolation processing branch; an output of the isolation processing branch; a first selection mark.

[0101] It can be seen that the optimization module provided by the embodiment first interacts the classification mark and the plurality of omics feature marks in the output of the second isolation unit, aiming to make the omics feature marks fuse global information, so as to obtain a plurality of splicing marks, then calculates score information of each splicing mark by using a gating mechanism, and divides all the splicing marks into a first category group and a second category group by using the score information of each splicing mark, the first category group has high prediction value, and the second category group is regarded as redundant features, then a part of features with high prediction value is used to aggregate the redundant features, which can effectively shield the redundant features irrelevant to survival prediction, reduce feature information with false association with survival prediction, and reduce the computational complexity of feature extraction, thereby reducing the influence of intra-modal sparsity on survival prediction, and effectively capturing features associated with survival prediction hidden in redundant features, which is conducive to improving the comprehensiveness of features. Finally, the optimized features are integrated with another part of omics feature marks with the highest prediction value to obtain the output of the isolation processing branch, which can effectively retain key information with prediction value and improve the comprehensiveness and accuracy of key features for survival prediction. In this way, the embodiment can effectively cope with the problem of intra-modal sparsity and improve the feature extraction capability of the survival prediction model.

[0102] In some embodiments, with reference to Figure 2 , to solve the problem of inter-modal heterogeneity, the interaction module is specifically configured to perform the following operations when used for feature interaction between the pathological isolation feature sequence and the genetic isolation feature sequence to obtain the pathological interaction feature sequence and the genetic interaction feature sequence:

[0103] processing the pathological isolation feature sequence and the genetic isolation feature sequence by using the cross-modal attention mechanism to obtain the pathological interaction feature sequence and the genetic interaction feature sequence.

[0104] In the embodiment, in the isolation processing structure, the output of the pathological isolation processing branch is the pathological isolation feature sequence, and the pathological isolation feature sequence includes a plurality of pathological isolation feature marks. The output of the genetic isolation processing branch is the genetic isolation feature sequence, and the genetic isolation feature sequence includes a plurality of genetic isolation feature marks. The isolation feature mark here refers to an omics feature mark processed by the isolation processing structure, such as Figure 2As shown, the "perfected feature" is the isolation feature label. Through the cross-modal attention mechanism, the pathological isolation feature sequence and the gene isolation feature sequence are interacted, aiming to capture the correlation between the pathology image and the genomic data, promote the interaction and fusion between different modalities, and solve the problem of inter-modal heterogeneity, so as to obtain the pathological interaction feature sequence and the gene interaction feature sequence, and realize the effective fusion and cooperation between the pathology image and the genomic data. Among them, the pathological interaction feature sequence includes a plurality of pathological interaction feature labels, and the pathological interaction feature label refers to the pathological isolation feature label processed by the interaction module, and the gene interaction feature sequence includes a plurality of gene interaction feature labels, and the gene interaction feature label refers to the gene isolation feature label processed by the interaction module, such as Figure 2 As shown, the "cooperated feature" output by the "cross attention mechanism" is the gene interaction feature label.

[0105] Specifically, the fusion result of the multi-modal feature is calculated by the cross-modal attention mechanism, the input features are interacted by using a plurality of trainable weights, the trainable means that the weight is updated in the training process of the survival prediction model, and the output of the cross-modal attention mechanism contains two important information of the pathological interaction feature sequence and the gene interaction feature sequence. The process is as follows:

[0106] First, the pathological isolation feature sequence is dimensionally converted by using a trainable first weight to obtain the key vector of the cross-modal attention mechanism, and the gene isolation feature sequence is dimensionally converted by using a trainable second weight to obtain the query vector of the cross-modal attention mechanism, as shown in the following formula (6):

[0107] , (6);

[0108] In formula (6), denotes the key vector of the cross-modal attention mechanism; denotes the first trainable weight; denotes the pathological isolation feature sequence; denotes the query vector of the cross-modal attention mechanism; denotes the second trainable weight; denotes the gene isolation feature sequence. It can be understood that the functions of the first weight and the second weight are to dimensionally convert the feature sequence, so as to facilitate the calculation of the cross-modal attention mechanism.

[0109] Then, the key vector and the query vector of the cross-modal attention mechanism are subjected to a Softmax operation in the column direction to cause the pathology omics features to focus on information in the genomics features associated with the survival prediction, to promote fusion of the pathology omics modality and the genomics modality, to obtain an attention weight matrix of the key vector, to transpose the attention weight matrix of the key vector, and to obtain a pathology interaction feature sequence based on the transposed attention weight matrix of the key vector, the pathology isolated feature sequence, and a third trainable weight, thereby reducing the influence of inter-modality heterogeneity on the survival prediction, as shown in the following formula (7):

[0110] (7);

[0111] In formula (7), denotes the pathology interaction feature sequence; denotes the third trainable weight; denotes a ColumnSoftmax function, i.e., a Softmax operation in the column direction. It can be understood that the third weight functions to map the fused pathology omics features.

[0112] Meanwhile, the key vector and the query vector of the cross-modal attention mechanism are subjected to a Softmax operation in the row direction to cause the genomics features to focus on information in the pathology omics features associated with the survival prediction, to promote fusion of the pathology omics modality and the genomics modality, to obtain an attention weight matrix of the query vector, to transpose the attention weight matrix of the query vector, and to obtain a gene interaction feature sequence based on the transposed attention weight matrix of the query vector, the gene isolated feature sequence, and a fourth trainable weight, thereby reducing the influence of inter-modality heterogeneity on the survival prediction, as shown in the following formula (8):

[0113] (8);

[0114] In formula (8), denotes the gene interaction feature sequence; denotes the fourth trainable weight; denotes a RowSoftmax function, i.e., a Softmax operation in the row direction. It can be understood that the fourth weight functions to map the fused genomics features.

[0115] In some embodiments, with reference to Figure 2The above-mentioned cooperative processing structure can include two parallel cooperative processing branches, one of which takes the pathological interaction feature sequence as input, and the output of one of the cooperative processing branches is a pathological cooperative feature sequence, the input of the other cooperative processing branch is a genetic interaction feature sequence, and the output of the other cooperative processing branch is a genetic cooperative feature sequence; the cooperative processing branch includes a second input layer and two sequentially connected cooperative units; the second input layer is used to obtain a second input sequence according to the input of the cooperative processing branch and a preset classification mark; the input of the first cooperative unit is the second input sequence, and the output of the second cooperative unit is the output of the cooperative processing branch; the cooperative unit is used to extract features from the input of the cooperative unit to obtain the output of the cooperative unit.

[0116] In this embodiment, the cooperative processing structure aims to further extract deep interaction features between modalities, especially key features associated with survival prediction between modalities, and to promote deep extraction and interaction of features of different modalities, improve the integrity and depth of the survival prediction features, and not introduce additional interference and redundant features. Similar to the isolation processing structure in the foregoing embodiment, the cooperative processing structure includes two parallel pathological cooperative processing branches and genetic cooperative processing branches, the input of the pathological cooperative processing branch is the pathological interaction feature sequence, and the output is the pathological cooperative feature sequence; the input of the genetic cooperative processing branch is the genetic interaction feature sequence, and the output is the genetic cooperative feature sequence, as shown in Figure 2 The data flow branch where the "pathological PREE" of the "cooperative unit" is located is the pathological cooperative processing branch, and the data flow branch where the "genetic PREE" of the "cooperative unit" is located is the genetic cooperative processing branch. Both of the two cooperative processing branches include a second input layer and two sequentially connected cooperative units, wherein:

[0117] Figure 2 The "initialization" in the above-mentioned "cooperative unit" indicates the second input layer, and the function of the second input layer is to obtain a second input sequence according to the input of the cooperative processing branch and a preset classification mark. Specifically, in the second input layer of the genetic cooperative processing branch, a third classification mark corresponding to genomics is added at the starting position of the genetic interaction feature sequence to obtain a second input sequence corresponding to genomics. The second input sequence corresponding to genomics includes the third classification mark and a plurality of genetic interaction feature marks. In the second input layer of the pathological cooperative processing branch, a fourth classification mark corresponding to pathology is added at the starting position of the pathological interaction feature sequence to obtain a second input sequence corresponding to pathology. The second input sequence corresponding to pathology includes the fourth classification mark and a plurality of pathological interaction feature marks. It should be noted that the third classification mark and the fourth classification mark are both preset classification marks, and different modalities have different classification marks, that is, the third classification mark and the fourth classification mark are different.

[0118] The input end of the first cooperative unit is connected with the output end of the second input layer, the input of the first cooperative unit is the second input sequence, the output end of the first cooperative unit is connected with the input end of the second cooperative unit, and the output of the second cooperative unit is the output of the cooperative processing branch. The role of the cooperative unit is to perform feature extraction on the input of the cooperative unit to obtain the output of the cooperative unit, aiming to perform specific feature extraction on the fused pathology omics features and the fused genomics features, and further capture key features for survival prediction from the fused omics features.

[0119] In some embodiments, with reference to Figure 2 The cooperative unit can include:

[0120] The second encoder is configured to perform encoding processing on the input of the cooperative unit to obtain a second encoded feature sequence;

[0121] The second expert module is configured to perform progressive feature extraction on the second encoded feature sequence to obtain the output of the cooperative unit.

[0122] In the embodiment, the cooperative unit can include the second encoder and the second expert module. The second encoder is a Transformer encoder, as shown in Figure 2 The "Transformer block" in the "cooperative unit" refers to the second encoder, and "PREE" refers to the second expert module, wherein "gene PREE" refers to the second expert module of the gene cooperative processing branch, and "pathology PREE" refers to the second expert module of the pathology cooperative processing branch. In the cooperative unit, first, the second encoder is used to perform encoding processing on the input of the cooperative unit to obtain a second encoded feature sequence, which aims to calculate the correlation between interaction feature markers through a multi-head self-attention mechanism, and combines a feedforward network to retain the basic semantic information of the interaction feature markers, thereby capturing the global dependency between modalities. The second encoded feature sequence includes classification markers and a plurality of interaction feature markers. The classification markers refer to classification markers processed by the second encoder, and the interaction feature markers refer to interaction feature markers processed by the second encoder. It is worth noting that the classification markers and all interaction feature markers in the input of the cooperative unit are simultaneously used as queries, keys and values to participate in the multi-head self-attention mechanism. Then, the second expert module is used to perform progressive feature extraction on the second encoded feature sequence output by the second encoder to obtain the output of the cooperative unit. The progressive feature extraction refers to multi-scale feature extraction from shallow to deep, aiming to perform specific feature extraction on the fused pathology omics features and the fused genomics features, further capture key features for survival prediction from the fused omics features, and effectively deal with the problems of data diversity and modal missing.

[0123] Optionally, asFigure 3 As shown, in the synergy unit of the pathological cooperative processing branch, a preset position code can be added in the second encoding feature sequence corresponding to the pathology group, so that the spatial information of the image and the relative position relationship between the pixels can be captured, thereby better preserving the local features and enhancing the understanding of the spatial context by the survival prediction model.

[0124] In some embodiments, with reference to Figure 2 , in order to solve the problems of intra-modality sparsity and intra-modality heterogeneity, when the second expert module is used to perform progressive feature extraction on the second encoding feature sequence to obtain the output of the synergy unit, the second expert module is specifically configured to perform the following operations:

[0125] processing the second encoding feature sequence by using a preset fourth multi-layer perceptron and a preset fourth expert network to obtain a third residual feature sequence;

[0126] processing the third residual feature sequence by using a preset fifth multi-layer perceptron and two preset fifth expert networks to obtain a fourth residual feature sequence;

[0127] processing the fourth residual feature sequence by using a preset sixth multi-layer perceptron and four preset sixth expert networks to obtain the output of the synergy unit.

[0128] In this embodiment, the structure of the second expert module is the same as that of the first expert module, and the second expert module can be divided into three layers of sequentially connected architecture. Among them, the first layer architecture includes a fourth multi-layer perceptron and a fourth expert network; the second layer architecture includes a fifth multi-layer perceptron and two fifth expert networks, and the second layer architecture selects the final expert network from the two fifth expert networks through a preset gating mechanism; the third layer architecture includes a sixth multi-layer perceptron and four sixth expert networks, and the third layer architecture also selects the final expert network from the four sixth expert networks through a preset gating mechanism.

[0129] Specifically, in the first layer architecture, the second encoding feature sequence is simultaneously input into the fourth multi-layer perceptron and the fourth expert network, the fourth multi-layer perceptron extracts the basic features of the current modality interacting with other modality information from the second encoding feature sequence, and the fourth expert network extracts more targeted features for the current modality interacting with other modality information from the second encoding feature sequence. Then, the output of the fourth multi-layer perceptron and the output of the fourth expert network are connected in residual, aiming to fuse deep features and shallow features, thereby obtaining a third residual feature sequence and inputting it into the second layer architecture.

[0130] In the second layer architecture, the basic features of the current modality interacting with other modality information are extracted from the third residual feature sequence by the fifth multi-layer perceptron, and the fifth expert network most suitable for the output of the first layer architecture is selected as the third target network from the two fifth expert networks by combining the preset gating mechanism with the third residual feature sequence. After selecting and activating the third target network, more targeted features for the current modality interacting with other modality information are extracted from the third residual feature sequence by the third target network, and then the output of the fifth multi-layer perceptron and the output of the third target network are connected in residual, aiming to fuse deep features and shallow features, thereby obtaining a fourth residual feature sequence and inputting to the third layer architecture.

[0131] In the third layer architecture, the basic features of the current modality interacting with other modality information are extracted from the fourth residual feature sequence by the sixth multi-layer perceptron, and the sixth expert network most suitable for the output of the second layer architecture is selected as the fourth target network from the four sixth expert networks by combining the preset gating mechanism with the fourth residual feature sequence. After selecting and activating the fourth target network, more targeted features for the current modality interacting with other modality information are extracted from the fourth residual feature sequence by the fourth target network, and then the output of the sixth multi-layer perceptron and the output of the fourth target network are connected in residual, aiming to fuse deep features and shallow features, thereby obtaining the output of the collaborative unit.

[0132] More specifically, the gating mechanism is a gating mechanism composed of a multi-layer perceptron and a Softmax function, which belongs to the existing gating mechanism. Simply put, for the second layer architecture and the third layer architecture, in the gating mechanism, the features input to the current layer architecture are mapped to a score vector of multiple expert networks by the multi-layer perceptron, and the Softmax function is used for scoring in the classification task. For example, for the second layer architecture, it has two expert networks, which is a binary classification task, and for the third layer architecture, it has four expert networks, which is a four-class classification task. The Softmax function can normalize the score vector of each expert network to a probability distribution to obtain the score information of each expert network, and select the expert network with the highest score information as the target network of the current layer architecture, which will participate in feature extraction, while other unselected expert networks do not participate in feature extraction.

[0133] It can be seen that for each layer architecture, the number of selectable expert networks is one, two and four, respectively, that is, gradually expanding from one expert network to two expert networks, four expert networks, and gradually increasing, which can match the gradually complex modal features and extract modal multi-scale features from shallow to deep. Further, the embodiment activates the expert network specific to the specific modal through the gating mechanism, and different expert networks can be activated for different modalities and different patient characteristics. The gating mechanism can select the most suitable expert network according to the input features, complete the optimal routing based on the current input, and realize flexible and efficient feature processing. This expert selection helps to gradually extract more complex deep modal features, and then uses the selected expert network to extract features and introduces residual connection to fuse shallow features and deep features, realizing progressive feature extraction. In this way, the embodiment can gradually extract modal multi-scale features from shallow to deep for specific modalities that interact with other modal information, capture more and more complex modal features, effectively deal with the data diversity problem (i.e. intra-modality heterogeneity) caused by factors such as genetic mutation, differences in tumor microenvironment, and the modal missing problem (i.e. intra-modality sparsity) caused by factors such as tumor distribution and insufficient blood samples, thereby improving the feature extraction capability of the survival prediction model and improving the accuracy of survival prediction.

[0134] Optionally, for the isolated unit of the pathological collaborative processing branch, the expert network of the second expert module is a convolutional neural network; and for the isolated unit of the gene collaborative processing branch, the expert network of the second expert module is a self-normalized neural network. In addition, during the training of the survival prediction model, for the second expert module of the collaborative unit, the hyperparameters of all multilayer perceptrons are frozen, that is, the hyperparameters of all multilayer perceptrons are not updated, and they are initialized using the model parameters in the preprocessing process; and the hyperparameters of all expert networks are trainable, that is, the hyperparameters of all expert networks are updated.

[0135] In some embodiments, referring to Figure 2 The fusion prediction module described above is used for performing the following operations when used for fusion survival prediction based on the pathological collaborative feature sequence and the gene collaborative feature sequence to obtain the survival rate of the cancer patient in the future time period:

[0136] Fusing the multiple pathological collaborative feature markers in the pathological collaborative feature sequence and the multiple gene collaborative feature markers in the gene collaborative feature sequence to obtain a target local feature;

[0137] Fusing the classification markers in the pathological collaborative feature sequence and the classification markers in the gene collaborative feature sequence to obtain a target global feature;

[0138] The target's local features and global features are fused to obtain the target fused features;

[0139] Survival prediction is performed based on target fusion features to obtain the survival rate of cancer patients in the future.

[0140] In this embodiment, the aforementioned fusion prediction module aims to extract local features through low-rank fusion, then fuse global and local features to generate a final survival prediction feature representation. Based on this feature representation, predictions are then made to obtain the survival rate of cancer patients in the future. Specifically, the pathological co-processing feature sequence is the output of the pathological co-processing branch in the co-processing structure, which includes classification labels and multiple pathological co-processing feature labels; the gene co-processing feature sequence is the output of the gene co-processing branch in the co-processing structure, which includes classification labels and multiple gene co-processing feature labels. Here, the classification label refers to the classification label introduced by and processed by the co-processing structure, and the co-processing feature label refers to the interaction feature label processed by the co-processing structure, such as... Figure 2 As shown, the "features after collaboration" output by the "collaboration unit" are the collaborative feature labels.

[0141] In the fusion prediction module, such as Figure 2 As shown in the "low-rank fusion" method, firstly, multiple pathological co-signature markers in the pathological co-signature sequence and multiple gene co-signature markers in the gene co-signature sequence are fused using the low-rank fusion method. Specifically, this involves decomposing multiple pathological signature markers in the pathological co-signature sequence into a low-rank subspace, resulting in... A matrix of pathological parameters is obtained, and multiple gene feature markers in the gene co-signature sequence are decomposed into low-rank subspaces. A matrix of gene parameters, and then based on A matrix of pathological parameters The target local features are obtained by fusing the gene parameter matrix, multiple pathological feature markers in the pathological co-signature sequence, and multiple gene feature markers in the gene co-signature sequence. It is understood that the decomposition to a low-rank subspace is an existing technology and will not be elaborated further. For ease of understanding, the target local features are represented by the following formula (9):

[0142] (9);

[0143] In equation (9), ; Represent local features of the target; This represents multiple pathological feature markers in a pathological co-signature sequence; This represents multiple gene feature markers within a gene co-signature sequence. Indicates the first a pathological parameter matrix; represents the gene parameter matrix.

[0144] As shown in the "classification head" in Figure 2 , after completing the low-rank fusion, the classification labels in the pathological collaborative feature sequence and the classification labels in the gene collaborative feature sequence are processed by a multilayer perceptron to fuse the global information of the two modalities, so as to obtain a target global feature. For ease of understanding, the target global feature is represented as formula (10) as follows:

[0145] (10);

[0146] In formula (10), represents the target global feature; represents the classification labels in the gene collaborative feature sequence; represents the classification labels in the pathological collaborative feature sequence.

[0147] Next, the target local feature and the target global feature are fused by linear combination to obtain a target fusion feature, which effectively improves the comprehensive representation ability of the key features of survival prediction, and is represented as formula (11) as follows:

[0148] (11);

[0149] In formula (11), is the target fusion feature; is a parameter for controlling the fusion ratio of local features and global features.

[0150] After obtaining the target fusion feature, the target fusion feature is input into a multilayer perceptron, and a probability distribution of the survival risk of a cancer patient is generated by the multilayer perceptron and a Sigmoid function, and the survival probability of each future time is calculated, so as to obtain the survival rate of the cancer patient in a future time period, as shown in formula (12) as follows:

[0151] (12);

[0152] In formula (12), represents the survival rate of the cancer patient in the future time period, which includes future time , future time , …, future time ; represents a sigmoid function.

[0153] In some embodiments, the above method can further include:

[0154] During the training of the survival prediction model, the survival prediction model is updated in parameters by using a negative log-likelihood loss function.

[0155] In this embodiment, in order to improve the performance of the survival prediction model, during the training of the model, the loss of the survival prediction model is calculated by the negative log-likelihood loss function shown in formula (13) as follows, and the gradient backpropagation is performed based on the loss to update the model parameters of the survival prediction model, such as the hyperparameters of the above isolation processing structure, the above interaction module, the above collaborative processing structure and the above fusion prediction module:

[0156] (13);

[0157] In formula (13), represents the loss of the survival prediction model; is an event indicator variable, which is one if the patient has the target event (i.e. death), and zero otherwise; represents the survival rate at time t when the sample is input as input; represents the survival rate at time t when the sample is input as input; represents the survival rate at time t when the sample is input as input; represents the cancer probability at time t when the sample is input as input. The survival rate at time t when the sample is input as input satisfies the following formula (14):

[0158] (14);

[0159] In formula (14), represents the cancer probability at time t when the sample is input as input, .

[0160] In order to facilitate the understanding of the above-mentioned multi-modal survival prediction method in the embodiments of the present application, the principle of the above-mentioned multi-modal survival prediction method in the embodiments of the present application will be explained in an application scenario.

[0161] ​​In the application scenario, the training data samples include pathological image samples and genomics data samples of multiple patients, and the labels of the training data samples are actual inventory conditions and event indicator variables of each patient at a certain time. If a patient dies at a certain time, the event indicator variable is a first numerical value, otherwise a second numerical value. The survival prediction model is trained by the training data samples and the labels. During the model training, the loss functions shown in the above formulas (13)-(14) are used to update the model parameters. In real-time prediction, the patient to be tested is a lung adenocarcinoma patient, the whole field digital slice image and the genomics data of the lung adenocarcinoma patient are obtained through a preset database, and the whole field digital slice image and the genomics data of the lung adenocarcinoma patient are input into the survival prediction model to predict the survival probability of the lung adenocarcinoma patient in a future time period.

[0162] Reference Figure 3 In the survival prediction model, the following operations are specifically performed:

[0163] S01, the whole field digital slice image and the genomics data are input into an isolation processing structure. The isolation processing structure includes a parallel pathological isolation processing branch and a genetic isolation processing branch. In the genetic isolation processing branch, the genomics data is first divided into six functional groups of tumor suppression, protein kinase, protein kinase, cell differentiation, transcription, cytokine and growth factor through a first input layer, then feature extraction is performed on the data of each functional group through a self-normalization neural network to obtain genetic feature markers of each functional group and splice them to obtain a genomics feature sequence, and finally a first classification marker corresponding to the genomics is added at the starting position of the genomics feature sequence to obtain a first input sequence corresponding to the genomics, which includes the first classification marker and a plurality of genomics feature markers; feature extraction is performed on the first input sequence corresponding to the genomics through two isolation units; the output of the second isolation unit is aggregated and optimized through an optimization module to obtain a genetic isolation feature sequence, which includes a plurality of genetic isolation feature markers. In the pathological isolation processing branch, the pathological image is first divided into a plurality of non-overlapping image blocks through a first input layer, then feature extraction is performed on each image block using a clustering constraint attention multi-instance learning model to obtain a plurality of pathological feature markers and splice them to obtain a pathological feature sequence, and finally a second classification marker corresponding to the pathological feature sequence is added at the starting position of the pathological feature sequence to obtain a first input sequence corresponding to the pathological feature sequence, which includes the second classification marker and a plurality of pathological feature markers; feature extraction is performed on the first input sequence corresponding to the pathological feature sequence through two isolation units; the output of the second isolation unit is aggregated and optimized through an optimization module to obtain a pathological isolation feature sequence, which includes a plurality of pathological isolation feature markers.

[0164] Furthermore, within the isolation unit, firstly, the input of the isolation unit is encoded using a Transformer encoder to obtain a first encoded feature sequence. Specifically, in the isolation unit of the pathological isolation processing branch, positional encoding is added to the first encoded feature sequence after the encoding process is completed. Then, the first expert module performs progressive feature extraction on the first encoded feature sequence to obtain the output of the isolation unit, such as... Figure 4 As shown, the process involves: processing the first encoded feature sequence using a first multilayer perceptron (MLP), then processing the first encoded feature sequence using a first expert network, and performing a residual concatenation between the outputs of the first MLP and the first expert network to obtain a first residual feature sequence; processing the first residual feature sequence using a second MLP, determining a first target network from two second expert networks using a gating mechanism consisting of a MLP and a Softmax function, processing the first residual feature sequence using the first target network, and performing a residual concatenation between the outputs of the second MLP and the first target network to obtain a second residual feature sequence; processing the second residual feature sequence using a third MLP, determining a second target network from four third expert networks using a gating mechanism consisting of a MLP and a Softmax function, processing the second residual feature sequence using the second target network, and performing a residual concatenation between the outputs of the third MLP and the second target network to obtain the output of the isolation unit. It can be understood that the output of the isolation unit includes a classification label processed by the isolation unit and multiple omics feature labels processed by the isolation unit.

[0165] Furthermore, referring to Figure 5 In the optimization module, firstly, a multilayer perceptron is used to reduce the dimensionality of the output of the second isolation unit, resulting in a dimensionality-reduced output. Each omics feature label in the dimensionality-reduced output is then concatenated with a classification label in the same output, yielding N concatenated labels. Next, a gating mechanism consisting of a multilayer perceptron and a Softmax function is used to calculate the importance score of each concatenated label, resulting in the scores of the N concatenated labels. Based on these scores, the N concatenated labels are sorted from largest to smallest. A first category group is constructed using K concatenated labels with scores greater than or equal to a first threshold, and a second category group is constructed using NK concatenated labels with scores less than the first threshold. Finally, the K concatenated labels in the first category group are further divided to obtain... First choice marker and A second choice label; furthermore, in the label dimension, The second selection marker and the NK concatenation markers in the second category group are concatenated to obtain a concatenated sequence, which is then pooled to generate... a pooling mark; finally, the a first selection mark and a pooling mark are added to obtain K refined features, i.e. the output of the isolation processing branch.

[0166] S02, the pathological isolation feature sequence and the gene isolation feature sequence are interacted by the cross-modal attention mechanism shown in the above formulas (6)-(8) to obtain a pathological interaction feature sequence and a gene interaction feature sequence. The pathological interaction feature sequence includes multiple pathological interaction feature marks, and the gene interaction feature sequence includes multiple gene interaction feature marks.

[0167] S03, the pathological interaction feature sequence and the gene interaction feature sequence are input into a cooperative processing structure, which includes a parallel pathological cooperative processing branch and a gene cooperative processing branch. In the gene cooperative processing branch, a third classification mark corresponding to genomics is added to the starting position of the gene interaction feature sequence by the second input layer to obtain a second input sequence corresponding to genomics, which includes the third classification mark and multiple gene interaction feature marks; two cooperative units are used to extract features from the second input sequence corresponding to genomics, and the second cooperative unit outputs a gene cooperative feature sequence, which includes the third classification mark processed by the cooperative unit and multiple gene cooperative feature marks. In the pathological cooperative processing branch, a fourth classification mark corresponding to pathology is added to the starting position of the pathological interaction feature sequence by the second input layer to obtain a second input sequence corresponding to pathology, which includes the fourth classification mark and multiple pathological interaction feature marks; two cooperative units are used to extract features from the second input sequence corresponding to pathology, and the second cooperative unit outputs a pathological cooperative feature sequence, which includes the fourth classification mark processed by the cooperative unit and multiple pathological cooperative feature marks. The processing principle of the cooperative unit is the same as that of the isolation unit, and will not be described here.

[0168] S04, the multiple pathological feature markers in the pathological synergy feature sequence and the multiple gene feature markers in the gene synergy feature sequence are fused by a low-rank fusion method, as shown in formula (9) above, to obtain a target local feature; meanwhile, the classification markers in the pathological synergy feature sequence and the classification markers in the gene synergy feature sequence are processed by a multilayer perceptron, as shown in formula (10) above, to obtain a target global feature; the target local feature and the target global feature are fused by a linear combination, as shown in formula (11) above, to obtain a target fusion feature; the target fusion feature is input into the multilayer perceptron, and a probability distribution of the survival risk of the cancer patient is generated by the multilayer perceptron and a Sigmoid function, and the survival probability at each future time is calculated, so as to obtain the survival rate of the cancer patient in the future time period, as shown in formula (12) above.

[0169] The performance of the survival prediction method of the embodiments of the present application will be verified by Examples 1 to 3 below.

[0170] Example 1: On the public bladder cancer (BLCA) dataset, breast cancer (BRCA) dataset, endometrial cancer (UCEC) dataset, glioma (GBMLGG) dataset and lung adenocarcinoma (LUAD) dataset, the performance of the survival prediction method of the present application is compared with that of mainstream survival prediction methods, and the results are shown in Table 1 below. In Table 1, P refers to pathological images, G refers to genomics data, the check mark indicates that the data of this modality is selected, the index is the accuracy, “AdaMHF” is the survival prediction method of the present application, and the rest are mainstream methods. As can be seen from Table 1, the present application has achieved the best value in the survival prediction tasks of bladder cancer patients, breast cancer patients, endometrial cancer patients, glioma lung adenocarcinoma patients.

[0171] Table 1: Performance table of the survival prediction method of the present application and mainstream survival prediction methods on different survival prediction tasks

[0172]

[0173] Example 2: The survival prediction method of the present application is verified on the public lung adenocarcinoma dataset by a statistical analysis method, and the results are shown in Figure 5 “Low Risk” represents the low-risk group, “High Risk” represents the high-risk group, the vertical coordinate “Overall Survival” represents the proportion of individuals still alive at the current time point, and the horizontal coordinate “Time (Months)” represents the passage of time, with the unit being months. As can be seen from Figure 6 , the p-value (i.e. p-value) of the present application reaches , which can accurately predict the survival probability of cancer patients in the future time period.

[0174] Example 3: On the disclosed lung adenocarcinoma dataset, the survival prediction method of the present application is compared with mainstream survival prediction methods in terms of overhead and performance, and the results are shown in FIG. 3. Figure 6 As shown in FIG. 3, the horizontal coordinate "FLOPs (G)" represents the computational complexity, i.e., the number of floating-point operations per second, with the unit of gigabytes, and the vertical coordinate "C-index" represents the consistency index, which is the full name of the performance evaluation index, "AdaMHF" is the survival prediction method of the present application, and the rest are mainstream methods. Through ​ It can be seen that the present application can significantly reduce the computational complexity of the survival prediction task, while achieving the best survival prediction effect.

[0175] In addition, the present application also provides a survival prediction device, which can include an acquisition module and a processing module, the acquisition module is used to acquire the pathological image and the genomic data of the cancer patient, and the processing module is used to perform survival prediction on the pathological image and the genomic data by using a preset survival prediction model to obtain the survival rate of the cancer patient in a future time period; wherein the survival prediction model includes an isolation processing structure, an interaction module, a collaborative processing structure and a fusion prediction module, the isolation processing structure is used to extract features from the pathological image and the genomic data to obtain a pathological isolated feature sequence and a genomic isolated feature sequence; the interaction module is used to interact the pathological isolated feature sequence and the genomic isolated feature sequence to obtain a pathological interaction feature sequence and a genomic interaction feature sequence; the collaborative processing structure is used to extract features from the pathological interaction feature sequence and the genomic interaction feature sequence to obtain a pathological collaborative feature sequence and a genomic collaborative feature sequence; and the fusion prediction module is used to perform fusion survival prediction based on the pathological collaborative feature sequence and the genomic collaborative feature sequence to obtain the survival rate of the cancer patient in the future time period.

[0176] The contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0177] In summary, the application can effectively solve the sparsity and heterogeneity problem in multi-modal survival prediction, and improve the accuracy and robustness of survival prediction. By introducing the expert module and the optimization module, the application realizes feature extraction and optimization for intra-modal and inter-modal heterogeneity. Among them, the expert module extracts specific features for different modal data by dynamically activating the expert network, effectively dealing with the diversity and missing of data; while the optimization module selects the most information value label and aggregates the redundant label, which retains the key information while greatly reduces the computational complexity. In addition, the fusion prediction module combines local information and global information, and effectively improves the comprehensive representation ability of data through low-rank fusion and global label extraction. A large number of experimental results show that the application performs superior performance under complete modal and missing modal conditions, surpassing the existing mainstream methods. At the same time, the application realizes efficient feature extraction and fusion in the resource-constrained scene, significantly reduces the computational overhead, is suitable for large-scale medical data analysis and survival prediction tasks, and has strong practicality and popularization value.

[0178] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks can sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of the application are provided by way of example only. The disclosed methods are not limited to the operational and logical flowcharts presented in this document. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed. Furthermore, although the application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the application. Rather, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be within the routine skill of the engineer, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein. Therefore, those skilled in the art, using ordinary skill, can implement the application as set forth in the claims without undue experimentation. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the application, which is determined by the full scope of the appended claims and their equivalents.

[0179] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of programs for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer readable medium for use by or in connection with a program execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can take programs from a program execution system, apparatus or device and execute them, or in conjunction with these program execution systems, apparatus or devices. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with program execution systems, apparatus or devices, or in conjunction with these program execution systems, apparatus or devices.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Additionally, computer-readable media can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory. It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable program execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0182] In the foregoing description of this specification, the reference to terms such as "one embodiment," "another embodiment," or "some embodiments" indicates that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents. The above is a detailed description of preferred embodiments of this application, but this application is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A multi-modal survival prediction method, characterized in that, The method comprises the following steps: obtaining pathological images and genomic data of a cancer patient; using a preset survival prediction model to perform survival prediction on the pathological images and the genomic data to obtain a survival rate of the cancer patient in a future time period; wherein the survival prediction model comprises: an isolation processing structure for performing feature extraction on the pathological images and the genomic data to obtain pathological isolated feature sequences and genomic isolated feature sequences; an interaction module for performing feature interaction on the pathological isolated feature sequences and the genomic isolated feature sequences to obtain pathological interaction feature sequences and genomic interaction feature sequences; a cooperative processing structure for performing feature extraction on the pathological interaction feature sequences and the genomic interaction feature sequences to obtain pathological cooperative feature sequences and genomic cooperative feature sequences; a fusion prediction module for performing fusion-type survival prediction based on the pathological cooperative feature sequences and the genomic cooperative feature sequences to obtain the survival rate of the cancer patient in the future time period.

2. The multimodal survival prediction method of claim 1, wherein, The isolation processing structure comprises two parallel isolation processing branches, one input of one of the isolation processing branches is the pathological images, the output of one of the isolation processing branches is the pathological isolated feature sequences, the input of the other of the isolation processing branches is the genomic data, and the output of the other of the isolation processing branches is the genomic isolated feature sequences; The isolation processing branch comprises a first input layer, an optimization module and two isolation units connected in sequence; The first input layer is used to obtain a first input sequence according to the input of the isolation processing branch and a preset classification label; The input of the first isolation unit is the first input sequence, and the isolation unit is used to perform feature extraction on the input of the isolation unit to obtain the output of the isolation unit; The optimization module is used to optimize the output of the second isolation unit to obtain the output of the isolation processing branch.

3. The multimodal survival prediction method of claim 2, wherein, The isolation unit comprises: a first encoder for performing encoding processing on the input of the isolation unit to obtain a first encoding feature sequence; a first expert module for performing progressive feature extraction on the first encoding feature sequence to obtain the output of the isolation unit.

4. The multimodal survival prediction method of claim 3, wherein, The progressive feature extraction on the first encoding feature sequence to obtain the output of the isolation unit comprises: using a preset first multilayer perceptron and a preset first expert network to process the first encoding feature sequence to obtain a first residual feature sequence; using a preset second multilayer perceptron and two preset second expert networks to process the first residual feature sequence to obtain a second residual feature sequence; using a preset third multilayer perceptron and four preset third expert networks to process the second residual feature sequence to obtain the output of the isolation unit.

5. The multimodal survival prediction method of claim 2, wherein, The optimization of the output of the second isolation unit to obtain the output of the isolation processing branch comprises: splicing classification labels and a plurality of omics feature labels in the output of the second isolation unit to obtain a plurality of spliced labels and score information of each spliced label; The plurality of stitching marks are classified based on the score information of each stitching mark, to obtain a first category group and a second category group, the first category group including a plurality of stitching marks with score information greater than or equal to a preset first threshold, and the second category group including a plurality of stitching marks with score information less than the first threshold; The first category group is selectively refined to obtain a plurality of first selection marks and a plurality of second selection marks; Adaptive pooling is performed on the second category group and the plurality of second selection marks to obtain a plurality of pooled marks; The output of the isolation processing branch is obtained according to the plurality of first selection marks and the plurality of pooled marks.

6. The multimodal survival prediction method of claim 1, wherein, The pathological isolation feature sequence and the genetic isolation feature sequence are interacted to obtain a pathological interaction feature sequence and a genetic interaction feature sequence, including: The pathological isolation feature sequence and the genetic isolation feature sequence are processed using a cross-modal attention mechanism to obtain the pathological interaction feature sequence and the genetic interaction feature sequence.

7. The multimodal survival prediction method of claim 1, wherein, The cooperative processing structure includes two parallel cooperative processing branches, one input of one of the cooperative processing branches is the pathological interaction feature sequence, one output of one of the cooperative processing branches is the pathological cooperative feature sequence, one input of the other of the cooperative processing branches is the genetic interaction feature sequence, and one output of the other of the cooperative processing branches is the genetic cooperative feature sequence. The cooperative processing branch includes a second input layer and two sequentially connected cooperative units; the second input layer is used to obtain a second input sequence according to the input of the cooperative processing branch and a preset classification mark; the input of the first cooperative unit is the second input sequence, and the output of the second cooperative unit is the output of the cooperative processing branch; the cooperative unit is used to extract features from the input of the cooperative unit to obtain the output of the cooperative unit.

8. The multimodal survival prediction method of claim 1, wherein, The fusion survival prediction is performed based on the pathological cooperative feature sequence and the genetic cooperative feature sequence to obtain the survival rate of the cancer patient in a future time period, including: A plurality of pathological cooperative feature marks in the pathological cooperative feature sequence and a plurality of genetic cooperative feature marks in the genetic cooperative feature sequence are fused to obtain a target local feature; Classification marks in the pathological cooperative feature sequence and classification marks in the genetic cooperative feature sequence are fused to obtain a target global feature; The target local feature and the target global feature are fused to obtain a target fusion feature; The survival prediction is performed based on the target fusion feature to obtain the survival rate of the cancer patient in a future time period. 9.The multi-modal survival prediction method of claim 1, wherein, The method further includes the following steps: During training of the survival prediction model, the survival prediction model is updated in parameters using a negative log-likelihood loss function.

10. A multi-modal survival prediction apparatus, characterized by, Including: An acquisition module is configured to acquire a pathological image and genomics data of a cancer patient; A processing module is configured to perform survival prediction on the pathological image and the genomics data using a preset survival prediction model to obtain a survival rate of the cancer patient in a future time period. The survival prediction model comprises: An isolation processing structure configured to perform feature extraction on the pathology image and the genomic data to obtain a pathology isolation feature sequence and a genomic isolation feature sequence; An interaction module configured to perform feature interaction on the pathology isolation feature sequence and the genomic isolation feature sequence to obtain a pathology interaction feature sequence and a genomic interaction feature sequence; A cooperative processing structure configured to perform feature extraction on the pathology interaction feature sequence and the genomic interaction feature sequence to obtain a pathology cooperative feature sequence and a genomic cooperative feature sequence; A fusion prediction module configured to perform fusion survival prediction based on the pathology cooperative feature sequence and the genomic cooperative feature sequence to obtain a survival rate of the cancer patient in a future time period.

Images