Multi-modal cancer biomarker prediction method oriented to modal deletion

Abstract

The invention discloses a modal deficiency-oriented multi-modal cancer biomarker prediction method. The method comprises the following steps: S1, acquiring a pathological image and an MRI image of a patient; s2, cutting the pathological image, extracting a 3D tumor block in the MRI image, and performing data enhancement processing; s3, performing feature extraction on the pathological image, and performing feature extraction on a 3D tumor block in the MRI image to obtain a multi-modal feature; s4, performing feature enhancement and feature reconstruction on the multi-modal features; and S5, fusing the enhanced multi-modal features through a graph fusion network, and inputting the fused features into a preset biomarker prediction model for prognosis evaluation. According to the method, the reconstruction of the missing modal features is realized, and the accuracy of biomarker prediction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cancer prediction, and more specifically, to a multimodal cancer biomarker prediction method oriented to modality loss. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, new possibilities have been provided for the prediction of cancer biomarkers. Traditional biomarkers usually only use a single data modality and cannot obtain the relationship between multiple data sources. When processing multimodal data, it is necessary to manually or rely on simple feature extraction methods, which makes it difficult to obtain the intrinsic relationship between data and cannot fully explore the potential information and advanced features of the data.

[0003] There are two types of 3D feature extraction in traditional MRI. One is to extract 3D texture, size and other features by combining the original MRI image and the manually annotated mass area using the pyradiomics library, without obtaining deeper features. The other is to directly extract deep learning features from the original MRI image without making full use of the mask.

[0004] With the improvement of medical equipment and technology, the types of data that can be obtained have diversified and high-dimensional characteristics. MRI images and pathological images can provide cancer prognosis information from macro and micro perspectives. However, the existing methods in related technologies lack the ability to represent the association of multimodal information, cannot make predictions in the absence of modalities, and the association structure between features is unclear, and the interpretability is poor.

[0005] In summary, it is necessary to propose a multimodal cancer biomarker prediction method for modality loss. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal cancer biomarker prediction method for modality loss to overcome the defects of the prior art.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A multimodal cancer biomarker prediction method for modality loss, comprising the following steps:

[0009] S1. Obtain pathological images and MRI images of the patient;

[0010] S2, cropping the pathological image, extracting the 3D tumor mass in the MRI image and performing data enhancement processing;

[0011] S3, extracting features from the pathological image, and extracting features from the 3D tumor mass in the MRI image to obtain multimodal features;

[0012] S4, performing feature enhancement and feature reconstruction on the multimodal features;

[0013] S5. The enhanced multimodal features are fused through a graph fusion network and input into the preset biomarker prediction model for prognosis evaluation.

[0014] Furthermore, the step of clipping the pathological image in step S2 specifically includes:

[0015] Using a threshold-based segmentation algorithm to crop the pathological image;

[0016] Sliding the pathological image with a fixed window and step size to generate non-overlapping image blocks;

[0017] The feature splicing function is used to reconstruct the cropped image and the image blocks after sliding processing to generate the overall heat map.

[0018] Furthermore, in step S2, 3D annotation software is used to extract a 3D tumor mass in the MRI image, and the step of data enhancement processing specifically includes:

[0019] According to the maximum loss rate parameter, calculate the number of pixels that need to be deleted;

[0020] Select the starting position, width, height and depth of the deletion block, which is filled with randomly generated content from the image or selected from other image blocks for replacement;

[0021] Randomly choose an angle to rotate the image.

[0022] Furthermore, in step S3, a ResNet50 model is used to extract multi-level pathological features including spatial information and local features from the pathological image.

[0023] Furthermore, the ResNet50 model includes a 50-layer deep neural network. Through the combination of basic convolution blocks and residual connections, the feature extraction process includes multiple 2D convolution layers, followed by Batch Normalization layers and activation layers, and downsampling layer by layer. By using convolution with a stride of 2 and maximum pooling operations, feature extraction is performed at different levels. The output of each residual block is fused with the features of the current layer, and information from previous layers is transmitted through jump connections. The ResNet50 model uses a global average pooling layer instead of a fully connected layer to extract spatially invariant feature representations.

[0024] Furthermore, the step of extracting features of the 3D tumor region in the MRI image in step S3 specifically includes: using a Swin Transformer model in combination with a mask encoder model to deeply extract spatial features of the 3D tumor region.

[0025] Furthermore, the step of using the Swin Transformer model in combination with the mask encoder model to deeply extract the spatial features of the 3D tumor area specifically includes:

[0026] The MRI image is input into the image feature extraction module of the Vision Transformer architecture. The SwinTransformer model slices the input MRI image in the spatial dimension and uses the self-attention mechanism to capture the dependencies between different regions in the image, outputting feature maps at multiple scales.

[0027] The 3D tumor blocks extracted from the MRI images are input to the mask encoder, which uses 3D structure layers, normalization and ReLU image activation functions between layers to obtain blank codes marking the tumor area, which are encoded into features of the same dimension as the SwinViT output and matched by interpolating their spatial dimensions;

[0028] The MRI image features and mask features are added and fused.

[0029] Furthermore, the step of performing feature enhancement on the multimodal feature in step S4 specifically includes:

[0030] The multimodal feature is represented as X=(X w ,X r ), where X w and X r Represent pathological and imaging features, respectively. and represents the missing pathological and imaging features, and m represents the missing modality;

[0031] A one-dimensional convolution is designed to capture the global and local information of the features. The pathological and imaging features are convolved separately to obtain C w and C r :

[0032]

[0033] In the formula, kal w , kal r The convolution kernels representing pathology and imaging are respectively used, and each modality is put into the convolution to obtain the feature information of different modalities;

[0034] The intra-modal encoder is used to extract three intra-modal features respectively, including encoding the intra-modal information by the intra-modal encoder, and using the position embedding module to improve the convolution sequence and extract features across different modalities. The results of intra-modal feature extraction are:

[0035]

[0036] Where f(·) represents the position embedded transformer encoder, X w→w , X r→r are the features extracted from the pathology and imaging modalities, respectively. Q, K, and V are extracted from the same modality by the internal encoder and are used to calculate the attention weights.

[0037] The inter-modality features are obtained by processing the modality pair through the inter-modality transformer encoder. The modality pair represents the correspondence between the modality and the other two modalities. The inter-modality features X of the pathological modality are obtained from the modality pair. r→w :

[0038]

[0039] The inter-modality feature X of the image modality obtained from the image modality pair w→r :

[0040]

[0041] Among them, the pathology modality pair is pathology->imaging (w->r), and the imaging modality pair is imaging->pathology (r->w);

[0042] The latent features obtained by all intra-modality and inter-modality encoders are concatenated as the enhanced sequence feature output, and the intra-modality and inter-modality features of pathology and imaging are concatenated as U w and U r :

[0043] U w =Concat([X r→w ,X w→w ]),U r =Concat([X w→r ,X r→r ])

[0044] The enhanced sequence feature U w and U r Input to the bimodal transformer for feature enhancement to obtain U' w and U' r :

[0045] U′ w =f transformer (U w,U w ,U w ),U′ r =f transformer (U r ,U r ,U r );

[0046] The step of reconstructing the multimodal features specifically includes:

[0047] The attention mechanism is used to encode the missing modality and learn the relationship between features, and the reconstruction result is mapped to the dimension of the extracted features. The formula for feature reconstruction is:

[0048]

[0049] In the formula, attention(·) represents the attention mechanism, U' w , U' r They represent the pathological and imaging features obtained by multimodal features, linear represents the linear mapping function, and E w 、E r Represent the reconstructed features of pathology and imaging modality, respectively.

[0050] Furthermore, the step S5 specifically includes:

[0051] The extracted two modal features are fused into an overall feature vector U' = GFN (U' w ,U' r ), and input it into the multi-layer perceptron network for final prognosis prediction;

[0052] Among them, GFN consists of two dynamic learning layers, unimodal and bimodal dynamic learning layers;

[0053] The SmoothL1Loss function is used as the loss function:

[0054]

[0055] In the formula, and loss con Represent pathological features, imaging features and overall reconstruction loss, δ w and δ r is the weight, using the binary cross entropy loss as the classification loss function loss cls :

[0056] loss cls = -[y·log(p)+(1-y)log(1-p)]

[0057] loss = θ1Losscon +θ2loss cls

[0058] Among them, y is the true label, assuming the value is 0 or 1, p is the model's predicted probability of 1, and loss con is the modality reconstruction loss, loss cls is the cancer biomarker prediction classification loss, θ1 and θ2 represent the weight parameters of reconstruction loss and classification loss, respectively;

[0059] Through the loss function, the model parameters are optimized to obtain the biomarker prediction model.

[0060] Compared with the prior art, the advantages of the present invention are: the present invention proposes missing modality reconstruction, reconstructs randomly missing modalities, improves the model prediction accuracy and the robustness of multimodal representation, and the combination of imaging features and pathological features provides macroscopic and microscopic patient information. The present invention realizes the reconstruction of missing modality features and improves the accuracy of biomarker prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 It is a flow chart of the multimodal cancer biomarker prediction method for modality loss of the present invention.

[0063] Figure 2 It is a schematic diagram of pathological image feature extraction of the present invention.

[0064] Figure 3 It is a schematic diagram of RI image feature extraction of the present invention.

[0065] Figure 4 It is a framework diagram of the prognosis prediction model of the present invention. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0067] See also Figure 1-Figure 4 As shown, this embodiment discloses a multimodal cancer biomarker prediction method for modality loss, comprising the following steps:

[0068] Step S1, obtaining a pathological image and an MRI image of a patient. Obtaining a pathological section of a patient stained with HE to obtain a pathological image and an MRI image.

[0069] Step S2: cropping the pathological image, extracting the 3D tumor mass in the MRI image and performing data enhancement processing. This step specifically includes:

[0070] Segmentation operation: The pathological image is cropped using a threshold-based segmentation algorithm, and the tissue area is extracted by combining Otsu and other techniques, and noise and holes are removed according to the set filtering parameters. The specific segmentation process is similar to the traditional convolution operation, using the thresholding operation to extract the edge information of the target area at different scales, and further optimizing the segmentation results through morphological operations (such as closing operations).

[0071] Feature extraction and patching (Patch): After segmentation, the target area is further cropped through feature extraction and patching functions. This process uses a fixed window size (256x256) and step size (256) to slide the entire image to generate non-overlapping image patches. This step is equivalent to using a fixed-size receptive field for feature extraction in the global feature extraction network. Due to the complex details of pathological images, this patching process can effectively reduce the computational complexity and retain the local information of different regions.

[0072] Stitching: After the segmentation process is completed, the feature stitching function is used to reconstruct the segmented and processed image blocks to generate a low-resolution overall heat map to enhance interpretability and visualize the relative contribution and importance of each tissue region to the model prediction. This step ensures that all segmentation results can be viewed from a global perspective by downsampling the segmented feature maps proportionally and then stitching them.

[0073] In this embodiment, a 3D tumor block in the MRI image is extracted using 3D annotation software (itk-snap) to remove background and soft tissue information irrelevant to the tumor. The steps of data enhancement processing specifically include:

[0074] Calculate the maximum number of pixels to be deleted: Calculate the number of pixels to be deleted based on the maximum loss rate parameter;

[0075] Randomly select the position and size of the deletion block: select the starting position, width, height and depth of the deletion block, and the deletion block is randomly generated from the image to fill in the content or select other image blocks to replace;

[0076] Normalized blocks: For deleted blocks, if there is no replacement block, it will be filled with random noise and the noise will be normalized.

[0077] Random rotation: Randomly select an angle (0°, 90°, 180°, or 270°) to rotate the image, simulate images from different perspectives, and enhance the model's ability to learn features from different perspectives of the image.

[0078] Step S3: extract features from the pathological image and extract features from the 3D tumor mass in the MRI image to obtain multimodal features.

[0079] This embodiment uses the ResNet50 model (which can also be replaced by other deep learning networks for extracting HE-stained slices) to extract multi-level pathological features including spatial information and local features from the pathological image.

[0080] In this embodiment, the network architecture adopts the ResNet50 architecture, uses the residual learning mechanism, and introduces jump connections to solve the gradient vanishing problem in the deep network, thereby improving the training efficiency and feature extraction ability of the model. The structure contains 50 layers of deep neural network, and forms the basis of feature extraction through the combination of basic convolution blocks (conv blocks) and residual connections.

[0081] Feature extraction: After the input image has been through multiple layers of convolution and activation (such as ReLU), the network gradually extracts low-level to high-level features. Specifically, the feature extraction process includes multiple 2D convolution layers, followed by BatchNormalization layers and activation layers, downsampling layer by layer, and effectively reducing the spatial dimension of the feature map by using convolution with a stride of 2 and maximum pooling operations, while retaining important spatial feature information.

[0082] Multi-scale feature fusion: By extracting features at different levels, the model achieves multi-scale feature fusion. The output of each residual block is not only fused with the features of the current layer, but also transmits information from previous layers through jump connections, ensuring that low-level detail information is effectively combined with high-level semantic information, thereby improving the model's sensitivity to complex structures (such as tumors).

[0083] Global average pooling: In the final stage of feature extraction, the model uses a global average pooling layer instead of a traditional fully connected layer to reduce the number of model parameters and extract spatially invariant feature representations. This step can convert high-dimensional feature maps into lower-dimensional feature vectors (such as 512 or 1024 dimensions), providing strong support for subsequent classification or other tasks.

[0084] Output processing: The output of the feature extraction module can be directly used in subsequent classification tasks or as input to other models to ensure the effective use and transmission of feature information.

[0085] In this embodiment, the step of extracting features from the 3D tumor region in the MRI image specifically includes: using a Swin Transformer model (which may also be replaced by other 3D feature extraction networks) combined with a mask encoder model to deeply extract spatial features of the 3D tumor region, the steps specifically include:

[0086] The MRI is input into the image feature extraction module, which is a backbone network based on the Vision Transformer (ViT) architecture. The Swin Transformer model slices the input 3D medical image MRI in the spatial dimension, and uses the self-attention mechanism to capture the dependencies between different regions in the image, and outputs feature maps at multiple scales.

[0087] The 3D tumor block extracted from the MRI image (also called MRI segmentation mask, which is obtained by annotating and extracting the tumor area in the MRI image data using 3D annotation software (itk-snap)) is input into the mask encoder, using a 3D structure layer (Conv3d), with normalization (InstanceNorm3d) and ReLU image activation function between layers, to obtain a blank code marking the tumor area, which is encoded into a feature of the same dimension as the SwinViT output, and matched by interpolating its spatial dimension;

[0088] The MRI image features and mask features are added and fused, that is, the features (MRI image features) obtained by inputting the MRI image into VisionTransformer and the features extracted by the mask encoder are added together to help the model combine global features with local mask information.

[0089] Before feature fusion, the mask features are adjusted to be consistent with the image feature space dimensions through interpolation operations to ensure feature dimension alignment.

[0090] By performing a pooling operation on the fused features, a final feature vector is generated for subsequent biomarker prediction.

[0091] Step S4, performing feature enhancement and feature reconstruction on the multimodal features;

[0092] Specifically, the step of performing feature enhancement on the multimodal feature specifically includes:

[0093] Missing at random pattern generation:

[0094] The multimodal feature is represented as X=(X w ,X r ), where X w and X rRepresent pathological and imaging features, respectively. and represents the missing pathological and imaging features, and m represents the missing modality.

[0095] A one-dimensional convolution is designed to capture the global and local information of the features. The pathological and imaging features are convolved separately to obtain C w and C r :

[0096]

[0097] In the formula, kal w , kal r The convolution kernels representing pathology and imaging are respectively used, and each modality is put into the convolution to obtain the feature information of different modalities.

[0098] Intra-modal feature extraction:

[0099] The intra-modal encoder is used to extract three intra-modal features respectively, including encoding the intra-modal information by the intra-modal encoder, and using the position embedding module to improve the convolution sequence and extract features across different modalities. The results of intra-modal feature extraction are:

[0100]

[0101] Where f(·) represents the position embedded transformer encoder, X w→w , X r→r are the features extracted from the pathology and imaging modalities, respectively. Q, K, and V are extracted from the same modality by the internal encoder and are used to calculate the attention weights.

[0102] The inter-modal features are obtained by processing the modality pair through the inter-modality encoder, and the modality pair represents the corresponding relationship between the modality and the other two modalities. The inter-modality features of different modalities are obtained by processing different modality pairs with the inter-modality encoder. The present invention designs two modality pairs, the pathology modality pair, namely pathology->image (w->r), and the image modality pair, namely image->pathology (r->w). The present invention uses a transformer encoder to extract features between different modalities, wherein the encoder takes paired modalities as input. Obtain the inter-modality feature X of the pathology modality from the pathology->image modality pair r→w .

[0103]

[0104] The inter-modality feature X of the image modality obtained from the image modality pair w→r :

[0105]

[0106] Among them, the pathology modality pair is pathology->imaging (w->r), and the imaging modality pair is imaging->pathology (r->w);

[0107] The latent features obtained by all intra-modality and inter-modality encoders are concatenated as the enhanced sequence feature output, and the intra-modality and inter-modality features of pathology and imaging are concatenated as U w and U r :

[0108] U w =Concat([X r→w ,X w→w ]),U r =Concat([X w→r ,X r→r ])

[0109] The enhanced sequence feature U w and U r Input to the bimodal transformer for feature enhancement to obtain U' w and U' r .

[0110] U′ w =f transformer (U w ,U w ,U w ),U′ r =f transformer (U r ,U r ,U r );

[0111] The step of reconstructing the multimodal features specifically includes:

[0112] After extracting multimodal features, we need to reconstruct the randomly missing modalities so that the entire network can learn the features of these missing modalities. In the reconstruction of multimodal missing features, we reconstruct the missing data of each single modality. In addition, we also calculate the reconstruction loss of each modality to enhance the reconstruction capability. We use the attention mechanism to encode the missing modalities and learn the relationship between features. We map the reconstruction results to the dimension of the extracted features. The formula for feature reconstruction is:

[0113]

[0114] In the formula, attention(·) represents the attention mechanism, U' w , U' r They represent the pathological and imaging features obtained by multimodal features, linear represents the linear mapping function, and E w 、E rRepresent the reconstructed features of pathology and imaging modality, respectively.

[0115] Step S5: The enhanced multimodal features are fused through a graph fusion network and input into a preset biomarker prediction model for prognosis evaluation, specifically including:

[0116] The extracted two modal features are fused into an overall feature vector U' = GFN (U' w ,U'′ r ), and input it into the multi-layer perceptron network for final prognosis prediction;

[0117] Among them, GFN consists of two dynamic learning layers, unimodal and bimodal dynamic learning layers, which handle the fusion process by modeling unimodal and bimodal interactions in sequence. In GFN, data from different modalities are usually represented as a graph, where nodes represent features extracted from each modality and edges capture the relationship or interaction between these features. After fusion of features, classification prediction is performed, taking MLP as an example.

[0118] Define the loss function loss and reconstruct the loss. Since the SmoothL1Loss function is smoother and more robust in processing outliers, smoother in gradients when the error is small, and has better tolerance for large errors, this embodiment uses the SmoothL1Loss function as the loss function.

[0119]

[0120] In the formula, and loss con Represent pathological features, imaging features and overall reconstruction loss, δ w and δ r is the weight, using the binary cross entropy loss as the classification loss function loss cls :

[0121] loss cls = -[y·log(p)+(1-y)log(1-p)]

[0122] loss = θ1loss con +θ2loss cls

[0123] Among them, y is the true label, assuming the value is 0 or 1, p is the model's predicted probability of 1, and loss con is the modality reconstruction loss, loss cls is the cancer biomarker prediction classification loss, θ1 and θ2 represent the weight parameters of the reconstruction loss and classification loss, respectively.

[0124] The present invention proposes missing modality reconstruction to reconstruct randomly missing modalities, thereby improving the prediction accuracy of the model and the robustness of multimodal representation. The combination of imaging features and pathological features provides macroscopic and microscopic patient information, effectively improving the one-sidedness of biomarker prediction and improving the accuracy of biomarker prediction.

[0125] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A multimodal cancer biomarker prediction method for modality loss, characterized in that: The following steps are involved: S1. Obtain the patient's pathological images and MRI images; S2. cropping the pathological image, extracting the 3D tumor mass in the MRI image, and performing data enhancement processing; S3. performing feature extraction on the pathological image and on the 3D tumor mass in the MRI image to obtain multimodal features; S4. performing feature enhancement and feature reconstruction on the multimodal features; S5. The enhanced multimodal features are fused through a graph fusion network and input into a preset biomarker prediction model for prognosis evaluation.

2. The method for predicting multimodal cancer biomarkers for modality loss according to claim 1, wherein: The step of trimming the pathological image in step S2 specifically includes: Using a threshold-based segmentation algorithm to crop the pathological image; Sliding the pathological image with a fixed window and step size to generate non-overlapping image blocks; The feature splicing function is used to reconstruct the cropped image and the image blocks after sliding processing to generate the overall heat map.

3. The method for predicting multimodal cancer biomarkers for modality loss according to claim 3, wherein: In step S2, 3D annotation software is used to extract the 3D tumor mass in the MRI image. The data enhancement processing step specifically includes: Calculate the number of pixels that need to be deleted based on the maximum loss rate parameter; Select the starting position, width, height and depth of the deletion block, which is filled with randomly generated content from the image or replaced with another image block; Randomly choose an angle and rotate the image.

4. The method for multimodal cancer biomarker prediction based on modality loss according to claim 1, wherein: In step S3, the ResNet50 model is used to extract multi-level pathological features including spatial information and local features from the pathological image.

5. The method for modality-missing multimodal cancer biomarker prediction according to claim 4, wherein: The ResNet50 model contains a 50-layer deep neural network. Through the combination of basic convolution blocks and residual connections, the feature extraction process includes multiple 2D convolution layers, followed by Batch Normalization layers and activation layers, and downsampling layer by layer. By using convolution with a stride of 2 and maximum pooling operations, feature extraction is performed at different levels. The output of each residual block is fused with the features of the current layer, and information from previous layers is transferred through skip connections. The ResNet50 model uses a global average pooling layer instead of a fully connected layer to extract spatially invariant feature representations.

6. The method for multimodal cancer biomarker prediction based on modality loss according to claim 5, wherein: The step of extracting features of the 3D tumor region in the MRI image in step S3 specifically includes: using a Swin Transformer model combined with a mask encoder model to deeply extract the spatial features of the 3D tumor region.

7. The method for multimodal cancer biomarker prediction based on modality loss according to claim 6, wherein: The steps of using the Swin Transformer model combined with the mask encoder model to deeply extract the spatial features of the 3D tumor area specifically include: The MRI image is input into the image feature extraction module of the Vision Transformer architecture. The SwinTransformer model slices the input MRI image in the spatial dimension and uses the self-attention mechanism to capture the dependencies between different regions in the image, outputting feature maps at multiple scales. The 3D tumor block extracted from the MRI image is input to the mask encoder, which uses a 3D structure layer with normalization and ReLU image activation function between layers to obtain a blank code marking the tumor area. It is encoded into a feature with the same dimension as the SwinViT output and matched by interpolating its spatial dimension; The MRI image features and mask features are added and fused.

8. The method for multimodal cancer biomarker prediction based on modality loss according to claim 1, wherein: The step of enhancing the multimodal features in step S4 specifically includes: The multimodal feature is expressed as X=(X w , X r ), where X w and X r Represent pathological and imaging features, respectively. and represents the missing pathological and imaging features, and m represents the missing modality; One-dimensional convolution is designed to capture the global and local information of features. Pathological and imaging features are convolved separately to obtain C w and C r : Where kal w , kal r The convolution kernels representing pathology and imaging are respectively used to put each modality into convolution to obtain the feature information of different modalities; The intra-modal encoder is used to extract three intra-modal features respectively, including encoding the intra-modal information by the intra-modal encoder, and using the position embedding module to improve the convolution sequence and extract features across different modalities. The results of the intra-modal feature extraction are: Where f(·) represents the position embedded transformer encoder, X w→w , X r→r are the features extracted from the pathology and imaging modalities, respectively. Q, K, and V are extracted from the same modality by the internal encoder and used to calculate the attention weights. The modality pair is processed by the inter-modality transformer encoder to obtain the inter-modality feature. The modality pair represents the correspondence between the modality and the other two modalities. The inter-modality feature X of the pathological modality is obtained from the modality pair. r-w : The inter-modality feature X of the image modality obtained from the image modality pair w→r : Among them, the pathology modality pair is pathology->imaging (w->r), and the imaging modality pair is imaging->pathology (r->w); The latent features obtained by all intra-modal and inter-modal encoders are concatenated as the enhanced sequence feature output, and the intra-modal and inter-modal features of pathology and imaging are concatenated as U w and U r : U w =Concat([X r→w ,X w→w ]),U r =Concat([X w-r ,X r→r ]) The enhanced sequence feature U w and U r Input to the bimodal transformer for feature enhancement to obtain U′ w and U′ r : IN w =f transformer (IN w ,IN w ,IN w ),IN r =f transformer (IN r ,IN r ,IN r ); The step of reconstructing the multimodal features specifically includes: The attention mechanism is used to encode the missing modality and learn the relationship between features. The reconstruction result is mapped to the dimension of the extracted features. The formula for feature reconstruction is: Where attention(·) represents the attention mechanism, U′ w , U′ r They represent the pathological and imaging features obtained by multimodal features, linear represents the linear mapping function, and E w 、E r Represent the reconstructed features of pathology and imaging modality, respectively.

9. The method for multimodal cancer biomarker prediction oriented to modality loss according to claim 1, characterized in that: The step S5 specifically includes: The extracted two modal features are fused into an overall feature vector U′=GFN(U′ w , U′ r ), and input it into the multi-layer perceptron network for final prognosis prediction; Among them, GFN consists of two dynamic learning layers, unimodal and bimodal dynamic learning layers; The SmoothLlLoss function is used as the loss function: Where, and loss con Represent pathological features, imaging features and overall reconstruction loss, δ w and δ r is the weight, using the binary cross entropy loss as the classification loss function loss cls : loss cls =-[y·log(p)+(1-y)log(1-p)] loss=θ1loss con +θ2loss cls Among them, y is the true label, assuming it takes a value of 0 or 1, p is the model's predicted probability of 1, and loss con is the modality reconstruction loss, loss cls is the cancer biomarker prediction classification loss, θ1 and θ2 represent the weight parameters of reconstruction loss and classification loss, respectively; Through the loss function, the model parameters are optimized to obtain the biomarker prediction model.

Images