A multimodal cancer biomarker prediction method for modality loss

Abstract

This invention discloses a method for predicting multimodal cancer biomarkers with missing modalities. The method comprises the following steps: S1, acquiring pathological images and MRI images of the patient; S2, cropping the pathological images, extracting 3D tumor blocks from the MRI images, and performing data augmentation; S3, extracting features from the pathological images and the 3D tumor blocks from the MRI images to obtain multimodal features; S4, enhancing and reconstructing the multimodal features; and S5, fusing the enhanced multimodal features through a graph fusion network and inputting them into a pre-defined biomarker prediction model for prognostic evaluation. This invention achieves the reconstruction of missing modal features, improving the accuracy of biomarker prediction.

Description

Technical Field

[0001] This invention relates to the field of cancer prediction technology, and more specifically, to a method for predicting multimodal cancer biomarkers in the context of modality loss. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology has provided new possibilities for cancer biomarker prediction. Traditional biomarkers typically utilize only a single data modality, failing to capture the interrelationships between multiple data sources. When processing multimodal data, manual methods or reliance on simple feature extraction are required, making it difficult to uncover the intrinsic connections between data and fully explore the potential information and advanced features of the data.

[0003] Traditional MRI 3D feature extraction falls into two categories. One involves combining raw MRI images with manually labeled mass regions and using the pyradiomics library to extract 3D texture, size, and other features, without capturing deeper-level features. The other involves directly extracting deep learning features from the raw MRI images, without fully utilizing the mask.

[0004] With advancements in medical equipment and technology, the types of data available are becoming increasingly diverse and multidimensional. MRI images and pathological images can provide cancer prognostic information from both macroscopic and microscopic perspectives. However, existing methods in related technologies lack the ability to represent multimodal information in relation to each other, cannot make predictions in scenarios where modalities are missing, have unclear relationships between features, and offer poor interpretability.

[0005] In summary, a method for predicting multimodal cancer biomarkers that addresses modality loss needs to be proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a method for predicting multimodal cancer biomarkers that addresses modality loss, thereby overcoming the shortcomings of existing technologies.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for predicting multimodal cancer biomarkers in response to modality loss includes the following steps:

[0009] S1. Obtain the patient's pathological and MRI images;

[0010] S2. The pathological image is cropped, the 3D tumor block in the MRI image is extracted and data enhancement processing is performed;

[0011] S3. Extract features from the pathological image and extract features from the 3D tumor block in the MRI image to obtain multimodal features;

[0012] S4. Perform feature enhancement and feature reconstruction on the multimodal features;

[0013] S5. The enhanced multimodal features are fused through a graph fusion network and input into a pre-defined biomarker prediction model for prognostic evaluation.

[0014] Furthermore, the step of cropping the pathological image in step S2 specifically includes:

[0015] The pathological image is cropped using a threshold-based segmentation algorithm;

[0016] The pathological image is slid across a fixed window and step size to generate non-overlapping image blocks;

[0017] The cropped image and the sliding image blocks are reconstructed using a feature stitching function to generate an overall heatmap.

[0018] Furthermore, in step S2, 3D annotation software is used to extract the 3D tumor mass from the MRI image. The data enhancement processing steps specifically include:

[0019] Calculate the number of pixels that need to be deleted based on the maximum loss rate parameter;

[0020] Choose the start position, width, height, and depth of the deletion block, which can be filled with randomly generated content from the image or replaced by selecting other image blocks;

[0021] Randomly select an angle and rotate the image.

[0022] Furthermore, in step S3, the ResNet50 model is used to extract multi-level pathological features from the pathological image, including spatial information and local features.

[0023] Furthermore, the ResNet50 model contains 50 layers of deep neural networks. Through a combination of basic convolutional blocks and residual connections, the feature extraction process includes multiple 2D convolutional layers, followed by Batch Normalization layers and activation layers. Downsampling is performed layer by layer. By using convolution with a stride of 2 and max pooling operations, features are extracted at different levels. The output of each residual block is fused with the features of the current layer. Information from previous layers is passed through skip connections. This ResNet50 model uses global average pooling layers instead of fully connected layers to extract spatially invariant feature representations.

[0024] Furthermore, the step of extracting features from the 3D tumor region in the MRI image in step S3 specifically includes: using the Swing Transformer model combined with the mask encoder model to perform depth extraction of the spatial features of the 3D tumor region.

[0025] Furthermore, the step of using the Swin Transformer model combined with the mask encoder model to perform depth extraction of the spatial features of the 3D tumor region specifically includes:

[0026] The MRI image is input into the image feature extraction module of the Vision Transformer architecture. The SwinTransformer model slices the input MRI image in the spatial dimension and uses a self-attention mechanism to capture the dependencies between different regions in the image, outputting feature maps at multiple scales.

[0027] The 3D tumor block extracted from the MRI image is input into the mask encoder. A 3D structure layer is used, and normalization and ReLU image activation functions are used between the layers to obtain blank codes that mark the tumor region. These codes are encoded into features with the same dimension as the output of SwinViT, and then matched by interpolating their spatial dimension.

[0028] The MRI image features and mask features are added and fused together.

[0029] Furthermore, the step of feature enhancement of the multimodal features in step S4 specifically includes:

[0030] The multimodal features are represented as X = (X w ,X r ), where X w and X r These represent pathological and imaging features, respectively. and This indicates missing pathological and imaging features, where m represents the missing modality;

[0031] One-dimensional convolution is designed to capture both global and local information of features. Pathological and imaging features are convolved separately to obtain C. w and C r :

[0032]

[0033] In the formula, kal w ,kal r The convolution kernels represent pathology and imaging respectively, and each modality is put into the convolution to obtain the feature information of different modalities;

[0034] Three intramodal features are extracted using an intramodal encoder. This includes encoding intramodal information using the intramodal encoder, improving the convolutional sequence using a position embedding module, and extracting features across different modalities. The results of the intramodal feature extraction are as follows:

[0035]

[0036] In the formula, f(·) represents the position-embedded transformer encoder, X w→w X r→r These are features extracted from the pathology and imaging modalities, respectively. Q, K, and V are features extracted from the same modality by the internal encoder and are used to calculate attention weights.

[0037] Intermodal features are obtained by processing mode pairs using an intermodal transformer encoder. Each mode pair represents the correspondence between a given mode and two other modes. For example, the intermodal features X of the pathological mode are obtained from the mode pair. r→w :

[0038]

[0039] Intermodal features X of image modes obtained from image mode pairs w→r :

[0040]

[0041] Among them, the pathology modality pair is pathology->image (w->r), and the image modality pair is image->pathology (r->w);

[0042] The latent features obtained from all intramodal and intermodal encoders are concatenated as the enhanced sequence feature output. The intramodal and intermodal features from pathology and imaging are spliced ​​into a U... w and U r :

[0043] U w =Concat([X r→w ,X w→w ]), U r =Concat([X w→r ,X r→r ])

[0044] Enhance sequence features U w and U r The input is fed into a bimodal transformer for feature enhancement to obtain U' w and U' r :

[0045] U′ w =f transformer (U wU w U w ),U′ r =f transformer (U r U r U r );

[0046] The specific steps for reconstructing the features from the multimodal features include:

[0047] The missing modalities are encoded using an attention mechanism, and the relationships between features are learned. The reconstructed results are then mapped to the dimensions of the extracted features. The formula for feature reconstruction is as follows:

[0048]

[0049] In the formula, attention(·) represents the attention mechanism, and U' w 、U' r These represent the pathological and imaging features obtained from multimodal features, respectively, and linear represents the linear mapping function, E. w E r These represent the reconstruction features of pathological and imaging modalities, respectively.

[0050] Furthermore, step S5 specifically includes:

[0051] The extracted features from the two modalities are fused into a single feature vector U' = GFN(U') using a graph fusion network. w ,U' r The data is then fed into a multilayer perceptron network for final prognostic prediction.

[0052] GFN consists of two dynamic learning layers: a single-peaked dynamic learning layer and a double-peaked dynamic learning layer.

[0053] The SmoothL1Loss function is used as the loss function:

[0054]

[0055] In the formula, and loss con δ represents the reconstruction loss of pathological features, imaging features, and overall features, respectively. w and δ r The weights are used, and the binary cross-entropy loss is used as the classification loss function. cls :

[0056] loss cls = -[y·log(p)+(1-y)log(1-p)]

[0057] loss = θ1Losscon +θ2loss cls

[0058] Where y is the true label, assumed to be 0 or 1, p is the model's prediction probability of 1, and loss is... con It is the modal reconstruction loss. cls θ1 and θ2 represent the weight parameters of the reconstruction loss and classification loss, respectively, for cancer biomarker prediction classification loss.

[0059] By optimizing the model parameters using the loss function, a biomarker prediction model can be obtained.

[0060] Compared with the prior art, the advantages of the present invention are as follows: The present invention proposes missing modality reconstruction, reconstructs random missing modalities, improves the model prediction accuracy and the robustness of multimodal representation, and the combination of imaging features and pathological features provides macroscopic and microscopic patient information. The present invention realizes the reconstruction of missing modality features and improves the accuracy of biomarker prediction. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of the multimodal cancer biomarker prediction method for modality loss according to the present invention.

[0063] Figure 2 This is a schematic diagram of pathological image feature extraction according to the present invention.

[0064] Figure 3 This is a schematic diagram of RI image feature extraction according to the present invention.

[0065] Figure 4 This is a framework diagram of the prognostic prediction model of the present invention. Detailed Implementation

[0066] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0067] See Figures 1-4 As shown in the figure, this embodiment discloses a method for predicting multimodal cancer biomarkers with modality loss, including the following steps:

[0068] Step S1: Obtain the patient's pathological and MRI images. This can be done by obtaining pathological and MRI images from HE-stained pathological sections of the patient.

[0069] Step S2: Cropping the pathological image, extracting the 3D tumor mass from the MRI image, and performing data enhancement processing. This step specifically includes:

[0070] Segmentation: A threshold-based segmentation algorithm is used to crop the pathological image, and techniques such as Otsu's are combined to extract tissue regions. Noise and holes are removed according to set filtering parameters. The specific segmentation process is similar to traditional convolution operations, using thresholding to extract edge information of the target region at different scales, and further optimizing the segmentation results through morphological operations (such as closing operations).

[0071] Feature Extraction and Patch Operation: After segmentation, the target region is further cropped using a feature extraction and patching function. This process slides across the entire image with a fixed window size (256x256) and stride (256) to generate non-overlapping image patches. This step is equivalent to using a fixed-size receptive field for feature extraction in a global feature extraction network. Due to the complex details of pathological images, this patching process can effectively reduce computational complexity and preserve local information in different regions.

[0072] Stitching: After segmentation, a feature stitching function is used to reconstruct the segmented and processed image patches, generating a low-resolution overall heatmap. This enhances interpretability and visualizes the relative contribution and importance of each tissue region to the model's predictions. This step ensures that all segmentation results can be viewed from a global perspective by downsampling the segmented feature maps proportionally before stitching.

[0073] This embodiment uses 3D annotation software (itk-snap) to extract the 3D tumor mass from the MRI image, removing the background and soft tissue information unrelated to the tumor. The data augmentation process specifically includes the following steps:

[0074] Calculate the maximum number of pixels to delete: Based on the maximum loss rate parameter, calculate the number of pixels that need to be deleted;

[0075] Randomly select the position and size of the deletion block: Select the starting position, width, height and depth of the deletion block, which will be filled with content randomly generated from the image or replaced by other image blocks;

[0076] Normalized block: For deleted blocks, if there is no replacement block, they will be filled with random noise and the noise will be normalized.

[0077] Random rotation: Randomly select an angle (0°, 90°, 180° or 270°) to rotate the image, simulating images from different perspectives, and enhancing the model's ability to learn features of images from different perspectives.

[0078] Step S3: Extract features from the pathological image and extract features from the 3D tumor block in the MRI image to obtain multimodal features.

[0079] This embodiment uses the ResNet50 model (or other deep learning networks for extracting HE-stained sections) to extract multi-level pathological features, including spatial information and local features, from the pathological image.

[0080] In this embodiment, the network architecture adopts the ResNet50 architecture, utilizing a residual learning mechanism to solve the gradient vanishing problem in deep networks by introducing skip connections, thereby improving the model's training efficiency and feature extraction capability. This structure contains 50 layers of deep neural networks, forming the basis for feature extraction through the combination of basic convolutional blocks and residual connections.

[0081] Feature extraction: After the input image undergoes multiple convolutional and activation layers (such as ReLU), the network gradually extracts features from low to high levels. Specifically, the feature extraction process includes multiple 2D convolutional layers, followed by BatchNormalization layers and activation layers, downsampling layer by layer. By using convolution with a stride of 2 and max pooling operations, the spatial dimension of the feature map is effectively reduced while retaining important spatial feature information.

[0082] Multi-scale feature fusion: By extracting features at different levels, the model achieves multi-scale feature fusion. The output of each residual block is not only fused with the features of the current layer, but also passes information from previous layers through skip connections, ensuring that low-level detailed information and high-level semantic information are effectively combined, thereby improving the model's sensitivity to complex structures (such as tumors).

[0083] Global average pooling: In the final stage of feature extraction, the model uses global average pooling layers instead of traditional fully connected layers to reduce the number of model parameters and extract space-invariant feature representations. This step can transform high-dimensional feature maps into lower-dimensional feature vectors (such as 512-dimensional or 1024-dimensional vectors), providing strong support for subsequent classification or other tasks.

[0084] Output processing: The output of the feature extraction module can be directly used for subsequent classification tasks or as input to other models, ensuring the effective use and transmission of feature information.

[0085] In this embodiment, the step of feature extraction of the 3D tumor region in the MRI image specifically includes: using the Swin Transformer model (or other 3D feature extraction networks) combined with the mask encoder model to perform depth extraction of the spatial features of the 3D tumor region. The specific steps include:

[0086] The MRI image is input to the image feature extraction module, which is based on the Vision Transformer (ViT) architecture backbone network. The Swin Transformer model slices the input 3D medical MRI image in spatial dimensions and uses a self-attention mechanism to capture the dependencies between different regions in the image, outputting feature maps at multiple scales.

[0087] The 3D tumor block extracted from the MRI image (also known as the MRI segmentation mask, which is obtained by annotating and extracting the tumor region in the MRI image data using 3D annotation software (itk-snap)) is input into the mask encoder. A 3D structure layer (Conv3d) is used, and normalization (InstanceNorm3d) and ReLU image activation function are used between layers to obtain blank codes that mark the tumor region. These codes are encoded into features of the same dimension as the output of SwinViT, and matching is performed by interpolating their spatial dimensions.

[0088] The MRI image features and mask features are added and fused together. That is, the features (MRI image features) obtained by inputting the MRI image into the VisionTransformer are added to the features extracted by the mask encoder. This helps the model combine global features and local mask information.

[0089] Before feature fusion, the mask features are adjusted to match the dimensions of the image feature space through interpolation operations to ensure feature dimension alignment.

[0090] By performing pooling operations on the fused features, a final feature vector is generated for subsequent biomarker prediction.

[0091] Step S4: Perform feature enhancement and feature reconstruction on the multimodal features;

[0092] Specifically, the steps for feature enhancement of the multimodal features include:

[0093] Random missing mode generation:

[0094] The multimodal features are represented as X = (X w ,X r ), where X w and X rThese represent pathological and imaging features, respectively. and represents the missing pathological and imaging features, and m represents the missing modality.

[0095] One-dimensional convolution is designed to capture both global and local information of features. Pathological and imaging features are convolved separately to obtain C. w and C r :

[0096]

[0097] In the formula, kal w ,kal r The convolution kernels represent pathology and imaging, respectively. Each modality is fed into the convolution to obtain feature information of different modalities.

[0098] Intramodal feature extraction:

[0099] Three intramodal features are extracted using an intramodal encoder. This includes encoding intramodal information using the intramodal encoder, improving the convolutional sequence using a position embedding module, and extracting features across different modalities. The results of the intramodal feature extraction are as follows:

[0100]

[0101] In the formula, f(·) represents the position-embedded transformer encoder, X w→w X r→r These are features extracted from the pathology and imaging modalities, respectively. Q, K, and V are features extracted from the same modality by the internal encoder and are used to calculate attention weights.

[0102] Intermodal features are obtained by processing modal pairs using an intermodal encoder. A modal pair represents the correspondence between a given modality and two other modalities. Different modal pairs are processed using the intermodal encoder to obtain intermodal features for different modalities. This invention designs two modal pairs: a pathology modal pair (pathology->imaging, w->r) and an imaging modal pair (imaging->pathology, r->w). This invention uses a transformer encoder to extract features between different modalities, where the encoder takes paired modalities as input. The intermodal features X of the pathology modality are obtained from the pathology->imaging modal pair. r→w .

[0103]

[0104] Intermodal features X of image modes obtained from image mode pairs w→r :

[0105]

[0106] Among them, the pathology modality pair is pathology->image (w->r), and the image modality pair is image->pathology (r->w);

[0107] The latent features obtained from all intramodal and intermodal encoders are concatenated as the enhanced sequence feature output. The intramodal and intermodal features from pathology and imaging are spliced ​​into a U... w and U r :

[0108] U w =Concat([X r→w ,X w→w ]), U r =Concat([X w→r ,X r→r ])

[0109] Enhance sequence features U w and U r The input is fed into a bimodal transformer for feature enhancement to obtain U' w and U' r .

[0110] U′ w =f transformer (U w U w U w ),U′ r =f transformer (U r U r U r );

[0111] The specific steps for reconstructing the features from the multimodal features include:

[0112] After extracting multimodal features, it is necessary to reconstruct the randomly missing modalities so that the entire network can learn the features of these missing modalities. In the multimodal missing feature reconstruction, the missing data of each single modality is reconstructed. Furthermore, we calculate a reconstruction loss for each modality to enhance the reconstruction capability. An attention mechanism is used to encode the missing modalities and learn the relationships between features. The reconstruction results are mapped to the dimensions of the extracted features. The formula for feature reconstruction is:

[0113]

[0114] In the formula, attention(·) represents the attention mechanism, and U' w 、U' r These represent the pathological and imaging features obtained from multimodal features, respectively, and linear represents the linear mapping function, E. w E rThese represent the reconstruction features of pathological and imaging modalities, respectively.

[0115] Step S5: The enhanced multimodal features are fused through a graph fusion network and input into a pre-defined biomarker prediction model for prognostic evaluation. This specifically includes:

[0116] The extracted features from the two modalities are fused into a single feature vector U' = GFN(U') using a graph fusion network. w ,U'′ r The data is then fed into a multilayer perceptron network for final prognostic prediction.

[0117] The GFN consists of two dynamic learning layers: a unimodal and a bimodal dynamic learning layer. These layers handle the fusion process by sequentially modeling the interactions between the unimodal and bimodal modes. In the GFN, data from different modalities are typically represented as a graph, where nodes represent features extracted from each modality, and edges capture the relationships or interactions between these features. After feature fusion, classification prediction is performed, using MLP as an example.

[0118] Define the loss function `loss` and reconstruct the loss. Since the SmoothL1Loss function handles outliers more smoothly and robustly, provides smoother gradients when errors are small, and has better tolerance for large errors, this embodiment uses the SmoothL1Loss function as the loss function.

[0119]

[0120] In the formula, and loss con δ represents the reconstruction loss of pathological features, imaging features, and overall features, respectively. w and δ r The weights are used, and the binary cross-entropy loss is used as the classification loss function. cls :

[0121] loss cls = -[y·log(p)+(1-y)log(1-p)]

[0122] loss = θ1 loss con +θ2loss cls

[0123] Where y is the true label, assumed to be 0 or 1, p is the model's prediction probability of 1, and loss is... con It is the modal reconstruction loss. cls θ1 and θ2 represent the weight parameters of the reconstruction loss and classification loss, respectively, for cancer biomarker prediction classification loss.

[0124] This invention proposes missing modality reconstruction, which reconstructs random missing modalities, improves the model's prediction accuracy and the robustness of multimodal representation. The combination of imaging features and pathological features provides macroscopic and microscopic patient information, effectively improving the one-sidedness of biomarker prediction and enhancing the accuracy of biomarker prediction.

[0125] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the patent owner may make various modifications or alterations within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present invention, they shall be within the protection scope of the present invention.

Claims

1. A method for predicting multimodal cancer biomarkers in response to modality loss, characterized in that, Includes the following steps: S1. Obtain the patient's pathological and MRI images; S2. The pathological image is cropped, the 3D tumor block in the MRI image is extracted and data enhancement processing is performed; S3. Extract features from the pathological image and extract features from the 3D tumor block in the MRI image to obtain multimodal features; S4. Perform feature enhancement and feature reconstruction on the multimodal features; S5. The enhanced multimodal features are fused through a graph fusion network and input into a pre-defined biomarker prediction model for prognostic evaluation. The step of feature enhancement of the multimodal features in step S4 specifically includes: The multimodal features are represented as X = (X w ,X r ), where X w and X r These represent pathological and imaging features, respectively. and This indicates missing pathological and imaging features, where m represents the missing modality; One-dimensional convolution is designed to capture both global and local information of features. Pathological and imaging features are convolved separately to obtain C. w and C r : In the formula, kal w ,kal r The convolution kernels represent pathology and imaging respectively, and each modality is put into the convolution to obtain the feature information of different modalities; Three intramodal features are extracted using an intramodal encoder. This process involves the intramodal encoder encoding intramodal information, and a position embedding module improving the convolutional sequence and extracting features across different modalities. The results of the intramodal feature extraction are as follows: In the formula, f(·) represents the position-embedded transformer encoder, X w→w X r→r These are features extracted from pathology and imaging modalities, respectively. Intermodal features are obtained by processing modal pairs using an intermodal transformer encoder. Each modal pair represents the correspondence between a given modality and two other modalities. The intermodal features X of the pathological modality are then extracted from the image modal pairs. r→w : Intermodal features X of image modes obtained from image mode pairs w→r : Among them, the pathology modality pair is pathology->image (w->r), and the image modality pair is image->pathology (r->w); The latent features obtained from all intramodal and intermodal encoders are concatenated as the enhanced sequence feature output. The intramodal and intermodal features from pathology and imaging are spliced ​​into a U... w and U r : U w =Concat([X r→w ,X w→w ]),U r =Concat([X w→r ,X r→r ]) Enhance sequence features U w and U r The input is fed into a bimodal transformer for feature enhancement to obtain U' w and U' r : IN' w =f transformer (IN w ,IN w ,IN w ),IN' r =f transformer (IN r ,IN r ,IN r ); The specific steps for reconstructing the features from the multimodal features include: The missing modalities are encoded using an attention mechanism, and the relationships between features are learned. The reconstructed results are then mapped to the dimensions of the extracted features. The formula for feature reconstruction is as follows: In the formula, attention(·) represents the attention mechanism, linear represents the linear mapping function, and E w E r These represent the reconstruction features of pathological and imaging modalities, respectively.

2. The method for predicting multimodal cancer biomarkers for modality loss according to claim 1, characterized in that, The step of cropping the pathological image in step S2 specifically includes: The pathological image is cropped using a threshold-based segmentation algorithm; The pathological image is slid across a fixed window and step size to generate non-overlapping image blocks; The cropped image and the sliding image blocks are reconstructed using a feature stitching function to generate an overall heatmap.

3. The method for predicting multimodal cancer biomarkers for modality loss according to claim 1, characterized in that, In step S2, 3D annotation software is used to extract the 3D tumor mass from the MRI image. The data enhancement processing steps specifically include: Calculate the number of pixels that need to be deleted based on the maximum loss rate parameter; Choose the start position, width, height, and depth of the deletion block, which can be filled with randomly generated content from the image or replaced by selecting other image blocks; Randomly select an angle and rotate the image.

4. The method for predicting multimodal cancer biomarkers for modality loss according to claim 1, characterized in that, In step S3, the ResNet50 model is used to extract multi-level pathological features from the pathological image, including spatial information and local features.

5. The method for predicting multimodal cancer biomarkers for modality loss according to claim 4, characterized in that, The ResNet50 model comprises a 50-layer deep neural network. Through a combination of basic convolutional blocks and residual connections, the feature extraction process involves multiple 2D convolutional layers followed by Batch Normalization and activation layers, with downsampling layer by layer. By using convolutions with a stride of 2 and max pooling operations, features are extracted at different levels. The output of each residual block is fused with the features of the current layer, and information from previous layers is passed through skip connections. This ResNet50 model uses global average pooling layers instead of fully connected layers to extract spatially invariant feature representations.

6. The method for predicting multimodal cancer biomarkers for modality loss according to claim 5, characterized in that, The step S3, which involves extracting features from the 3D tumor region in the MRI image, specifically includes: using the Swing Transformer model combined with the mask encoder model to perform depth extraction of the spatial features of the 3D tumor region.

7. The method for predicting multimodal cancer biomarkers for modality loss according to claim 6, characterized in that, The steps for deep extraction of spatial features of the 3D tumor region using the Swing Transformer model combined with the mask encoder model specifically include: The MRI image is input into the image feature extraction module of the Vision Transformer architecture. The SwinTransformer model slices the input MRI image in the spatial dimension and uses a self-attention mechanism to capture the dependencies between different regions in the image, outputting feature maps at multiple scales. The 3D tumor block extracted from the MRI image is input into the mask encoder. A 3D structure layer is used, and normalization and ReLU image activation functions are used between the layers to obtain blank codes that mark the tumor region. These codes are encoded into features of the same dimension as the output of SwinViT, and then matched by interpolating their spatial dimensions. The MRI image features and mask features are added and fused together.

8. The method for predicting multimodal cancer biomarkers for modality loss according to claim 1, characterized in that, Step S5 specifically includes: The extracted features from the two modalities are fused into a single feature vector U' = GFN(U') using a graph fusion network. w ,U' r The data is then fed into a multilayer perceptron network for final prognostic prediction. GFN consists of two dynamic learning layers: a single-peaked dynamic learning layer and a double-peaked dynamic learning layer. The SmoothL1Loss function is used as the loss function: In the formula, and loss con δ represents the reconstruction loss of pathological features, imaging features, and overall features, respectively. w and δ r The weights are used, and the binary cross-entropy loss is used as the classification loss function. cls : loss cls =-[y·log(p)+(1-y)log(1-p)] loss=θ1loss con +θ2loss cls Where y is the true label, assumed to be 0 or 1, p is the model's prediction probability of 1, and loss is... con It is the modal reconstruction loss. cls θ1 and θ2 represent the weight parameters of the reconstruction loss and classification loss, respectively, for cancer biomarker prediction classification loss. By optimizing the model parameters using the loss function, a biomarker prediction model can be obtained.

Images