Brain glioma segmentation method based on CrossMAE

Abstract

The invention discloses a brain glioma segmentation method based on CrossMAE, and aims to enhance segmentation performance by extracting visual language representation through combination of MRI and a diagnostic report. According to the method, image and text information is randomly masked, masked content is reconstructed, and cross-modal deep association is learned, so that accurate segmentation of MRI brain glioma is realized. The method comprises the following steps: S1, collecting a multi-modal MR brain tumor image and a diagnosis report, and constructing a data set; s2, designing an image-text joint mask strategy, and performing selective masking processing on the data; s3, constructing an MRI (Magnetic Resonance Imaging) brain glioma segmentation model based on the CrossMAE; s4, designing a multi-task cascade loss function, and training and optimizing the segmentation model by using the data set to obtain a trained MRI brain glioma segmentation model; and S5, performing MRI brain glioma segmentation according to the trained MRI brain glioma segmentation model. According to the method, the cross-modal complementary information is effectively extracted and utilized from the MRI, so that accurate segmentation of the MRI brain glioma is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a brain glioma segmentation method based on CrossMAE. Background Art

[0002] With the rapid development of medical imaging technology, the application potential of cross-modal pre-training methods based on medical images and language in medical image understanding is receiving increasing attention. Medical data itself has multimodal characteristics, of which vision (such as radiology images, magnetic resonance imaging and computed tomography) and language (such as radiology reports, medical text and pathology reports) are the two main forms of expression. In the medical field, vision and language pre-training is considered a key technology that helps to achieve a deep joint understanding of medical images and text information. However, existing technologies still face many challenges in practical applications. For example, there is a widespread lack of large-scale annotated data in the medical field, the high degree of specialization of domain knowledge, and the complexity of efficient fusion of cross-modal information. These problems seriously limit the widespread application of cross-modal pre-training methods in tasks such as medical image segmentation and auxiliary diagnosis.

[0003] In medical image segmentation tasks, especially in MRI image segmentation of tumors such as gliomas, traditional convolutional neural network (CNN)-based methods are often limited by model complexity and the lack of room for improvement in accuracy. Existing research primarily focuses on the training and processing of single images (e.g., CT and MRI images), with little in-depth exploration of joint image and text modeling. However, images and text in the medical field are often complementary: images provide intuitive information about the lesion region, while text contains detailed descriptions of the lesion and clinical background knowledge.

[0004] Therefore, how to effectively combine image and text information to improve the accuracy of glioma image segmentation has become a key issue that needs to be addressed in this field. Cross-modal learning, especially joint modeling based on visual and language information, provides new ideas and solutions for improving the accuracy of medical image segmentation. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a brain glioma segmentation method based on CrossMAE, which provides users with a method for realizing accurate segmentation of MRI brain glioma by combining image and text feature information.

[0006] Specifically, the method comprises the following steps:

[0007] S1: Acquire multimodal MR brain tumor images and accompanying diagnostic reports to construct a dataset;

[0008] S2: Design a joint image and text masking strategy to selectively mask the data;

[0009] S3: Constructing an MRI glioma segmentation model based on CrossMAE;

[0010] S4: Design a multi-task cascade loss function, use the dataset to train and optimize the segmentation model, and obtain a trained MRI brain glioma segmentation model;

[0011] S5: Perform MRI glioma segmentation based on the trained MRI glioma segmentation model.

[0012] Preferably, S1 comprises the following steps:

[0013] S1.1: MRI Data Processing: Cropping the MRI images to remove irrelevant areas to ensure that all images are of the same size. Normalizing the pixel values ​​of each image to a uniform range (0 to 1). Using filtering techniques to remove noise from the MRI images, preserving edge features and some detail information, and ensuring the accuracy and usability of the MRI images in subsequent processing.

[0014] S1.2: Diagnostic report processing: Clean the diagnostic report to remove irrelevant information and unify the format. Extract key descriptive information related to glioma.

[0015] S1.3: Dataset partitioning: Divide the processed dataset into training set, validation set, and test set.

[0016] Preferably, S2 includes the following steps:

[0017] S2.1: Align MRI images with corresponding diagnostic reports using patient IDs to ensure that each MRI image is correctly matched to its corresponding diagnostic report. This ensures that images and text information are correctly associated during training, avoiding the impact of information mismatch.

[0018] S2.2: Considering the difference in information density between visual information and language information, random sampling with different mask ratios is used for MRI images and text respectively, so that the model can extract more valuable feature information from images and text.

[0019] Preferably, in S3, the CrossMAE-based MRI glioma segmentation model includes: a cross-modal encoder module designed based on the different characteristics of images and text, a collaborative representation module that integrates T1, T1ce, T2, Flair modalities in MRI and text feature information, an inference reconstruction module, and a decoder module for MRI glioma segmentation. The following steps are included:

[0020] S3.1: The constructed MRI glioma segmentation model uses a cross-modal encoder module, using different encoders to extract image and text features. The image encoder is composed of ViT (Vision Transformer), which extracts features from the MRI T1, T1ce, T2, and Flair modalities. The text encoder is composed of Transformer, which extracts corresponding features from the input text sequence.

[0021] S3.2: Design a collaborative representation module and introduce a collaborative attention mechanism to fuse contextual information from different MRI modalities and text sequences, thereby integrating and compressing the information of each modality while sharing necessary key information;

[0022] S3.3: Design an inference reconstruction module to learn the mapping relationship between different modalities and achieve accurate feature reconstruction in each modality. Design and calculate the joint image and text reconstruction loss to further improve the model's reconstruction capabilities.

[0023] S3.4: Use the decoder module to extract the upsampled features of the fused local features of different modalities, and gradually refine and optimize the final segmentation results to ensure high-precision segmentation effects.

[0024] S3.1 includes the following steps:

[0025] Based on the ViT image encoder, first, the input image I∈R H×W×C Split into multiple image blocks {p1,p2,…,p N}, where H×W is the image spatial resolution and C is the number of image channels. Each image block The one-dimensional feature vector is obtained by flattening the image encoder based on ViT, and the size is P 2 ×C. Next, the unmasked image block is transformed by a linear Converted to block embedding, where D represents the embedding space dimension. This process maps the high-dimensional information of each image block to a low-dimensional embedding space, thereby reducing the dimension of the information and improving computational efficiency. In order to better aggregate visual information, each image block introduces a special learnable tag embedding p I ∈R D , used to represent the global information in the image. The embeddings of all image patches are combined with the learnable one-dimensional position embedding Add. Positional embedding It is used to provide position information for the embedding of each small block. This position embedding is learnable during the training process, which helps the network better understand the spatial arrangement of each small block in the image. The input is represented as:

[0026]

[0027] Finally, the processed input representation X v Will be fed into a v The Transformer model processes these input representations through the self-attention mechanism to obtain the context-related image representation H v . The representation is a sequence of vectors expressed as: in, Represents the image feature vector output by the i-th layer Transformer. In this way, the Transformer can effectively capture local and global information in the image, thereby achieving efficient representation of the image.

[0028] The Transformer-based text encoder first uses the WordPiece algorithm to segment the text sequence into multiple word tags {w1,w2,…,w M}. Split the text sequence into smaller subword units, effectively handle unknown words and improve the vocabulary coverage of the model. All tags w M It is represented in the form of one-hot encoding, where the dimension V represents the size of the vocabulary. Then, a linear transformation matrix E l ∈R V×D Linearly project the one-hot encoded tokens into embedding vectors to map the one-hot encoded tokens to a lower-dimensional embedding space. In order to make the Transformer model understand the beginning and end of the text, two special token embedding vectors are added at the beginning and end of the text sequence, namely the sequence start token (w T ∈R D ), special delimiter marker (w SEP ∈R D ). So, the text input X l The initial representation is:

[0029] X l =[w T ;w1E l ;…;w SEP ]

[0030] Then embed the position Add to the text embedding representation to record the position of each token in the sequence. The embedding matrix contains (M+2) position embedding vectors, corresponding to the position information of each subword token, which can help Transformer capture the order relationship of tokens in the sequence, so the input is finally represented as Y l for:

[0031]

[0032] Finally, the text input represents Y l It will be fed into a Transformer model and processed by multiple Transformer layers to generate a context-related text representation H l The output vector sequence is represented as: in Indicates the start of a column. represents each subword token, Indicates a special delimiter token. In this way, Transformer can effectively process and encode information in text sequences.

[0033] S3.2 includes the following steps:

[0034] The collaborative representation module consists of two Transformer models, each of which consists of N m Each layer of Transformer consists of three sub-layers: self-attention sub-layer, cross-attention sub-layer, and feed-forward sub-layer. The attention mechanism is applied in the self-attention sub-layer and cross-attention sub-layer, which is defined as follows: ATTN(Q,K,V)=softmax(QK T )·V

[0035] In the self-attention sublayer, the interaction is performed only within the same modality to capture the information within a single modality. For the image modality and text modality, the following calculations are performed respectively: v , self-attention is H vs =ATTN(H v ,H v ,H v ); for text mode H l , self-attention is H ls =ATTN(H l ,H l ,H l );

[0036] The cross-attention sublayer is the core of the collaborative attention mechanism. In this sublayer, the representations of the image and text modalities interact, thereby fusing information from one modality into the representation of the other modality. Specifically, for the image modality H v , attention is H vc =ATTN(H vs ,H ls ,H ls ); for text mode H l , attention is H lc=ATTN(H ls ,H vs ,H vs );

[0037] Representation of image and text modalities H vc and H lc It will be input into the feedforward sublayer (MLP) to obtain the multimodal representation of image and text, which are expressed as:

[0038] S3.3 includes the following steps:

[0039] The input of the reasoning reconstruction module comes from the collaborative representation module, which uses different levels of visual and text features for reconstruction. Specifically, the visual output of the k-th layer Transformer is selected from the collaborative representation module. As the image reconstruction input. For the mask word prediction of the text, the collaborative representation module is used to finally output Z l , this representation can fully capture cross-modal contextual information and is particularly suitable for extracting semantic information in text, thereby performing accurate missing word prediction.

[0040] The goal of the inference reconstruction module is to map images and text back to their original inputs. This module contains multiple Transformer layers.

[0041] S3.4 includes the following steps:

[0042] The decoder module consists of multiple upsampling and fusion stages. This module effectively fuses the multimodal information of images and text, ultimately achieving accurate MRI brain tumor segmentation.

[0043] Preferably, S4 includes the following steps:

[0044] S4.1: The joint image-text reconstruction loss is the weighted sum of the image reconstruction loss (Irecon loss) and the text reconstruction loss (Lrecon loss):

[0045] L recon =αL Irecon +(1-α)L Lrecon

[0046] Where α is a weight coefficient, and back propagation is used during training to minimize the reconstruction task loss and optimize the model reconstruction capability.

[0047] S4.2: During training, the decoder module calculates the segmentation task loss, which consists of Dice loss and cross entropy loss:

[0048] L seg =Dice(fd(Fconcat ),Y)+CrossEntropy(fd(F concat ),Y)

[0049] Among them, Y represents the true label, Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and cross entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label.

[0050] S4.3 considers both the reconstruction task loss and the subsequent segmentation task loss to optimize the model. Therefore, a multi-task cascade loss is designed, which consists of the weighted sum of the reconstruction loss and the segmentation loss:

[0051] L total =βL recon +(1-β)L seg

[0052] Where β is a hyperparameter used to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascade loss.

[0053] Among them, S4.1 includes the following steps:

[0054] The mean square error (MSE) between the reconstructed image and the original image in pixel space is calculated:

[0055]

[0056] Where N is the number of pixels in the image, is the pixel value of the pth row and qth column of the reconstructed image, I pq is the pixel value of the p-th row and q-th column of the original image.

[0057] The text reconstruction loss is obtained by calculating the negative log-likelihood loss on the masked vocabulary tokens:

[0058]

[0059] Where M is the set of masked words, is the same as the masked word w i Related contextual vocabulary. is the masked vocabulary predicted by the model. is the probability that the model predicts the masked word based on the context.

[0060] Preferably, S5 includes the following steps:

[0061] S5.1 performs binarization processing on the multimodal tumor segmentation mask image at the same position to obtain a two-dimensional pixel matrix of the multimodal tumor image;

[0062] S5.2 superimposes the two-dimensional pixel matrices of tumor images of multiple modalities to obtain a tumor segmentation result image of the target patient.

[0063] S5.1 includes the following steps:

[0064] The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model:

[0065]

[0066] Among them, K represents the two-dimensional pixel matrix of the tumor segmentation result image, I pq Represents the pixel value at the pth row and qth column.

[0067] The beneficial effects of the method of the present invention are as follows: the present invention first sets different masking ratios for MRI images and diagnostic reports and extracts the feature information of images and texts respectively through a cross-modal encoder module, and uses an image encoder and a text encoder to perform deep feature extraction and embedding of key information in MRI images and diagnostic reports respectively. Then, the collaborative representation module introduces a collaborative attention mechanism to efficiently fuse image and text features, integrating contextual information from different modalities, so that the model can share and utilize valuable cross-modal information, and further enrich the joint representation of images and texts. Through the inference and reconstruction module, the model self-reconstructs feature information at different levels to optimize information expression and improve the mutual dependence between images and texts, ensuring more accurate cross-modal understanding. Finally, the decoder module performs layer-by-layer upsampling and convolution operations on the fused multimodal features to refine the image features, and ultimately achieve accurate MRI brain glioma segmentation. At the same time, the present invention effectively solves the challenges of existing neural network models in multimodal data processing and tumor segmentation. On the one hand, through the cross-modal feature extraction module, the model can enrich its understanding of image and text information; on the other hand, through the collaborative attention mechanism, the model's attention to image and text features is enhanced, and the fusion efficiency of multimodal features is optimized; at the same time, through the CrossMAE-based MRI segmentation model, the accuracy and visualization effect of MRI brain tumor segmentation are ultimately improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. The following drawings are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 Schematic diagram of a process of a brain glioma segmentation method based on CrossMAE of the present invention;

[0070] Figure 2 This is the overall architecture diagram of the model of a CrossMAE-based brain glioma segmentation method of the present invention. Specific implementation plan

[0071] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0072] The embodiment of the present invention provides a glioma segmentation method based on CrossMAE for combining MRI with diagnostic reports, which effectively extracts and utilizes multimodal complementary information from MR images to achieve accurate segmentation of gliomas.

[0073] A typical embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0074] S1: Acquire multimodal MR brain tumor images and accompanying diagnostic reports to construct a dataset;

[0075] S2: Design a joint image and text masking strategy to selectively mask the data;

[0076] S3: Constructing an MRI glioma segmentation model based on CrossMAE;

[0077] S4: Design a multi-task cascade loss function, use the dataset to train and optimize the segmentation model, and obtain a trained MRI brain glioma segmentation model;

[0078] S5: Perform MRI glioma segmentation based on the trained MRI glioma segmentation model.

[0079] Each step is described in detail below.

[0080] Furthermore, as a preferred embodiment of the present method, S1 includes the following steps:

[0081] S1.1: MRI Data Processing: Crop the MRI images and remove irrelevant areas to ensure that all images are of the same size. Normalize the pixel values ​​of each image to a uniform range (0 to 1). Use median filtering to remove noise from the MRI images, preserving edge features and some detail information to ensure the accuracy and usability of the MRI images in subsequent processing.

[0082] S1.2: Diagnostic report processing: Clean the diagnostic report to remove irrelevant information and unify the format. Extract key descriptive information related to glioma.

[0083] S1.3: Dataset partitioning: Divide the processed dataset into training set, validation set, and test set.

[0084] Furthermore, the step of dividing the data set specifically includes:

[0085] The dataset is divided into 80% (training set), 10% (validation set), and 10% (test set).

[0086] Further, S2 includes the following steps:

[0087] S2.1: Align MRI images with corresponding diagnostic reports using patient IDs to ensure that each MRI image is correctly matched to its corresponding diagnostic report. This ensures that images and text information are correctly associated during training, avoiding the impact of information mismatch.

[0088] S2.2: Considering the difference in information density between visual information and language information, random sampling with different mask ratios is used for MRI images and text respectively, so that the model can extract more valuable feature information from images and text.

[0089] Furthermore, the mask operation step specifically includes:

[0090] MRI images were randomly sampled with a mask ratio of 75% to remove redundant information in the images; diagnostic reports were randomly sampled with a mask ratio of 15%.

[0091] Further, refer to Figure 2 , S3 includes the following steps:

[0092] S3.1: The constructed MRI glioma segmentation model uses a cross-modal encoder module, using different encoders to extract image and text features. The image encoder is composed of ViT (Vision Transformer), which extracts features from the MRI T1, T1ce, T2, and Flair modalities. The text encoder is composed of Transformer, which extracts corresponding features from the input text sequence.

[0093] Furthermore, the step of adopting the cross-modal encoder module specifically includes:

[0094] Based on the ViT image encoder, first, the input image I∈R H×W×C Split into multiple image blocks {p1,p2,…,pN}, where H×W is the image spatial resolution and C is the number of image channels. Each image block The one-dimensional feature vector is obtained by flattening the image encoder based on ViT, and the size is P 2 ×C. Next, the unmasked image block is transformed by a linear Converted to block embedding, where D represents the embedding space dimension. This process maps the high-dimensional information of each image block to a low-dimensional embedding space, thereby reducing the dimension of the information and improving computational efficiency. In order to better aggregate visual information, each image block introduces a special learnable tag embedding p I ∈R D , used to represent the global information in the image. The embeddings of all image patches are combined with the learnable one-dimensional position embedding Add. Positional embedding Used to provide position information for the embedding of each small block. This position embedding is learnable during the training process, which helps the network better understand the spatial arrangement of each small block in the image.

[0095] Expressed as:

[0096]

[0097] Finally, the processed input representation X v Will be fed into a v The Transformer model processes these input representations through the self-attention mechanism to obtain the context-related image representation H v . The representation is a sequence of vectors expressed as: in, Represents the image feature vector output by the i-th layer Transformer. In this way, the Transformer can effectively capture local and global information in the image, thereby achieving efficient representation of the image.

[0098] The Transformer-based text encoder first uses the WordPiece algorithm to segment the text sequence into multiple word tags {w1,w2,…,w M}. Split the text sequence into smaller subword units, effectively handle unknown words and improve the vocabulary coverage of the model. All tags w M It is represented in the form of one-hot encoding, where the dimension V represents the size of the vocabulary. Then, a linear transformation matrix E l ∈R V×DLinearly project the one-hot encoded tokens into embedding vectors to map the one-hot encoded tokens to a lower-dimensional embedding space. In order to make the Transformer model understand the beginning and end of the text, two special token embedding vectors are added at the beginning and end of the text sequence, namely the sequence start token (w T ∈R D ), special delimiter marker (w SEP ∈R D ). So, the text input X l The initial representation is:

[0099] X l =[w T ;w1E l ;…;w SEP ]

[0100] Then embed the position Add to the text embedding representation to record the position of each token in the sequence. The embedding matrix contains (M+2) position embedding vectors, corresponding to the position information of each subword token, which can help Transformer capture the order relationship of tokens in the sequence, so the input is finally represented as Y l for:

[0101]

[0102] Finally, the text input represents Y l It will be fed into a Transformer model and processed by multiple Transformer layers to generate a context-related text representation H l The output vector sequence is represented as: in Indicates the start of a column. represents each subword token, Indicates a special delimiter token. In this way, Transformer can effectively process and encode information in text sequences.

[0103] S3.2: Design a collaborative representation module and introduce a collaborative attention mechanism to fuse contextual information from different MRI modalities and text sequences, thereby integrating and compressing the information of each modality while sharing necessary key information;

[0104] Furthermore, the step of adopting the reasoning reconstruction module specifically includes:

[0105] The collaborative representation module consists of two Transformer models, each of which consists of N mEach layer of Transformer consists of three sub-layers: self-attention sub-layer, cross-attention sub-layer, and feed-forward sub-layer. The attention mechanism is applied in the self-attention sub-layer and cross-attention sub-layer, which is defined as follows: ATTN(Q,K,V)=softmax(QK T )·V

[0106] In the self-attention sublayer, the interaction is performed only within the same modality to capture the information within a single modality. For the image modality and text modality, the following calculations are performed respectively: v , self-attention is H vs =ATTN(H v ,H v ,H v ); for text mode H l , self-attention is H ls =ATTN(H l ,H l ,H l );

[0107] The cross-attention sublayer is the core of the collaborative attention mechanism. In this sublayer, the representations of the image and text modalities interact, thereby fusing information from one modality into the representation of the other modality. Specifically, for the image modality H v , attention is H vc =ATTN(H vs ,H ls ,H ls ); for text mode H l , attention is H lc =ATTN(H ls ,H vs ,H vs );

[0108] Representation of image and text modalities H vc and H lc It will be input into the feedforward sublayer (MLP) to obtain the multimodal representation of image and text, which are expressed as:

[0109] S3.3: Design an inference reconstruction module to learn the mapping relationship between different modalities and achieve accurate feature reconstruction in each modality. Design and calculate the joint image and text reconstruction loss to further improve the model's reconstruction capabilities.

[0110] Furthermore, the step of adopting the reasoning reconstruction module specifically includes:

[0111] Select the visual output of the k-th layer Transformer from the collaborative representation module As the image reconstruction input, a 4-layer Transformer block with a hidden dimension of 512 is added at the bottleneck of the 3D U-Net network to achieve accurate cross-modal information mapping.

[0112] S3.4: Use the decoder module to extract the upsampled features of the fused local features of different modalities, and gradually refine and optimize the final segmentation results to ensure high-precision segmentation effects.

[0113] Furthermore, the step of adopting the decoder module specifically includes:

[0114] Each stage first performs an upsampling operation through 3D deconvolution to restore the spatial resolution of the feature map. The convolution kernel size is 3×3×3, the stride is 2, and the ReLU activation function is used. Then, multiple layers of convolution operations (with a kernel size of 3×3×3 and a stride of 1) are combined with batch normalization and ReLU activation to further refine the feature map. Simultaneously, skip connections are used to fuse the low-level features of the encoder with the output of the current decoder stage to enhance local details. Finally, a segmentation map is generated using a 1×1×1 convolution operation. The final segmentation map is output through a sigmoid activation function. The size of the final segmentation map is the same as the input image and is used to mark the lesion area in the image.

[0115] Further, S4 includes the following steps:

[0116] S4.1: The joint image-text reconstruction loss is the weighted sum of the image reconstruction loss (Irecon loss) and the text reconstruction loss (Lrecon loss):

[0117] L recon =αL Irecon +(1-α)L Lrecon

[0118] Where α is a weight coefficient, and back propagation is used during training to minimize the reconstruction task loss and optimize the model reconstruction capability.

[0119] Furthermore, the step of calculating the reconstruction task loss specifically includes:

[0120] The mean square error (MSE) between the reconstructed image and the original image in pixel space is calculated:

[0121]

[0122] Where N is the number of pixels in the image, is the pixel value of the pth row and qth column of the reconstructed image, I pq is the pixel value of the p-th row and q-th column of the original image.

[0123] The text reconstruction loss is obtained by calculating the negative log-likelihood loss on the masked vocabulary tokens:

[0124]

[0125] Where M is the set of masked words, is the same as the masked word w i Related contextual vocabulary. is the masked vocabulary predicted by the model. is the probability that the model predicts the masked word based on the context.

[0126] S4.2: During training, the decoder module calculates the segmentation task loss, which consists of Dice loss and cross entropy loss:

[0127] L seg =Dice(fd(F concat ),T)+CrossEntropy(fd(F concat ),Y)

[0128] Among them, Y represents the true label, Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and cross entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label.

[0129] S4.3 considers both the reconstruction task loss and the subsequent segmentation task loss to optimize the model. Therefore, a multi-task cascade loss is designed, which consists of the weighted sum of the reconstruction loss and the segmentation loss:

[0130] L total =βL recon +(1-β)L seg

[0131] Where β is a hyperparameter used to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascade loss.

[0132] Further, S5 includes the following steps:

[0133] S5.1 performs binarization processing on the multimodal tumor segmentation mask image at the same position to obtain a two-dimensional pixel matrix of the multimodal tumor image;

[0134] Furthermore, the step of calculating the two-dimensional pixel matrix of the multimodal tumor image specifically includes:

[0135] The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model:

[0136]

[0137] Among them, K represents the two-dimensional pixel matrix of the tumor segmentation result image, I pq Represents the pixel value at the pth row and qth column.

[0138] S5.2 superimposes the two-dimensional pixel matrices of tumor images of multiple modalities to obtain a tumor segmentation result image of the target patient.

[0139] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A brain glioma segmentation method based on CrossMAE, characterized in that: The method comprises the following steps: S1: Collect multimodal MR brain tumor images and supporting diagnostic reports to build a data set; S2: Design a joint image-text masking strategy to selectively mask the data; S3: Building a CrossMAE-based MRI glioma segmentation model; S4: Design a multi-task cascade loss function, use the dataset to train and optimize the segmentation model, and obtain a trained MRI brain glioma segmentation model; S5: Perform MRI glioma segmentation based on the trained MRI glioma segmentation model.

2. The method for segmenting glioma based on CrossMAE according to claim 1, characterized in that: Step S1 includes the following steps: S1.1: MRI data processing: Crop the MRI images and remove irrelevant areas to ensure that all images are of the same size. Normalize the pixel values ​​of each image to a uniform range (0 to 1). Use filtering technology to remove noise from the MRI images, maintain the edge features and some detail information of the images, and ensure the accuracy and availability of the MRI images in subsequent processing; S1.2: Diagnosis report processing: Clean the text of the diagnosis report, remove irrelevant information and unify the format. Extract descriptive key information related to brain glioma; S1.3: Dataset division: Divide the processed dataset into training set, validation set and test set.

3. The method for segmenting glioma based on CrossMAE according to claim 1, characterized in that: Step S2 includes the following steps: S2.1: Align the MRI images with the corresponding diagnostic reports by patient ID to ensure that each MRI image is correctly matched with its corresponding diagnostic report, thereby ensuring that the image and text information are correctly associated during the training process to avoid the impact of information mismatch; S2.2: Considering the difference in information density between visual information and language information, random sampling with different mask ratios is used for MRI images and texts respectively, so that the model can extract more valuable feature information from images and texts.

4. The method for segmenting glioma based on CrossMAE according to claim 1, characterized in that: In step S3, the MRI glioma segmentation model based on CrossMAE includes: a cross-modal encoder module designed according to the different characteristics of images and texts, a collaborative representation module that integrates T1, T1ce, T2, Flair modalities in MRI and text feature information, an inference reconstruction module, and a decoder module for MRI glioma segmentation. Step S3 includes: S3.1: The constructed MRI brain glioma segmentation model adopts a cross-modal encoder module, using different encoders to extract image and text features respectively. Among them, the image encoder is composed of ViT (Vision Transformer), and the ViT image encoder is used to extract features from the T1, T1ce, T2, and Flair modalities of MRI. The text encoder is composed of Transformer, which extracts corresponding features by inputting text sequences; S3.2: Design a collaborative representation module and introduce a collaborative attention mechanism to fuse contextual information from different MRI modalities and text sequences, thereby integrating and compressing the information of each modality while sharing necessary key information; S3.3: Design a reasoning reconstruction module to learn the mapping relationship between different modalities and achieve accurate feature reconstruction in each modality. Design and calculate the joint reconstruction loss of image and text to further improve the reconstruction ability of the model; S3.4: Use the decoder module to extract the upsampled features of the fused local features of different modalities, gradually refine and optimize the final segmentation results, and ensure high-precision segmentation effects. S3.1 includes the following steps: Based on the ViT image encoder, first, the input image I∈R H×W×C Split into multiple image blocks {p1,p2,…,p N }, where H×W is the image spatial resolution and C is the number of image channels. Each image block The one-dimensional feature vector is flattened by the ViT-based image encoder, and the size is P 2 ×C. Next, the unmasked image blocks are transformed by a linear Convert to block embedding, where D represents the embedding space dimension. This process maps the high-dimensional information of each image block to a low-dimensional embedding space, thereby reducing the dimensionality of the information and improving computational efficiency. In order to better aggregate visual information, each image block introduces a special learnable tag embedding p I ∈E D , used to represent the global information in the image. The embeddings of all image patches are combined with the learnable one-dimensional position embedding Addition. Positional Embedding It is used to provide position information for the embedding of each small block. This position embedding is learnable during the training process, which helps the network better understand the spatial arrangement of each small block in the image. The input is represented as: Finally, the processed input representation X v will be fed into a v The Transformer model processes these input representations through a self-attention mechanism to obtain context-dependent image representations H v . The representation is a sequence of vectors expressed as: in, Represents the image feature vector output by the i-th layer Transformer. In this way, the Transformer can effectively capture the local and global information in the image, thereby achieving efficient representation of the image. The Transformer-based text encoder first uses the WordPiece algorithm to segment the text sequence into multiple word tags {w1,w2,…,w M }. Split the text sequence into smaller subword units, effectively handle unknown words and improve the vocabulary coverage of the model. All tags w M It is represented in the form of one-hot encoding, where the dimension V represents the size of the vocabulary. Then, a linear transformation matrix E l ∈R V×D Linearly project the one-hot encoded tokens into embedding vectors to map the one-hot encoded tokens to a lower-dimensional embedding space. In order for the Transformer model to understand the beginning and end of the text, two special token embedding vectors are added to the front and back of the text sequence, namely the sequence start token (w T ∈R D ), special separator mark (w SEP ∈R D ). So, the text input X l The initial representation is: X l =[w T ;w1E l ;…;w SEP ] Then embed the location Add to the text embedding representation to record the position of each token in the sequence. The embedding matrix contains (M+2) position embedding vectors, corresponding to the position information of each subword token. This helps the Transformer capture the order relationship of the tokens in the sequence, so the text input is finally represented by Y l for: Finally, the text input represents Y l It will be sent to a Transformer model and processed by multiple Transformer layers to generate context-related text representation H l The output vector sequence is represented as: in Indicates the start of a column. represents each subword token, Represents a special delimiter token. In this way, Transformer can effectively process and encode information in text sequences. S3.2 includes the following steps: The collaborative representation module contains two Transformer models, each of which consists of N m The Transformer layer is composed of three sub-layers: self-attention sub-layer, cross-attention sub-layer, and feed-forward sub-layer. The attention mechanism is applied in the self-attention sub-layer and the cross-attention sub-layer, which is defined as follows: ATTN(Q,K,V)=softmax(QK T )·V In the self-attention sublayer, the representation interacts only within the same modality to capture the information within a single modality. For the image modality and text modality, the following calculations are performed respectively: v , self-attention is H vs =ATTN(H v ,H v ,H v ); for text mode H l , self-attention is H ls =ATTN(H l ,H l ,H l ); The cross-attention sublayer is the core part of the collaborative attention mechanism. In this sublayer, the representations of the image and text modalities interact, so that the information from one modality is fused into the representation of the other modality. Specifically, for the image modality H v , attention is H vc =ATTN(H vs ,H ls ,H ls ); for text mode H l , attention is H lc =ATTN(H ls ,H vs ,H vs ); Representation of image and text modalities H vc and H lc It will be input into the feedforward sublayer (MLP) to obtain the multimodal representation of image and text, which are represented as: S3.3 includes the following steps: The input of the reasoning reconstruction module comes from the collaborative representation module, which uses different levels of visual and text features for reconstruction. Specifically, the visual output of the k-th layer Transformer is selected from the collaborative representation module. As the image reconstruction input. For the mask word prediction of the text, the collaborative representation module is used to finally output Z l , this representation can fully capture cross-modal contextual information and is particularly suitable for extracting semantic information in text, thereby performing accurate missing word prediction. The goal of the inference reconstruction module is to map images and text back to their original inputs. This module contains multiple Transformer layers. S3.4 includes the following steps: The decoder module consists of multiple upsampling and fusion stages. This module effectively fuses the multimodal information of images and texts, and finally achieves accurate MRI brain tumor segmentation.

5. The method for segmenting glioma based on CrossMAE according to claim 1, characterized in that: S4 includes the following steps: S4.1: The joint reconstruction loss of image and text is the weighted sum of the image reconstruction loss (Irecon loss) and the text reconstruction loss (Lrecon loss): L recon =αL Irecon +(1-α)L Lrecon Among them, α is a weight coefficient. Back propagation is used during training to minimize the reconstruction task loss and optimize the model reconstruction ability. S4.2: During training, the decoder module calculates the segmentation task loss, which consists of Dice loss and cross entropy loss: L seg =Dice(fd(F concat ),Y)+CrossEntropy(fd(F concat ),Y) Among them, Y represents the true label, Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and cross entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label. S4.3 considers both the reconstruction task loss and the subsequent segmentation task loss to optimize the model. Therefore, a multi-task cascade loss is designed, which is composed of the weighted sum of the reconstruction loss and the segmentation loss: L total =βL recon +(1-β)L seg Where β is a hyperparameter used to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascade loss. Among them, S4.1 includes the following steps: By calculating the mean square error (MSE) between the reconstructed image and the original image in pixel space, we can get: Where N is the number of all pixels in the image, is the pixel value of the pth row and qth column of the reconstructed image, I pq is the pixel value of the pth row and qth column of the original image. The text reconstruction loss is obtained by calculating the negative log-likelihood loss on the masked vocabulary tokens: Where M is the set of masked words, is the same as the masked word w i Related context words. is the masked vocabulary predicted by the model. is the probability that the model predicts the masked word based on the context.

6. The method for segmenting glioma based on CrossMAE according to claim 1, characterized in that: S5 includes the following steps: S5.1 performs binarization processing on the multimodal tumor segmentation mask image at the same position to obtain a two-dimensional pixel matrix of the multimodal tumor image; S5.2 superimposes the two-dimensional pixel matrices of tumor images of multiple modalities to obtain a tumor segmentation result image of the target patient. Among them, S5.1 includes the following steps: The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model: Where K represents the two-dimensional pixel matrix of the tumor segmentation result image, I pq Represents the pixel value of the pth row and qth column.

Images