A crossmae-based brain glioma segmentation method

Abstract

The application discloses a brain glioma segmentation method based on CrossMAE, aiming at extracting visual language representation by combining MRI and diagnostic report to enhance segmentation performance. The method learns cross-modal deep correlation by randomly masking images and text information and reconstructing the masked content, thereby realizing accurate segmentation of MRI brain glioma. The method comprises the following steps: S1: collecting multi-modal MR brain tumor images and diagnostic reports, and constructing a data set; S2: designing a graphic-text mask encoder to selectively mask the data; S3: constructing a CrossMAE-based MRI brain glioma segmentation model; S4: designing a multi-task cascaded loss function, training and optimizing the segmentation model using the data set, and obtaining the trained MRI brain glioma segmentation model; S5: performing MRI brain glioma segmentation according to the trained MRI brain glioma segmentation model. The application effectively extracts and utilizes cross-modal complementary information from MRI, thereby realizing accurate segmentation of MRI brain glioma.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a brain glioma segmentation method based on CrossMAE. BACKGROUND

[0002] With the rapid development of medical imaging technology, the application potential of cross-modal pre-training methods based on medical images and language in medical image understanding is attracting more and more attention. Medical data itself has multi-modal characteristics, among which vision (such as radiographic images, magnetic resonance imaging and computed tomography) and language (such as radiology reports, medical texts and pathology reports) are two main forms of expression. In the medical field, visual and language pre-training is considered a key technology that helps achieve deep joint understanding of medical images and text information. However, existing technologies still face many challenges in practical applications, such as the lack of large-scale labeled data, the high specialization of domain knowledge, and the complexity of efficient fusion of cross-modal information in the medical field. These problems seriously limit the widespread application of cross-modal pre-training methods in medical image segmentation, auxiliary diagnosis and other tasks.

[0003] In the medical image segmentation task, especially in the MRI image segmentation of brain glioma and other tumors, traditional methods based on convolutional neural networks (CNN) are often limited by the complexity of the model and the improvement space of the precision. Existing researches mainly focus on single training and processing of images (such as CT images and MRI images), and few in-depth explorations are made on joint modeling of images and texts.

[0004] However, images and texts in the medical field are often complementary, images provide intuitive lesion region information, while texts contain detailed descriptions of lesions and clinical background knowledge.

[0005] Therefore, how to effectively combine image and text information to improve the precision of brain glioma image segmentation has become a key problem to be solved in this field. Cross-modal learning, especially joint modeling based on visual and language information, provides a new idea and solution for improving the precision of medical image segmentation. SUMMARY

[0006] To solve the above technical problems, the present application provides a brain glioma segmentation method based on CrossMAE, which provides a method for realizing precise MRI brain glioma segmentation by combining image and text feature information for users.

[0007] Specifically, the method comprises the following steps:

[0008] S1: Collecting multi-modal MR brain tumor images and supporting diagnostic reports to construct a data set;

[0009] S2: design a graphic-text combined mask strategy to selectively mask the data;

[0010] S3: build a CrossMAE-based MRI brain glioma segmentation model;

[0011] S4: design a multi-task cascaded loss function, train and optimize the segmentation model using the dataset, and obtain the trained MRI brain glioma segmentation model;

[0012] S5: perform MRI brain glioma segmentation according to the trained MRI brain glioma segmentation model.

[0013] Preferably, S1 includes the following steps:

[0014] S1.1: MRI data processing: crop the MRI image to remove irrelevant areas to ensure that all image sizes are consistent. Normalize the pixel values of each image to a uniform range (0 to 1). Use filtering techniques to remove noise from the MRI image, preserve edge features and some detail information, and ensure the accuracy and availability of the MRI image in subsequent processing;

[0015] S1.2: Diagnosis report processing: clean the text of the diagnosis report, remove irrelevant information and unify the format. Extract descriptive key information related to brain glioma;

[0016] S1.3: Data set division: divide the processed data set into training set, validation set and test set.

[0017] Preferably, S2 includes the following steps:

[0018] S2.1: Align the MRI image and the corresponding diagnosis report by patient ID to ensure that each MRI image is correctly matched with its corresponding diagnosis report, so that the image and text information are correctly associated during training, avoiding the impact of information mismatch;

[0019] S2.2: Considering the difference in information density between visual information and language information, randomly sample different mask ratios for MRI images and text respectively, so that the model can extract more valuable feature information from images and text.

[0020] Preferably, in S3, the CrossMAE-based MRI brain glioma segmentation model includes a cross-modal encoder module designed according to the different characteristics of images and text, a collaborative representation module that fuses T1, T1ce, T2, Flair modalities in MRI and text feature information, an inference reconstruction module, and a decoder module for MRI brain glioma segmentation. It includes the following steps:

[0021] S3.1: The constructed MRI brain glioma segmentation model adopts a cross-modal encoder module, which uses different encoders to extract image and text features respectively. Among them, the image encoder is composed of ViT (Vision Transformer), and the features of MRI T1, T1ce, T2 and Flair modalities are extracted through the ViT image encoder. The text encoder is composed of Transformer, and the corresponding features are extracted by inputting the text sequence;

[0022] S3.2: Design a collaborative representation module, introduce a collaborative attention mechanism to fuse the context information from different modalities of MRI and text sequences, so as to integrate and compress the information of each modality while sharing the necessary key information;

[0023] S3.3: Design a reasoning reconstruction module to learn the mapping relationship between different modalities and realize accurate feature reconstruction in each modality. Design and calculate the joint reconstruction loss of image and text to further improve the reconstruction ability of the model;

[0024] S3.4: Use the decoder module to extract the up-sampling features of the fused local features of different modalities, gradually refine and optimize the final segmentation result, and ensure high-precision segmentation effect.

[0025] S3.1 includes the following steps:

[0026] The ViT-based image encoder first flattens the input image into multiple image blocks , where is the image spatial resolution, is the number of image channels. Each image block is flattened into a one-dimensional feature vector with size by the ViT-based image encoder. Then, the image blocks not masked are converted into block embeddings by a linear transformation , where represents the embedding space dimension. This process maps the high-dimensional information of each image block to a low-dimensional embedding space, reducing the dimension of the information and improving the computational efficiency. In order to better aggregate visual information, a special learnable token embedding is introduced for each image block to represent the global information in the image. The embeddings of all image blocks are added to the learnable one-dimensional position embedding . The position embedding is used to provide position information for each small block's embedding, and this position embedding is learnable during training, which helps the network better understand the spatial arrangement of each small block in the image. The input is represented as:

[0027]

[0028] The processed input representation will be fed into a Transformer model with Transformer layers. The Transformer model processes these input representations through self-attention mechanisms to obtain contextually relevant image representations . This representation is a sequence of vectors, denoted as , where represents the image feature vector output by the th Transformer layer. In this way, the Transformer can effectively capture both local and global information in the image, enabling efficient representation of the image.

[0029] The Transformer-based text encoder first tokenizes the text sequence into multiple word tokens using the WordPiece algorithm . The text sequence is split into smaller subword units, effectively handling unknown words and improving the model's vocabulary coverage. All tokens are represented in the form of one-hot encoding, with dimensions representing the size of the vocabulary. Next, a linear transformation matrix is used to project the one-hot encoded tokens linearly into embedding vectors. The one-hot encoded tokens are mapped to a lower-dimensional embedding space. To let the Transformer model understand the beginning and end of the text, two special tokens are added at the beginning and end of the text sequence, respectively, embedding vectors, the sequence start token ( ), and the special separator token ( ). Therefore, the initial representation of the text input is:

[0030]

[0031] Next, position embeddings are added to the text's embedding representation, recording the position of each token in the sequence. The embedding matrix contains position embedding vectors, corresponding to the position information of each subword token, which helps the Transformer capture the order relationship of tokens in the sequence. Therefore, the final representation of the input is:

[0032]

[0033] Finally, the text input representation It will be fed into a Transformer model and processed by multiple Transformer layers to generate context-related text representations. The output vector sequence is represented as: in Indicates the start of a column. represents each subword token, Indicates a special delimiter token. In this way, Transformer can effectively process and encode information in text sequences.

[0034] S3.2 includes the following steps:

[0035] The collaborative representation module consists of two Transformer models, each of which consists of The Transformer layer is composed of three sub-layers: self-attention sub-layer, cross-attention sub-layer, and feed-forward sub-layer. The attention mechanism is applied in the self-attention sub-layer and cross-attention sub-layer, which is defined as follows:

[0036] In the self-attention sublayer, the representation is only interacted within the same modality to capture the information within a single modality. For the image modality and text modality, the following calculations are performed respectively: , self-attention is ; For text mode , self-attention is ;

[0037] The cross-attention sublayer is the core of the collaborative attention mechanism. In this sublayer, the representations of the image and text modalities interact, thereby fusing information from one modality into the representation of the other modality. Specifically, for the image modality , attention is For text modal , attention is ;

[0038] Representation of image and text modalities and It will be input into the feedforward sublayer (MLP) to obtain multimodal representations of images and text, which are expressed as: , ;

[0039] S3.3 includes the following steps:

[0040] The input of the reasoning reconstruction module comes from the collaborative representation module, which uses different levels of visual and text features for reconstruction. Specifically, the first - the visual output of the layer Transformer as the image reconstruction input. For the masked word prediction of the text, the collaborative representation module outputs which can sufficiently capture the cross-modal context information and is suitable for extracting semantic information in the text, thus making accurate missing word prediction.

[0041] The goal of the inference reconstruction module is to map the image and the text back to their original inputs. The module contains multiple Transformer layers.

[0042] wherein S3.4 comprises the following steps:

[0043] The decoder module is composed of multiple upsampling and fusion stages. This module effectively fuses the multi-modal information of the image and the text, and finally achieves accurate MRI brain tumor segmentation.

[0044] Preferably, S4 comprises the following steps:

[0045] S4.1: The image-text joint reconstruction loss is the weighted sum of the image reconstruction loss (Irecon loss) and the text reconstruction loss (Lrecon loss):

[0046]

[0047] wherein is a weight coefficient, which is used to minimize the reconstruction task loss by backpropagation during training, and optimize the reconstruction ability of the model.

[0048] S4.2: The segmentation task loss is calculated by the decoder module during training, and the segmentation task loss is composed of the Dice loss and the cross-entropy loss:

[0049]

[0050] wherein, represents the true label, the Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and the cross-entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label.

[0051] S4.3: The model is optimized by considering both the reconstruction task loss and the subsequent segmentation task loss, so a multi-task cascaded loss is designed, which is composed of the weighted sum of the reconstruction loss and the segmentation loss:

[0052]

[0053] wherein is a hyperparameter used to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascaded loss.

[0054] wherein S4.1 comprises the following steps:

[0055] The reconstruction image is obtained by calculating the mean square error (MSE) of the reconstruction image and the original image in the pixel space:

[0056]

[0057] wherein is the number of all pixels in the image, is the pixel value of the reconstruction image in the i-th row and the j-th column, is the pixel value of the original image in the i-th row and the j-th column. The text reconstruction loss is obtained by calculating the negative log-likelihood loss of the masked word token:

[0058] wherein M is the set of masked words,

[0059] is the context word related to the masked word. is the model predicted masked word. is the probability of the model predicting the masked word according to the context.

[0060] Preferably, S5 comprises the following steps: S5.1 binarizes the multi-modal tumor segmentation mask image at the same position to obtain a two-dimensional pixel matrix of the multi-modal tumor image; S5.2 superimposes the two-dimensional pixel matrices of the tumor images of multiple modalities to obtain a tumor segmentation result image of the target patient.

[0061] wherein S5.1 comprises the following steps:

[0062] The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model:

[0063]

[0064]

[0065] wherein

[0066] represents the two-dimensional pixel matrix of the tumor segmentation result image, represents the pixel value in the i-th row and the j-th column.

[0067]

[0068] ​​​​​​​The method has the advantages that the method firstly sets different masking ratios for the MRI image and the diagnosis report, extracts feature information of the image and the text through a cross-modal encoder module respectively, and uses an image encoder and a text encoder to extract and embed the key information in the MRI image and the diagnosis report respectively. Then, the collaborative representation module introduces a collaborative attention mechanism to efficiently fuse the image and text features, integrates the context information from different modalities, so that the model can share and utilize valuable cross-modal information, further enriching the joint representation of the image and the text. Through the inference reconstruction module, the model reconstructs itself in different levels of feature information to optimize information expression and improve the mutual dependence between the image and the text, ensuring more accurate cross-modal understanding. Finally, the decoder module performs layer-by-layer upsampling and convolution operations on the fused multi-modal features to refine the image features, and finally realizes accurate MRI glioma segmentation. At the same time, the method effectively solves the challenges of existing neural network models in multi-modal data processing and tumor segmentation. On the one hand, through the cross-modal feature extraction module, the model can enrich the understanding of image and text information; on the other hand, through the collaborative attention mechanism, the model's attention to image and text features is enhanced, and the fusion efficiency of multi-modal features is optimized; at the same time, through the MRI segmentation model based on CrossMAE, the accuracy and visualization effect of MRI brain tumor segmentation are finally improved. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings needed to be used in the prior art and embodiments, and the following drawings are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0070] Fig. 1 is a flowchart of a brain glioma segmentation method based on CrossMAE according to the present application;

[0071] Fig. 2 is a model overall architecture diagram of a brain glioma segmentation method based on CrossMAE according to the present application. DETAILED DESCRIPTION

[0072] In order to make the inventive purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all other embodiments obtained by those skilled in the art without creative labor based on the embodiments in the present application are within the scope of protection of the present application.

[0073] The embodiment of the application provides a brain glioma segmentation method based on CrossMAE for MRI and diagnostic report combination, which effectively extracts and utilizes multi-modal complementary information from MR images, and realizes accurate segmentation of brain glioma.

[0074] In one typical embodiment of the application, with reference to Figure 1 The method comprises the following steps:

[0075] S1: Collect multi-modal MR brain tumor images and matching diagnostic reports, and construct a data set;

[0076] S2: Design a graphic-text joint mask strategy to selectively mask the data;

[0077] S3: Construct an MRI brain glioma segmentation model based on CrossMAE;

[0078] S4: Design a multi-task cascade loss function, train and optimize the segmentation model using the data set, and obtain a trained MRI brain glioma segmentation model;

[0079] S5: Perform MRI brain glioma segmentation according to the trained MRI brain glioma segmentation model.

[0080] Each step is described in detail as follows.

[0081] Further, as a preferred embodiment of the method, S1 comprises the following steps:

[0082] S1.1: MRI data processing: crop the MRI image to remove irrelevant areas to ensure that all image sizes are consistent. Normalize the pixel values of each image to a uniform range (0 to 1). Use median filtering technology to remove noise from the MRI image, preserve the edge features and some detail information of the image, and ensure the accuracy and availability of the MRI image in subsequent processing;

[0083] S1.2: Diagnostic report processing: clean the text of the diagnostic report, remove irrelevant information and unify the format. Extract descriptive key information related to brain glioma;

[0084] S1.3: Data set division: divide the processed data set into a training set, a validation set and a test set.

[0085] Further, the data set division step specifically comprises:

[0086] According to the proportions of 80% (training set), 10% (validation set) and 10% (test set).

[0087] Further, S2 comprises the following steps:

[0088] S2.1: Align the MRI images with the corresponding diagnostic reports by patient ID, ensuring that each MRI image is correctly matched with its corresponding diagnostic report, thereby ensuring that the image and text information are correctly associated during the training process, avoiding the impact of information mismatch;

[0089] S2.2: Considering the difference in information density between visual information and language information, different mask ratios are used for random sampling of MRI images and text, so that the model can extract more valuable feature information from images and text.

[0090] Further, the mask operation step specifically includes:

[0091] Random sampling of MRI images with a mask ratio of 75% aims to remove redundant information in the image; random sampling of diagnostic reports with a mask ratio of 15%.

[0092] Further, referring to Figure 2 , S3 includes the following steps:

[0093] S3.1: The constructed MRI brain glioma segmentation model uses a cross-modal encoder module, which uses different encoders to extract image and text features. Among them, the image encoder is composed of ViT (Vision Transformer), which extracts features of MRI T1, T1ce, T2, Flair modalities through ViT image encoder. The text encoder is composed of Transformer, which extracts corresponding features by inputting text sequences;

[0094] Further, the step of using a cross-modal encoder module specifically includes:

[0095] The image encoder based on ViT first divides the input image into multiple image blocks , where is the image spatial resolution, is the number of image channels. Each image block is flattened to obtain a one-dimensional feature vector with a size of by the image encoder based on ViT. Then, the image blocks that are not masked are converted to block embeddings by a linear transformation , where represents the embedding space dimension. This process maps the high-dimensional information of each image block to a low-dimensional embedding space, thereby reducing the dimension of information and improving computational efficiency. In order to better aggregate visual information, a special learnable token embedding is introduced for each image block, which represents the global information in the image. The embeddings of all image blocks are combined with a learnable one-dimensional position embedding Additive. Positional Embeddings To provide position information for each patch's embedding, this positional embedding is learnable during training, which helps the network better understand the spatial arrangement of individual patches in the image. Input

[0096] is represented as:

[0097]

[0098] The processed input representation will be fed into a Transformer model with Transformer layers. The Transformer model processes these input representations through self-attention mechanisms to obtain context-dependent image representations . This representation is a sequence of vectors, represented as: where is the image feature vector output by the th Transformer layer. In this way, the Transformer can effectively capture both local and global information in the image, enabling efficient representation of the image.

[0099] The Transformer-based text encoder first uses the WordPiece algorithm to tokenize the text sequence into multiple word tokens . The text sequence is split into smaller subword units, effectively handling unknown words and improving the model's vocabulary coverage. All tokens are represented in the form of one-hot encoding, with dimensions representing the size of the vocabulary. Then, a linear transformation matrix is used to linearly project the one-hot encoded tokens into embedding vectors, mapping one-hot encoded tokens to a lower-dimensional embedding space. To let the Transformer model understand the beginning and end of the text, two special token embeddings are added at the beginning and end of the text sequence, respectively, the sequence start token ( ), and the special separator token ( ). Therefore, the initial representation of the text input is:

[0100]

[0101] Then, positional embeddings are added to the embedding representation of the text, recording the position of each token in the sequence. The embedding matrix contains position embedding vectors, one for each subword token, which helps the Transformer capture the order of tokens in the sequence, so the final input representation is:

[0102]

[0103] The final text input representation is fed into a Transformer model, which processes it through multiple Transformer layers to generate a context-aware text representation . The output vector sequence is represented as: where denotes the start-of-sequence token, denotes each subword token, and denotes the special separator token. In this way, the Transformer can effectively process and encode the information in the text sequence.

[0104] S3.2: Design a collaborative representation module that introduces a collaborative attention mechanism to integrate context information from different modalities of MRI and the text sequence, thereby integrating and compressing the information of each modality while sharing necessary key information;

[0105] Further, the step of adopting the reasoning reconstruction module specifically includes:

[0106] The collaborative representation module includes two Transformer models, each of which is stacked by layers of Transformer. Each layer of Transformer is composed of three sub-layers: a self-attention sub-layer, a cross-attention sub-layer, and a feed-forward sub-layer. In the self-attention sub-layer and the cross-attention sub-layer, the attention mechanism is applied, which is defined as follows:

[0107] In the self-attention sub-layer, the representation only interacts within the same modality to capture the information within a single modality. For the image modality and the text modality, the following calculations are performed respectively: for the image modality , the self-attention is ; for the text modality , the self-attention is ;

[0108] The cross-attention sub-layer is the core part of the collaborative attention mechanism. In this sub-layer, the representations of the image and text modalities interact, thereby fusing the information from one modality into the representation of the other modality. Specifically, for the image modality , the attention is for the text modality , attention is ;

[0109] representation of image and text modalities and will be input to the feedforward sublayer (MLP) to obtain the multimodal representation of image and text, denoted as: , ;

[0110] S3.3: The reasoning reconstruction module is designed to learn the mapping relationship between different modalities and to realize accurate feature reconstruction in each modality. A joint image-text reconstruction loss is designed and calculated to further improve the reconstruction ability of the model;

[0111] Further, the step of adopting the reasoning reconstruction module specifically comprises:

[0112] The first layer of the visual output of the Transformer as the image reconstruction input. At the bottleneck of the 3D U-Net network, a 4-layer Transformer block with a hidden dimension of 512 is added to realize accurate cross-modal information mapping.

[0113] S3.4: Use the decoder module to extract the up-sampled features of the fused local features of different modalities, gradually refine and optimize the final segmentation result, and ensure high-precision segmentation effect.

[0114] Further, the step of adopting the decoder module specifically comprises:

[0115] Each stage first performs an up-sampling operation through 3D deconvolution to restore the spatial resolution of the feature map, with a convolution kernel size of 3x3x3, a stride of 2, and a ReLU activation function. Then, after multiple convolution operations (convolution kernel size of 3x3x3, stride of 1), combined with batch normalization and ReLU activation function, the feature map is further refined. At the same time, the low-level features of the encoder are fused with the output of the current stage of the decoder using a skip connection to enhance local detail information. Finally, a 1x1x1 convolution operation is used to generate a segmentation map, and a Sigmoid activation function is used to output the final segmentation map, which has the same size as the input image and is used to mark the lesion area in the image.

[0116] Further, S4 comprises the following steps:

[0117] S4.1: The joint image-text reconstruction loss is the weighted sum of the image reconstruction loss (Irecon loss) and the text reconstruction loss (Lrecon loss):

[0118]

[0119] wherein is a weight coefficient, which is trained by backpropagation to minimize the reconstruction task loss and optimize the reconstruction capability of the model.

[0120] Further, the step of calculating the reconstruction task loss specifically comprises:

[0121] The reconstruction image and the original image in the pixel space are calculated by the mean square error (MSE) as follows:

[0122]

[0123] wherein is the number of all pixels in the image, is the pixel value of the th row and the th column of the reconstructed image, is the pixel value of the th row and the th column of the original image.

[0124] The text reconstruction loss is calculated by the negative log-likelihood loss of the masked vocabulary as follows:

[0125]

[0126] wherein, M is the set of masked vocabulary, is the context vocabulary related to the masked vocabulary. is the model-predicted masked vocabulary. is the probability of the model predicting the masked vocabulary according to the context.

[0127] S4.2: The segmentation task loss is calculated by the decoder module during training, and the segmentation task loss is composed of the Dice loss and the cross-entropy loss:

[0128]

[0129] wherein, indicates the true label, the Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and the cross-entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label.

[0130] S4.3: The model is optimized by considering the reconstruction task loss and the subsequent segmentation task loss, and therefore a multi-task cascaded loss is designed, which is composed of the weighted sum of the reconstruction loss and the segmentation loss:

[0131]

[0132] wherein ​is a hyper parameter to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascaded loss.

[0133] Further, S5 comprises the following steps:

[0134] S5.1 binarizing the multi-modal tumor segmentation mask images at the same position to obtain a two-dimensional pixel matrix of the multi-modal tumor image;

[0135] Further, the step of calculating the two-dimensional pixel matrix of the multi-modal tumor image specifically comprises:

[0136] The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model:

[0137]

[0138] wherein, represents the two-dimensional pixel matrix of the tumor segmentation result image, represents the pixel value in the i-th row and the j-th column.

[0139] S5.2 superimposing the two-dimensional pixel matrices of the tumor images of the plurality of modalities to obtain a tumor segmentation result image of the target patient.

[0140] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.​​

Claims

1. A glioma segmentation method based on CrossMAE, characterized in that: The method comprises the following steps: S1: Acquire multimodal MR brain tumor images and accompanying diagnostic reports to construct a dataset; S2: Design a joint image and text masking strategy to selectively mask the data; S3: Constructing an MRI glioma segmentation model based on CrossMAE; S3 includes the following steps: S3.1: The constructed MRI glioma segmentation model uses a cross-modal encoder module, using different encoders to extract image and text features respectively. The image encoder is composed of ViT (Vision Transformer), which extracts features from the T1, T1ce, T2, and Flair modalities of MRI. The text encoder is composed of Transformer, which extracts corresponding features from the input text sequence. S3.2: Design a collaborative representation module and introduce a collaborative attention mechanism to fuse contextual information from different MRI modalities and text sequences, thereby integrating and compressing the information of each modality while sharing necessary key information; S3.3: Design an inference reconstruction module to learn the mapping relationship between different modalities, achieve accurate feature reconstruction in each modality, and design and calculate the joint image and text reconstruction loss to further improve the model's reconstruction capabilities. S4: Design a multi-task cascade loss function, use the dataset to train and optimize the segmentation model, and obtain a trained MRI brain glioma segmentation model; Among them, S4 includes the following steps: S4.1: Calculate the joint reconstruction loss of image and text: S4.2: Calculate the segmentation task loss; S4.3: Design a multi-task cascade loss that considers both the reconstruction task loss and the subsequent segmentation task loss to optimize the model. It consists of a weighted sum of the reconstruction loss and the segmentation loss. S5: Perform MRI glioma segmentation based on the trained MRI glioma segmentation model.

2. The method for brain glioma segmentation based on CrossMAE according to claim 1, characterized in that: Step S1 includes the following steps: S1.1: MRI data processing: Crop MRI images and remove irrelevant areas to ensure that all images are of the same size; normalize the pixel values ​​of each image to a value between 0 and 1; use filtering techniques to remove noise from MRI images, preserve image edge features and some details, and ensure the accuracy and usability of MRI images in subsequent processing; S1.2: Diagnostic report processing: Clean the diagnostic report to remove irrelevant information, unify the format, and extract key descriptive information related to brain glioma; S1.3: Dataset partitioning: Divide the processed dataset into training set, validation set, and test set.

3. The method for brain glioma segmentation based on CrossMAE according to claim 1, characterized in that: Step S2 includes the following steps: S2.1: Align MRI images with corresponding diagnostic reports using patient IDs to ensure that each MRI image is correctly matched to its corresponding diagnostic report. This ensures that images and text information are correctly associated during training, avoiding the impact of information mismatch. S2.2: Considering the difference in information density between visual information and language information, random sampling with different mask ratios is used for MRI images and text respectively, so that the model can extract more valuable feature information from images and text.

4. The method for brain glioma segmentation based on CrossMAE according to claim 1, characterized in that: In step S3, the CrossMAE-based MRI glioma segmentation model includes: a cross-modal encoder module designed based on the different characteristics of images and text, a collaborative representation module that integrates T1, T1ce, T2, and Flair modalities in MRI and text feature information, an inference and reconstruction module, and a decoder module for MRI glioma segmentation; Step S3 includes: S3.1: The constructed MRI glioma segmentation model uses a cross-modal encoder module, using different encoders to extract image and text features respectively. The image encoder is composed of ViT (Vision Transformer), which extracts features from the T1, T1ce, T2, and Flair modalities of MRI. The text encoder is composed of Transformer, which extracts corresponding features from the input text sequence. S3.2: Design a collaborative representation module and introduce a collaborative attention mechanism to fuse contextual information from different MRI modalities and text sequences, thereby integrating and compressing the information of each modality while sharing necessary key information; S3.3: Design an inference reconstruction module to learn the mapping relationship between different modalities, achieve accurate feature reconstruction in each modality, and design and calculate the joint image and text reconstruction loss to further improve the model's reconstruction capabilities. S3.4: Use the decoder module to extract the upsampled features of the fused local features of different modalities, and gradually refine and optimize the final segmentation results to ensure high-precision segmentation results; S3.1 includes the following steps: The image encoder based on ViT first converts the input image Split into multiple image blocks ,in is the image spatial resolution, is the number of image channels, each image block , flattened by the ViT-based image encoder to obtain a one-dimensional feature vector of size ; Next, the unmasked image blocks are transformed by a linear Convert to block embedding, where Represents the embedding space dimension; this process maps the high-dimensional information of each image block into a low-dimensional embedding space, thereby reducing the dimension of information and improving computational efficiency; in order to better aggregate visual information, each image block introduces a special learnable tag embedding , used to represent the global information in the image; the embedding of all image patches is combined with the learnable one-dimensional position embedding Addition; Positional Embedding It is used to provide position information for the embedding of each small block. This position embedding is learnable during the training process, which helps the network better understand the spatial arrangement of each small block in the image. The input is represented as: Finally, the processed input representation will be sent to a In the Transformer model with 10 layers of Transformer, the Transformer model processes these input representations through the self-attention mechanism to obtain context-related image representations ; The representation is a vector sequence expressed as: ,in, Indicates the The image feature vector output by the layer Transformer; in this way, the Transformer can effectively capture local and global information in the image, thereby achieving efficient representation of the image; The Transformer-based text encoder first uses the WordPiece algorithm to segment the text sequence into multiple word tokens , splits the text sequence into smaller subword units, effectively handles unknown words and improves the vocabulary coverage of the model; all tags Represented in the form of one-hot encoding, its dimension is Represents the size of the vocabulary; then, through a linear transformation matrix Linearly project the one-hot encoded tokens into embedding vectors to map the one-hot encoded tokens to a lower-dimensional embedding space; in order for the Transformer model to understand the beginning and end of the text, two special token embedding vectors are added before and after the text sequence, namely the sequence start token , special delimiter marker , text input The initial representation is: Then embed the position Added to the embedded representation of the text, recording the position of each token in the sequence; the embedding matrix contains Position embedding vectors, corresponding to the position information of each subword token, can help Transformer capture the order relationship of tokens in the sequence, so the text input is finally represented for: Finally, the text input representation It will be fed into a Transformer model and processed by multiple Transformer layers to generate context-related text representations. ; The output vector sequence is represented as: ,in Indicates the start of a column. represents each subword token, Represents a special delimiter token; in this way, the Transformer can effectively process and encode information in text sequences; S3.2 includes the following steps: The collaborative representation module consists of two Transformer models, each of which consists of The Transformer layer is stacked, and each Transformer layer consists of three sub-layers: self-attention sub-layer, cross-attention sub-layer, and feed-forward sub-layer. The attention mechanism is applied in the self-attention sub-layer and cross-attention sub-layer, which is defined as follows: In the self-attention sublayer, it is indicated that the interaction is only within the same modality to capture the information within a single modality; for the image modality and text modality, the following calculations are performed respectively: , self-attention is ; For text mode , self-attention is ; The cross-attention sublayer is the core part of the collaborative attention mechanism; in this sublayer, the representations of the image and text modalities interact to fuse information from one modality into the representation of the other modality; specifically, the image modality , attention is For text modal , attention is ; Representation of image and text modalities and It will be input into the feedforward sublayer (MLP) to obtain multimodal representations of images and text, which are expressed as: , ; S3.3 includes the following steps: The input of the reasoning reconstruction module comes from the collaborative representation module, which uses different levels of visual and text features for reconstruction; specifically, the first The visual output of the layer Transformer As the input for image reconstruction; for text mask word prediction, the collaborative representation module is used to output the final , this representation can fully capture cross-modal contextual information and is suitable for extracting semantic information from text, thereby making accurate missing word predictions; the reasoning and reconstruction module contains multiple Transformer layers, whose goal is to map images and text back to their original inputs; S3.4 includes the following steps: The decoder module consists of multiple upsampling and fusion stages, which effectively fuses the multimodal information of images and text, ultimately achieving accurate MRI brain tumor segmentation.

5. The method for brain glioma segmentation based on CrossMAE according to claim 1, characterized in that: S4 includes the following steps: S4.1: The joint image-text reconstruction loss is the image reconstruction loss and text reconstruction loss The weighted sum of: in is a weight coefficient, which is used to minimize the reconstruction task loss and optimize the model reconstruction ability through back propagation during training; S4.2: During training, the decoder module calculates the segmentation task loss, which consists of Dice loss and cross entropy loss: in, represents the true label, Dice loss is used to measure the overlap between the predicted segmentation result and the true label, and cross entropy loss is used to calculate the pixel-level difference between the predicted segmentation map and the true label; S4.3 Considering both the reconstruction task loss and the subsequent segmentation task loss to optimize the model, a multi-task cascade loss is designed, which is composed of the weighted sum of the reconstruction loss and the segmentation loss: in It is a hyperparameter used to adjust the weight of the reconstruction task loss and the subsequent segmentation task loss in the multi-task cascade loss; Among them, S4.1 includes the following steps: The mean square error (MSE) between the reconstructed image and the original image in pixel space is calculated: in is the number of all pixels in the image, The reconstructed image Rank The pixel value of the column, The original image Rank The pixel value of the column; The text reconstruction loss is obtained by calculating the negative log-likelihood loss on the masked vocabulary tokens: Where M is the set of masked words, is the same as the masked word Related context words, is the masked vocabulary predicted by the model, is the probability that the model predicts the masked word based on the context.

6. The method for brain glioma segmentation based on CrossMAE according to claim 1, characterized in that: S5 includes the following steps: S5.1 performs binarization processing on the multimodal tumor segmentation mask image at the same position to obtain a two-dimensional pixel matrix of the multimodal tumor image; S5.2 superimposing the two-dimensional pixel matrices of tumor images from multiple modalities to obtain a tumor segmentation result image of the target patient; S5.1 includes the following steps: The two-dimensional pixel matrix of the brain glioma segmentation result image satisfies the following model: in, Represents the two-dimensional pixel matrix of the tumor segmentation result image, Indicates the Rank The pixel value of the column.

Images