A Brain Tumor MRI Image Segmentation Method Based on Mask Attention Mechanism

Abstract

The present invention discloses a brain tumor MRI image segmentation method based on a masked self-attention mechanism. The segmentation method includes: S1: constructing a 3D Masked Swin Transformer module; S2: constructing a three-dimensional medical image segmentation model; S3: obtaining an original medical image dataset, performing data preprocessing on the original medical image dataset, and dividing the preprocessed medical image dataset into a training set and a test set; S4: setting hyperparameters for network model training and an online data augmentation method, and using the training set to train the three-dimensional medical image segmentation model; S5: using the trained three-dimensional medical image segmentation model to segment the tumor region in the original brain MRI medical image. The advantages of the present invention are as follows: This method can effectively apply Transformer to medical image segmentation, has strong generalization ability, high anti-interference ability, and high segmentation accuracy.

Description

[0001] The present invention relates to the field of medical image segmentation, and particularly to a method for segmenting brain tumor MRI images based on a masked attention mechanism. Background Art

[0002] Medical image segmentation is an important part of computer-aided diagnosis, monitoring, intervention, and treatment. Its main task is to segment the objects of interest (such as organs or lesions) in medical images, which can provide important references for aspects such as accurate disease identification, detailed analysis, and accurate diagnosis. With the rapid development of medical imaging equipment and technology, a large amount of medical image data is generated globally every day. However, limited by the imbalance between medical supply and demand, that is, the contradiction between the rapid growth of the number of medical images and the shortage of image analysts, the image analysis speed lags far behind the image generation speed. Therefore, designing and developing an automatic medical image segmentation algorithm with high accuracy, fast speed, and good generalization ability is an urgent task in the current field of medical image analysis.

[0003] Early traditional image segmentation algorithms include, but are not limited to, edge detection-based segmentation algorithms, threshold-based segmentation algorithms, and region-based segmentation algorithms. However, medical images have characteristics such as low contrast, complex texture, and blurred boundary regions, which greatly limit the effects and application scenarios of such image segmentation algorithms. Subsequently, segmentation algorithms that design handcrafted features for specific tasks became the mainstream of medical image segmentation research for a long time. However, the design of handcrafted features highly depends on doctors' professional prior knowledge and often has poor generalization ability. Therefore, in practical applications, medical image segmentation systems based on traditional image segmentation algorithms are still not mature enough to achieve satisfactory segmentation results.

[0004] After 2000, with the development of computer hardware, deep learning has gradually become the mainstream in the field of medical image segmentation due to its powerful modeling ability. So far, image segmentation methods based on deep learning can be divided into semantic segmentation methods based on convolutional neural networks and those based on Transformers. For example, the patent document named "A Medical Image Segmentation Method Based on an Improved U-Net Neural Network with Residual Connections" (application number: CN2021107251053) discloses a medical image segmentation method based on an improved U-Net convolutional neural network with residual connections. This method uses a residual network to improve the U-Net neural network to increase the fusion of lower-layer features and uses the ResNeXt algorithm to extract deep features from the image. This method can obtain relatively accurate prediction results for new categories based on a small number of samples. However, this method performs segmentation based on two-dimensional slice images, ignoring the spatial information of three-dimensional images. Another example is the patent document named "A Medical Image Segmentation Method Based on a Dual-Type U-Net Model" (application number: CN2021107283406), which discloses a medical image segmentation method based on a dual-type U-Net model. This method uses two U-Net networks in series and introduces an ASPP structure to resample the feature layer to capture more context information. However, both of the above two methods based on convolutional neural networks are limited by the local receptive field of the convolutional neural network and cannot model global or long-distance context interaction information and spatial dependence information. When segmenting medical images with complex background content or unclear image features, their accuracy is still not high.

[0005] Although the semantic segmentation method based on Transformers can effectively extract global or long-distance context interaction information and spatial dependence information, due to its excessively high computational complexity and large number of model parameters, it requires a longer model training time and higher hardware configuration. Summary of the Invention

[0006] To solve the problems existing in the method based on convolutional neural networks in three-dimensional medical image segmentation, such as insufficient receptive field, inability to model global or long-distance context interaction information and three-dimensional spatial dependence information, and being interfered by image background noise, the present invention discloses a brain tumor MRI image segmentation method based on a masked attention mechanism.

[0007] The present invention discloses a brain tumor MRI image segmentation method based on a masked attention mechanism, which is characterized in that the method comprises the following steps:

[0008] S1: Construct a 3D Masked Swin Transformer module based on a standard Swin Transformer module;

[0009] The 3D Masked Swin Transformer module is composed of 3D Masked Swin Transformer module A and 3D Masked Swin Transformer module B;

[0010] The 3D Masked Swin Transformer module A is composed of a first Layer Normalization layer, a 3D Window based Multi-head Masked Self Attention layer, a second Layer Normalization layer, and a first Multi-Layer Perceptron layer connected in sequence. A residual connection is added between the input end of the first Layer Normalization layer and the output end of the 3D Window based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the second Layer Normalization layer and the output end of the first Multi-Layer Perceptron layer;

[0011] The second 3D Masked Swin Transformer module B is composed of a third Layer Normalization layer, a 3D Shifted Window based Multi-head Masked Self Attention layer, a fourth Layer Normalization layer, and a second Multi-Layer Perceptron layer connected in sequence. A residual connection is added between the input end of the third Layer Normalization layer and the output end of the 3D Shifted Window based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the fourth Layer Normalization layer and the output end of the second Multi-Layer Perceptron layer;

[0012] S2: Construct a three-dimensional medical image segmentation model A; the three-dimensional medical image segmentation model A consists of an encoder module and a decoder module based on the 3D Masked Swin Transformer module; the input of the three-dimensional medical image segmentation model A is the preprocessed brain MRI medical image, and the output is the tumor region segmentation result; the encoder module based on the 3D Masked Swin Transformer module is sequentially connected by a 3D Patch Partition module, a first 3D Masked Swin Transformer module, a first downsampling module, a second 3D Masked Swin Transformer module, a second downsampling module, a third 3D Masked Swin Transformer module, a third downsampling module, and a fourth 3D Masked Swin Transformer module; the decoder module is sequentially connected by a first upsampling module, a first skip connection module, a first decoding module, a second upsampling module, a second skip connection module, a second decoding module, a third upsampling module, a third skip connection module, a third decoding module, and a segmentation head; the first skip connection module, the second skip connection module, and the third skip connection module are all sequentially connected by a Cross Attention block and a feature fusion block, and the first skip connection module, the second skip connection module, and the third skip connection module all have two input paths and one output path;

[0013] Among them, the output path of the first 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the first downsampling module, and the output of the second path is input into the second input path of the third skip connection module; the output path of the second 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the second downsampling module, and the output of the second path is input into the second input path of the second skip connection module; the output path of the third 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the third downsampling module, and the output of the second path is input into the second input path of the first skip connection module; the output path of the fourth 3D Masked Swin Transformer module is one path, and the output of this path is input into the first upsampling module of the decoder module; the output of the first upsampling module is input into the first input path of the first skip connection module, and the output of the first skip connection module is input into the first decoding module; the output of the second upsampling module is input into the first input path of the second skip connection, and the output of the second skip connection module is input into the second decoding module; the output of the third upsampling module is input into the first input path of the third skip connection, and the output of the third skip connection module is input into the third decoding module; the output of the third decoding module is input into the segmentation head, and the segmentation head finally outputs the required segmentation result;

[0014] S3: Obtain the original medical image dataset, perform data preprocessing on the original medical image dataset, and divide the preprocessed medical image dataset into a training set and a test set; Obtain the original medical image dataset, and each original brain MRI medical image in the original medical image dataset includes I t1 、I t2 、I t1ce 、I flair three-dimensional images of four modalities, as well as a real label three-dimensional image containing tumor annotation label information; perform data preprocessing on each original brain MRI medical image in the original medical image dataset to obtain the preprocessed brain MRI medical image, and all the preprocessed brain MRI medical images are combined to form the preprocessed medical image dataset; finally, divide the preprocessed medical image dataset into a training set and a test set, and the division ratios are α and 1 - α respectively, where α is a set parameter;

[0015] S4: Set the optimizer, the number of iterations e, the initial learning rate l, the loss function, and the online data augmentation method for network model training. Use the training set to train the 3D medical image segmentation model A constructed in step S2 to obtain the trained 3D medical image segmentation model B, and use the test set to evaluate the training effect of the 3D medical image segmentation model B;

[0016] S5: Use the 3D medical image segmentation model B to segment the tumor region in the original brain MRI medical image; perform data preprocessing on the original brain MRI medical image to obtain the preprocessed brain MRI medical image, and input the preprocessed brain MRI medical image into the trained 3D medical image segmentation model B in step S4 for prediction to obtain the segmentation result.

[0017] In step S1, given the input feature X of the 3D Window based Multi-head Masked Self Attention layer, the input feature X of the 3D Window based Multi-head Masked Self Attention layer is operated by the 3D Window based Multi-head Masked Self Attention layer to obtain the output feature Y of the 3D Window based Multi-head Masked Self Attention layer; among them, the operation process of the 3D Window based Multi-head Masked Self Attention layer is as follows: first, evenly divide the input feature X of the 3D Window based Multi-head Masked Self Attention layer into N sets of feature blocks with a window size of w×w×w Among them, the spatial position coordinates of the feature block are denoted as (a i , b i , c i ); then perform the Multi-head Masked Self Attention operation on each feature block in the set of feature blocks to obtain the set of transformed feature blocks Finally, splice all the transformed feature blocks in the set of transformed feature blocks , and place the transformed feature block at the spatial position (a i , b i , c i) At this point, the output feature Y of the 3DWindow based Multi-head Masked Self Attention layer is obtained; where the calculation formula of the Multi-head Masked Self Attention operation is:

[0018]

[0019] In the formula, and respectively represent the input feature of the Multi-head Masked Self Attention operation and the output feature of the Multi-head Masked Self Attention operation; f1, f2, and f3 are all linear transformation functions. Preferably, f1, f2, and f3 all adopt convolution functions; M represents the three-dimensional background region mask map of the brain MRI medical image; d represents the number of channels; softmax represents the softmax function.

[0020] In step S1, given the input feature X NS of the 3D Shifted Window based Multi-head Masked Self Attention layer, the input feature X NS of the 3D Shifted Window based Multi-head Masked Self Attention layer undergoes operations through the 3D Shifted Window based Multi-head Masked Self Attention layer to obtain the output feature Y NS of the 3D Shifted Window based Multi-head Masked Self Attention layer; where the operation process of the 3D Shifted Window based Multi-head Masked Self Attention layer is as follows: First, the input feature X NS undergoes cyclic shift operations in the positive directions of the three spatial axes to obtain the shifted feature X S , where the shift step is where w > 1, and the symbol represents rounding down the value in the symbol; then the input shifted feature X S is evenly divided into N sets of shifted feature blocks with a window size of w×w×w where the shifted feature block has its spatial position coordinates denoted as (ai , b i , c i ); Then, for each shifted feature block in the set of shifted feature blocks , perform the Multi-head Masked Self Attention operation to obtain the set of shifted transformation feature blocks Then, concatenate all the shifted transformation feature blocks in the set of shifted transformation feature blocks . Place the shifted transformation feature block at the spatial position (a i , b i , c i ) to obtain the shifted transformation feature Y S ; Finally, perform three cyclic shift operations in the reverse direction of the three spatial axes on the shifted transformation feature Y S to obtain the output feature Y NS of the 3D Shifted Window based Multi-head Masked Self Attention layer.

[0021] In step S2, the calculation formula of the 3D Patch Partition module is:

[0022] P = LN(conv(K))

[0023] In the formula, K and P respectively represent the input feature and the output feature of the 3D Patch Partition module; conv represents the convolution operation; LN represents the Layer Normalization operation.

[0024] In step S2, the calculation formula of the first 3D Masked Swin Transformer module is:

[0025]

[0026] In the formula, F represents the input feature of the first 3D Masked Swin Transformer module; Z 1 and Z 2 respectively represent the output features of the first 3D Swin Transformer module and the second 3D Swin Transformer module; and respectively represent the output features of the 3D Window based Multi-head Masked SelfAttention layer in the 3D Masked Swin Transformer module A and the output features of the 3D Shifted Window based Multi-head Masked Self Attention layer in the 3D Masked Swin Transformer module B; 3DW-MMSA and 3DSW-MMSA respectively represent the 3D Window based Multi-head Masked Self Attention layer and the 3D Shifted Window based Multi-head Masked Self Attention layer; MLP 1 and MLP 2 respectively represent the first Multi-Layer Perceptron layer and the second Multi-Layer Perceptron layer; LN 1 and LN 2 and LN 3 and LN 4 respectively represent the first Layer Normalization layer, the second Layer Normalization layer, the third Layer Normalization layer and the fourth Layer Normalization layer.

[0027] In step S2, the second 3D Masked Swin Transformer module, the third 3D Masked Swin Transformer module and the fourth 3D Masked Swin Transformer module adopt the same calculation method as the first 3D Masked Swin Transformer module.

[0028] In step S2, the calculation formula of the first skip connection module is as follows:

[0029]

[0030] In the formula, represents the output features of the Cross Attention block in the first skip connection module; X1 and X2 respectively represent the input features of the first input path and the second input path in the first skip connection module; Z represents the output features of the first skip connection module; CA and Fusion respectively represent the Cross Attention block and the feature fusion block.

[0031] In step S2, the second skip connection module and the third skip connection module adopt the same calculation method as the first skip connection module.

[0032] In step S2, the calculation formula of the Cross Attention block is as follows:

[0033]

[0034] In the formula, X1 and X2 respectively represent the two input features of the Cross Attention block; represents the output feature of the Cross Attention block; f4, f5, and f6 are all linear transformation functions. Preferably, f4, f5, and f6 all adopt convolution functions; d represents the number of channels of X2; softmax represents the softmax function.

[0035] In step S2, the calculation formula of the feature fusion block is as follows:

[0036] O = ReLU(conv(cat(I1, I2)))

[0037] In the formula, I1 and I2 respectively represent the first input feature of the feature fusion block and the second input feature of the feature fusion block; O represents the output feature of the feature fusion block; conv represents the convolution operation; cat represents the splicing operation of the two input features in the channel dimension; ReLU represents the ReLU activation function.

[0038] In step S2, the calculation formula of the first decoding module is as follows:

[0039] D out = conv(ReLU(BN(conv(ReLU(BN(D in ))))) + D in

[0040] In the formula, D in and D out respectively represent the input feature of the first decoding module and the output feature of the first decoding module; conv represents the convolution operation; BN represents the batch normalization operation; ReLU represents the ReLU activation function.

[0041] In step S2, the second decoding module and the third decoding module adopt the same calculation method as the first decoding module.

[0042] In step S2, the calculation formula of the segmentation head is as follows:

[0043] Seg = softmax(conv(upsample(L)))

[0044] In the formula, L and Seg respectively represent the input feature of the segmentation head and the output result of the segmentation head; conv represents the convolution operation; upsample represents the upsampling operation; softmax represents the softmax function.

[0045] In steps S3 and S5, data preprocessing is performed on the original brain MRI medical image to obtain the brain MRI medical image after data preprocessing. The specific steps are as follows:

[0046] (4) Multi-modal image merging: Perform multi-modal image merging on the original brain MRI medical image to obtain a multi-modal mosaic image I raw , and the specific operation process of the multi-modal image merging is as follows: Select 4 modal images I t1 、I t2 、I t1ce 、I flair in the original brain MRI medical image, and perform a splicing operation on I t1 、I t2 、I t1ce 、I flair on the channel dimension to obtain a multi-modal mosaic image I raw ;

[0047] (5) Data standardization: Perform data standardization on the multi-modal mosaic image I raw to obtain a standardized image I z-score , and the calculation formula of the data standardization is as follows:

[0048]

[0049] In the formula, I raw and I z-score respectively represent the multi-modal mosaic image I raw and the standardized image I z-score ; μ represents the mean value of the multi-modal mosaic image I raw ; σ represents the standard deviation of the multi-modal mosaic image I raw ;

[0050] (6) Background mask map extraction: Perform background mask map extraction on the standardized image I z-score to obtain a background mask map I masked , and the specific operation process of the background mask map extraction is as follows: First, slice the standardized image I z-score in the cross-sectional direction to obtain a two-dimensional slice image Then, perform two-dimensional background mask map extraction on all two-dimensional slice images respectively to obtain two-dimensional background mask maps Finally, arrange all two-dimensional background mask maps in the original slice order Perform stitching in the cross-sectional direction dimension to obtain the background mask image I masked ;

[0051] In the extraction of the background mask image, the specific operation process of the two-dimensional background mask image extraction is as follows: two-dimensional slice image First, perform image binarization operation to obtain a binary image Then, the binary image Perform morphological operations to obtain a binary image after morphological operations Finally, obtain the largest connected region of the binary image after morphological operations and label it as the image background region, and create an initial two-dimensional background region mask image whose size is the same as that of the binary image, and set the pixel values of the initial two-dimensional background region mask image in the image background region to -999, and set the pixel values of the initial two-dimensional background region mask image in the non-background region of the image to 0 to obtain a two-dimensional background region mask image

[0052] In step S4, preferably, the optimizer for model training uses the Adam optimizer, the number of training iterations e is set to 100000, the learning rate l is set to 0.01, and the loss function uses the BCEDiceLoss loss function; where, the calculation formula of the BCEDiceLoss is as follows:

[0053]

[0054] In the formula, I and J respectively represent the total number of three-dimensional image pixels and the number of prediction categories; G i,j represents the one-hot encoding of the true label at the j-th category of the i-th pixel, and Y i,j represents the one-hot encoding of the model prediction result at the j-th category of the i-th pixel.

[0055] In step S4, preferably, the online data augmentation method includes random translation, random horizontal mirror flipping, random vertical mirror flipping, random image scaling, and image cropping.

[0056] A brain tumor MRI image segmentation method based on a mask attention mechanism proposed by the present invention has the following main innovations:

[0057] In a method for segmenting brain tumor MRI images based on a masked attention mechanism proposed by the present invention, a 3D Masked Swin Transformer module is creatively proposed; using the 3D Masked Swin Transformer module to extract features from images can effectively extract the global or long-range context interaction information and spatial dependence information of the images; at the same time, a masked self-attention mechanism is introduced to activate only the features of the key non-background regions, thereby reducing the interference of the background regions and enabling the segmentation model proposed by the present invention to obtain better segmentation performance; compared with most standard transformer modules, the 3D Masked Swin Transformer module proposed by the present invention has a simpler computational complexity and fewer parameters.

[0058] In a method for segmenting brain tumor MRI images based on a masked attention mechanism proposed by the present invention, a three-dimensional medical image segmentation model is proposed; the three-dimensional medical image segmentation model effectively performs feature fusion on the encoded features and the decoded features by using a skip connection module, improving the segmentation performance of the three-dimensional medical image segmentation model proposed by the present invention.

[0059] The advantages of the present invention are as follows: This method can effectively apply the Transformer to medical image segmentation, with strong generalization ability, high anti-interference ability and segmentation accuracy. Brief Description of the Drawings

[0060] The following further elaborates the present invention in detail in conjunction with the drawings and specific embodiments.

[0061] Figure 1 It is a flowchart of the method of the embodiment of the present invention.

[0062] Figure 2 It is a structural diagram of the 3D Masked Swin Transformer module proposed in the embodiment of the present invention.

[0063] Figure 3 It is a structural diagram of the three-dimensional medical image segmentation model proposed in the embodiment of the present invention.

[0064] Figure 4 It is a structural diagram of the skip connection module proposed in the embodiment of the present invention.

[0065] Figure 5 It is a flowchart of extracting the background mask image in the embodiment of the present invention. Detailed Description of the Invention

[0066] Next, the technical solutions in the specific embodiments of the present invention will be described in complete and clear detail in conjunction with the accompanying drawings in the embodiments of the present invention; it should be noted that here, specific embodiments of a brain tumor MRI image segmentation method based on a masked attention mechanism according to the present invention are only for illustration and are not used to limit the present invention;

[0067] A brain tumor MRI image segmentation method based on a masked attention mechanism provided in this embodiment is as Figure 1 shown, and the specific steps are as follows:

[0068] S1: Build a 3D Masked Swin Transformer module based on the standard Swin Transformer module; the 3D Masked Swin Transformer module consists of a 3D Masked Swin Transformer module A and a 3D Masked Swin Transformer module B; the 3D Masked Swin Transformer module A is composed of a first Layer Normalization layer, a 3D Window based Multi-head Masked Self Attention layer, a second Layer Normalization layer, and a first Multi-Layer Perceptron layer connected in sequence. Among them, a residual connection is added between the input end of the first Layer Normalization layer and the output end of the 3D Window based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the second Layer Normalization layer and the output end of the first Multi-Layer Perceptron layer; the second 3D Masked Swin Transformer module B is composed of a third Layer Normalization layer, a 3D Shifted Window based Multi-head Masked Self Attention layer, a fourth Layer Normalization layer, and a second Multi-Layer Perceptron layer connected in sequence. Among them, a residual connection is added between the input end of the third Layer Normalization layer and the output end of the 3D Shifted Window based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the fourth Layer Normalization layer and the output end of the second Multi-Layer Perceptron layer;

[0069] Specifically, given the input feature X of the 3D Window based Multi-head Masked Self Attention layer, the input feature X of the 3D Window based Multi-head Masked Self Attention layer is operated by the 3D Window based Multi-head Masked Self Attention layer to obtain the output feature Y of the 3D Window based Multi-head Masked Self Attention layer; wherein, the operation process of the 3D Window based Multi-head Masked Self Attention layer is as follows: First, the input feature X of the 3D Window based Multi-head Masked Self Attention layer is evenly divided into N sets of feature blocks with a window size of 4×4×4 Among them, the feature block The spatial position coordinates of are denoted as (a i , b i , c i ); Then, perform the Multi-head Masked Self Attention operation on each feature block in the set of feature blocks to obtain a set of transformed feature blocks Finally, splice all the transformed feature blocks in the set of transformed feature blocks , and place the transformed feature block at the spatial position (a i , b i , c i ) to obtain the output feature Y of the 3D Window based Multi-head Masked Self Attention layer; wherein, the calculation formula of the Multi-head Masked Self Attention operation is:

[0070]

[0071] In the formula, and respectively represent the input feature of the Multi-head Masked Self Attention operation and the output feature of the Multi-head Masked Self Attention operation; f1, f2, f3 all use convolution functions, and the convolution kernel size is 1×1×1; M represents the three-dimensional background region mask map of the brain MRI medical image; d represents The number of channels; softmax represents the softmax function;

[0072] Specifically, given the input feature X of the 3D Shifted Window based Multi-head Masked Self Attention layer NS , the input feature X of the 3D Shifted Window based Multi-head Masked Self Attention layer NS Through the operation of the 3D Shifted Window based Multi-head Masked Self Attention layer, the output feature Y of the 3D Shifted Window based Multi-head Masked Self Attention layer is obtained NS ; Among them, the operation process of the 3D Shifted Window based Multi-head Masked Self Attention layer is as follows: First, the input feature X NS Obtain the shifted feature X through three cyclic shift operations in the positive direction of the spatial axes S , where the shift step is 2; Then divide the input shifted feature X S Evenly divide it into N sets of shifted feature blocks with a window size of 4×4×4 Among them, the spatial position coordinates of the shifted feature block Are denoted as (a i , b i , c i ); Then perform the Multi-head Masked Self Attention operation on each feature block in the set of shifted feature blocks To obtain a set of shifted transformation feature blocks Then splice all the shifted transformation feature blocks in the set of shifted transformation feature blocks Place the shifted transformation feature block At the spatial position (a i , b i , c i ) to obtain the shifted transformation feature Y S ; Finally, perform three cyclic shift operations in the reverse direction of the spatial axes on the shifted transformation feature Y S To obtain the output feature Y of the 3D Shifted Window based Multi-head Masked Self Attention layer NS ;

[0073] The standard Swin Transformer module mentioned in step S1 can be found in "Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 10012-10022.";

[0074] The Multi-Layer Perceptron layer mentioned in steps S1-S2 can be found in "Rumelhart D E, Hinton G E, Williams R J. Learning representations by back-propagating errors[J]. nature, 1986, 323(6088): 533-536.";

[0075] The Layer Normalization layer mentioned in steps S1-S2 can be found in "Ba J L, Kiros J R, Hinton G E. Layer normalization[J]. arXiv preprint arXiv:1607.06450, 2016.";

[0076] S2: Construct a three-dimensional medical image segmentation model A; Figure 3Shown is a three-dimensional medical image segmentation model A constructed in an embodiment of the present invention; the three-dimensional medical image segmentation model A is composed of an encoder module and a decoder module based on a 3D Masked SwinTransformer module; the input of the three-dimensional medical image segmentation model A is a pre-processed brain MRI medical image, and the output is a tumor region segmentation result; the encoder module based on the 3D Masked SwinTransformer module is sequentially connected by a 3D Patch Partition module, a first 3D Masked SwinTransformer module, a first downsampling module, a second 3D Masked Swin Transformer module, a second downsampling module, a third 3D Masked Swin Transformer module, a third downsampling module, and a fourth 3D Masked SwinTransformer module; the decoder module is sequentially connected by a first upsampling module, a first skip connection module, a first decoding module, a second upsampling module, a second skip connection module, a second decoding module, a third upsampling module, a third skip connection module, a third decoding module, and a segmentation head; the first skip connection module, the second skip connection module, and the third skip connection module are all sequentially connected by a Cross Attention block and a feature fusion block, and the first skip connection module, the second skip connection module, and the third skip connection module all have two input paths and one output path; Figure 4 Shown is a structural diagram of a skip connection module constructed in an embodiment of the present invention;

[0077] Specifically, the output path of the first 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the first downsampling module, and the output of the second path is input into the second input path of the third skip connection module; the output path of the second 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the second downsampling module, and the output of the second path is input into the second input path of the second skip connection module; the output path of the third 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the third downsampling module, and the output of the second path is input into the second input path of the first skip connection module; the output path of the fourth 3D Masked Swin Transformer module is one path, and the output of this path will be input into the first upsampling module of the decoder module; the output of the first upsampling module is input into the first input path of the first skip connection module, and the output of the first skip connection module is input into the first decoding module; the output of the second upsampling module is input into the first input path of the second skip connection, and the output of the second skip connection module is input into the second decoding module; the output of the third upsampling module is input into the first input path of the third skip connection, and the output of the third skip connection module is input into the third decoding module; the output of the third decoding module is input into the segmentation head, and the segmentation head finally outputs the required segmentation result;

[0078] Specifically, the calculation formula of the 3D Patch Partition module is:

[0079] P = LN(conv(K))

[0080] In the formula, K and P respectively represent the input feature of the 3D Patch Partition module and the output feature of the 3D Patch Partition module; conv represents the convolution operation; LN represents the Layer Normalization operation;

[0081] Specifically, the calculation formula of the first 3D Masked Swin Transformer module is:

[0082]

[0083] In the formula, F represents the input feature of the first 3D Masked Swin Transformer module; Z 1 and Z 2respectively represent the output features of the first 3D Swin Transformer module and the second 3D Swin Transformer module; and respectively represent the output features of the 3D Window based Multi-head Masked SelfAttention layer in the 3D Masked Swin Transformer module A and the output features of the 3D Shifted Windowbased Multi-head Masked Self Attention layer in the 3D Masked Swin Transformer module B; 3DW-MMSA and 3DSW-MMSA respectively represent the 3D Window based Multi-head Masked Self Attention layer and the 3D Shifted Window basedMulti-head Masked Self Attention layer; MLP 1 and MLP 2 respectively represent the first Multi-LayerPerceptron layer and the second Multi-Layer Perceptron layer; LN 1 and LN 2 and LN 3 and LN 4 respectively represent the first LayerNormalization layer, the second Layer Normalization layer, the third Layer Normalization layer and the fourth Layer Normalization layer;

[0084] Specifically, the second 3D Masked Swin Transformer module, the third 3D Masked SwinTransformer module and the fourth 3D Masked Swin Transformer module adopt the same calculation method as the first 3DMaskedSwin Transformer module;

[0085] Specifically, the calculation formula of the first skip connection module is as follows:

[0086]

[0087] In the formula, Denote the output features of the Cross Attention block in the first skip connection module; X1 and X2 respectively denote the input features of the first input path and the second input path in the first skip connection module; Z denotes the output features of the first skip connection module; CA and Fusion respectively denote the Cross Attention block and the feature fusion block;

[0088] Specifically, the second skip connection module and the third skip connection module adopt the same calculation method as the first skip connection module;

[0089] Specifically, the calculation formula of the Cross Attention block is as follows:

[0090]

[0091] In the formula, X1 and X2 respectively denote the two input features of the Cross Attention block; Denote the output features of the Cross Attention block; f4, f5, and f6 all adopt convolution functions; d denotes the number of channels of X2; softmax denotes the softmax function;

[0092] Specifically, the calculation formula of the feature fusion block is as follows:

[0093] O = ReLU(conv(cat(I1, I2)))

[0094] In the formula, I1 and I2 respectively denote the first input feature and the second input feature of the feature fusion block; O denotes the output feature of the feature fusion block; conv denotes the convolution operation; cat denotes the operation of concatenating the two input features in the channel dimension; ReLU denotes the ReLU activation function;

[0095] Specifically, the calculation formula of the first decoding module is as follows:

[0096] D out = conv(ReLU(BN(conv(ReLU(BN(D in ))))) + D in

[0097] In the formula, D in and D out respectively denote the input feature and the output feature of the first decoding module; conv denotes the convolution operation; BN denotes the batch normalization operation; ReLU denotes the ReLU activation function;

[0098] Specifically, the second decoding module and the third decoding module adopt the same calculation method as the first decoding module;

[0099] Specifically, the calculation formula of the segmentation head is as follows:

[0100] Seg = softmax(conv(upsample(L)))

[0101] In the formula, L and Seg respectively represent the input feature of the segmentation head and the output result of the segmentation head; conv represents the convolution operation; upsample represents the upsampling operation; softmax represents the softmax function;

[0102] Specifically, the input image of the three-dimensional medical image segmentation model A is the preprocessed brain MRI medical image X input , where the size of the preprocessed brain MRI medical image X input is 128×128×128, and the number of channels is 4; the brain MRI medical image X input is operated by the 3D Patch Partition module to obtain the block feature X patch , and the size of the block feature X patch is 32×32×32, and the number of channels is 64; the block feature X patch is input into the first 3D Masked Swin Transformer module;

[0103] Specifically, after the block feature X patch is operated by the first 3D Masked Swin Transformer module, the first encoded feature e1 is obtained. The size of the first encoded feature e1 is 32×32×32, and the number of channels is 64; the first encoded feature e1 is input into the first downsampling module from the first output path of the first 3D Masked Swin Transformer module. At the same time, the first encoded feature e1 is input into the second input path of the third skip connection module from the second output path of the first 3D Masked Swin Transformer module;

[0104] Specifically, after the first encoded feature e1 is operated by the first downsampling module, the first downsampled feature e d1 is obtained. The size of the first downsampled feature e d1 is 16×16×16, and the number of channels is 128; the first downsampled feature e d1 is input into the second 3D Masked Swin Transformer module;

[0105] Specifically, the first downsampled feature e d1 After being processed by the second 3D Masked Swin Transformer module, the second encoded feature e2 is obtained. The size of the second encoded feature e2 is 16×16×16, and the number of channels is 128. The second encoded feature e2 is input into the second downsampling module from the first output path of the second 3D Masked Swin Transformer module. At the same time, the second encoded feature e2 is input into the second input path of the second skip connection module from the second output path of the second 3D Masked Swin Transformer module;

[0106] Specifically, after the second encoded feature e2 is processed by the second downsampling module, the second downsampled feature e d2 is obtained. The size of the second downsampled feature e d2 is 8×8×8, and the number of channels is 256. The second downsampled feature e d2 is input into the third 3D Masked Swin Transformer module;

[0107] Specifically, after the second downsampled feature e d2 is processed by the third 3D Masked Swin Transformer module, the third encoded feature e3 is obtained. The size of the third encoded feature e3 is 8×8×8, and the number of channels is 256. The third encoded feature e3 is input into the third downsampling module from the first output path of the third 3D Masked Swin Transformer module. At the same time, the third encoded feature e3 is input into the second input path of the first skip connection module from the second output path of the third 3D Masked Swin Transformer module;

[0108] Specifically, after the third encoded feature e3 is processed by the third downsampling module, the third downsampled feature e d3 is obtained. The size of the third downsampled feature e d3 is 4×4×4, and the number of channels is 512. The third downsampled feature e d3 is input into the fourth 3D Masked Swin Transformer module;

[0109] Specifically, the third downsampled feature e d3After the operation of the fourth D Masked Swin Transformer module, the fourth encoded feature e4 is obtained. The size of the fourth encoded feature e4 is 4×4×4, and the number of channels is 512. The fourth encoded feature e4 is input into the first upsampling module;

[0110] Specifically, after the operation of the first upsampling module on the fourth encoded feature e4, the first upsampled feature d is obtained u1 , the first upsampled feature d u1 has a size of 8×8×8 and the number of channels is 256. The first upsampled feature d u1 is input into the first input path of the first skip connection module;

[0111] Specifically, after the operation of the first skip connection module on the first upsampled feature d u1 and the third encoded feature e3, the first skip connection feature d is obtained s1 , the first skip connection feature d s1 has a size of 8×8×8 and the number of channels is 256. The first skip connection feature d s1 is input into the first decoding module;

[0112] Specifically, after the operation of the first decoding module on the first skip connection feature d s1 , the first decoded feature d1 is obtained. The size of the first decoded feature d1 is 8×8×8, and the number of channels is 256. The first decoded feature d1 is input into the second upsampling module;

[0113] Specifically, after the operation of the second upsampling module on the first decoded feature d1, the second upsampled feature d is obtained u2 , the second upsampled feature d u2 has a size of 16×16×16 and the number of channels is 128. The second upsampled feature d u2 is input into the first input path of the second skip connection module;

[0114] Specifically, after the operation of the second skip connection module on the second upsampled feature d u2 and the second encoded feature e2, the second skip connection feature d is obtained s2 , the second skip connection feature d s2 has a size of 16×16×16 and the number of channels is 128. The second skip connection feature d s2 is input into the second decoding module;

[0115] Specifically, the second skip connection feature d s2After being processed by the second decoding module, the second decoded feature d2 is obtained. The size of the second decoded feature d2 is 16×16×16, and the number of channels is 128. The second decoded feature d2 is input into the third upsampling module.

[0116] Specifically, after being processed by the third upsampling module, the second decoded feature d2 obtains the third upsampled feature d u3 , and the size of the third upsampled feature d u3 is 32×32×32, and the number of channels is 64. The third upsampled feature d u3 is input into the first input path of the third skip connection module.

[0117] Specifically, after being processed by the third skip connection module, the third upsampled feature d u3 and the first encoded feature e1 obtain the third skip connection feature d s3 , and the size of the third skip connection feature d s3 is 32×32×32, and the number of channels is 64. The third skip connection feature d s3 is input into the third decoding module.

[0118] Specifically, after being processed by the third decoding module, the third skip connection feature d s3 obtains the third decoded feature d3. The size of the third decoded feature d3 is 32×32×32, and the number of channels is 64. The third decoded feature d3 is input into the segmentation head.

[0119] Specifically, through the segmentation head, the third decoded feature d3 obtains the tumor region segmentation result Y output , and the size of the tumor region segmentation result Y output is 128×128×128, and the number of channels is 3.

[0120] For the ReLU activation function mentioned in step S2, see "Glorot X, Bordes A, Bengio Y. Deepsparse rectifier neural networks[C] / / Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011: 315-323.";

[0121] S3: Obtain the original medical image dataset, perform data preprocessing on the original medical image dataset, and divide the preprocessed medical image dataset into a training set and a test set; obtain the brain tumor medical image dataset BraTS2019, where the brain tumor medical image dataset BraTS2019 contains 335 case data, and each case data in the brain tumor medical image dataset BraTS2019 includes I t1 、I t2 、I t1ce 、I flair three-dimensional images of four modalities, as well as a ground truth three-dimensional image containing tumor annotation label information; perform data preprocessing on each original brain MRI medical image in the brain tumor medical image dataset BraTS2019 to obtain the brain MRI medical image after data preprocessing, and all the brain MRI medical images after data preprocessing are combined together to form the medical image dataset after data preprocessing; finally, divide the medical image dataset after data preprocessing into a training set and a test set, and the division ratios are 80% and 20% respectively;

[0122] Specifically, the steps for performing data preprocessing on the original brain MRI medical image to obtain the brain MRI medical image after data preprocessing are as follows:

[0123] (1) Multi-modal image merging: Perform multi-modal image merging on the original brain MRI medical image to obtain a multi-modal mosaic image I raw , and the specific operation process of the multi-modal image merging is as follows: Select 4 modal images I t1 、I t2 、I t1ce 、I flair in the original brain MRI medical image. The sizes of the 4 modal images I t1 、I t2 、I t1ce 、I flair are all 240×240×155, and the number of channels is 1; perform a splicing operation on I t1 、I t2 、I t1ce 、I flair in the channel dimension to obtain a multi-modal mosaic image I raw . The size of the multi-modal mosaic image I raw is 240×240×155, and the number of channels is 4;

[0124] (2) Data standardization: Perform data standardization on the multi-modal mosaic image I raw to obtain a standardized image I z-score , and the standardized image I z-scoreIt has a size of 240×240×155 and 4 channels; the calculation formula for data standardization is as follows:

[0125]

[0126] In the formula, I raw and I z-score respectively represent the multi-modal mosaic image I raw and the standardized image I z-score ; μ represents the mean value of I z-score ; σ represents the standard deviation of I z-score ;

[0127] (3) Background mask map extraction: Extract the background mask map from the standardized image I z-score to obtain the background mask map I masked . The specific operation process for background mask map extraction is as follows: First, slice the standardized image I z-score in the cross-sectional direction to obtain two-dimensional slice images Then, perform two-dimensional background mask map extraction on all two-dimensional slice images respectively to obtain two-dimensional background mask maps Finally, splice all two-dimensional background mask maps in the cross-sectional direction dimension according to the original slice order to obtain the background mask map I masked ;

[0128] Specifically, the specific operation process for two-dimensional background mask map extraction is as follows: The two-dimensional slice image first undergoes image binarization operation to obtain a binary image Then, the binary image undergoes morphological operations to obtain a binary image after morphological operations Finally, obtain the largest connected region of the binary image after morphological operations and mark it as the image background region. Create an initial two-dimensional background region mask map with the same size as the binary image. Set the pixel values of the initial two-dimensional background region mask map in the image background region to -999, and set the pixel values of the initial two-dimensional background region mask map in the non-background region of the image to 0 to obtain the two-dimensional background region mask map

[0129]

[0130] S4: Set the optimizer, the number of iterations e, the initial learning rate l, the loss function, and the online data augmentation method for network model training. Use the training set to train the 3D medical image segmentation model A in step S2 to obtain the trained 3D medical image segmentation model B, and use the test set to evaluate and test the training effect of the 3D medical image segmentation model B;

[0131] Specifically, the optimizer for model training uses the Adam optimizer, the training iteration number e is set to 100000, the learning rate l is set to 0.01, and the loss function uses the BCEDiceLoss loss function; among them, the calculation formula of the BCEDiceLoss is as follows:

[0132]

[0133] In the formula, I and J respectively represent the total number of 3D image pixels and the number of prediction categories; G i,j represents the one-hot encoding of the true label in the j-th category of the i-th pixel, and Y i,j represents the one-hot encoding of the model prediction result in the j-th category of the i-th pixel;

[0134] Specifically, the online data augmentation method includes random translation, random horizontal mirror flipping, random vertical mirror flipping, random image scaling, and image cropping; among them, the size of the image cropping is 128×128×128;

[0135] S5: Use the 3D medical image segmentation model B to segment the tumor region in the original brain MRI medical image; perform data preprocessing on the original brain MRI medical image to obtain the preprocessed brain MRI medical image, and input the preprocessed brain MRI medical image into the 3D medical image segmentation model B in step S4 for prediction to obtain the segmentation result.

Claims

1. A method for segmenting brain tumor MRI images based on a masked attention mechanism, characterized in that, The method comprises the following steps: S1: Construct a 3D Masked Swin Transformer module based on the standard Swin Transformer module; The 3D Masked Swin Transformer module consists of a 3D Masked Swin Transformer module A and a 3D Masked Swin Transformer module B; The 3D Masked Swin Transformer module A is composed of a first Layer Normalization layer, a 3D Window-based Multi-head Masked Self Attention layer, a second Layer Normalization layer, and a first Multi-Layer Perceptron layer connected in sequence. A residual connection is added between the input end of the first Layer Normalization layer and the output end of the 3D Window-based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the second Layer Normalization layer and the output end of the first Multi-Layer Perceptron layer; The 3D Masked Swin Transformer module B is composed of a third Layer Normalization layer, a 3D Shifted-Window-based Multi-head Masked Self Attention layer, a fourth Layer Normalization layer, and a second Multi-Layer Perceptron layer connected in sequence. A residual connection is added between the input end of the third Layer Normalization layer and the output end of the 3D Shifted Window-based Multi-head Masked Self Attention layer, and a residual connection is added between the input end of the fourth Layer Normalization layer and the output end of the second Multi-Layer Perceptron layer; S2: Construct a three-dimensional medical image segmentation model A; the three-dimensional medical image segmentation model A consists of an encoder module and a decoder module based on the 3D Masked Swin Transformer module; the input of the three-dimensional medical image segmentation model A is a pre-processed brain MRI medical image, and the output is the tumor region segmentation result; The encoder module based on the 3D Masked Swin Transformer module is composed of a 3D Patch Partition module, a first 3D Masked Swin Transformer module, a first downsampling module, a second 3D Masked Swin Transformer module, a second downsampling module, a third 3D Masked Swin Transformer module, a third downsampling module, and a fourth 3D Masked Swin Transformer module connected in sequence; the decoder module is composed of a first upsampling module, a first skip connection module, a first decoding module, a second upsampling module, a second skip connection module, a second decoding module, a third upsampling module, a third skip connection module, a third decoding module, and a segmentation head connected in sequence; the first skip connection module, the second skip connection module, and the third skip connection module are each composed of a Cross Attention block and a feature fusion block connected in sequence, and the first skip connection module, the second skip connection module, and the third skip connection module each have two input paths and one output path; Among them, the output path of the first 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the first downsampling module, and the output of the second path is input into the second input path of the third skip connection module; the output path of the second 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the second downsampling module, and the output of the second path is input into the second input path of the second skip connection module; the output path of the third 3D Masked Swin Transformer module is divided into two paths. The output of the first path is input into the third downsampling module, and the output of the second path is input into the second input path of the first skip connection module; the output path of the fourth 3D Masked Swin Transformer module is one path, and the output of this path is input into the first upsampling module of the decoder module; the output of the first upsampling module is input into the first input path of the first skip connection module, and the output of the first skip connection module is input into the first decoding module; the output of the second upsampling module is input into the first input path of the second skip connection, and the output of the second skip connection module is input into the second decoding module; the output of the third upsampling module is input into the first input path of the third skip connection, and the output of the third skip connection module is input into the third decoding module; the output of the third decoding module is input into the segmentation head, and the segmentation head finally outputs the required segmentation result; S3: Obtain the original medical image dataset, perform data preprocessing on the original medical image dataset, and divide the preprocessed medical image dataset into a training set and a test set; Obtain the original medical image dataset, where each original brain MRI medical image in the original medical image dataset includes I t1 、I t2 、I t1ce 、I flair three-dimensional images of four modalities, as well as a true label three-dimensional image containing tumor annotation label information; Perform data preprocessing on each original brain MRI medical image in the original medical image dataset to obtain a preprocessed brain MRI medical image, and all the preprocessed brain MRI medical images are combined to form a preprocessed medical image dataset; Finally, divide the preprocessed medical image dataset into a training set and a test set, with the division ratios being α and 1 - α respectively, where α is a set parameter; S4: Set the optimizer, number of iterations e, initial learning rate l, loss function, and online data augmentation method for network model training. Use the training set to train the 3D medical image segmentation model A constructed in step S2 to obtain the trained 3D medical image segmentation model B, and use the test set to evaluate the training effect of the 3D medical image segmentation model B; S5: Use the 3D medical image segmentation model B to segment the tumor region in the original brain MRI medical image; perform data preprocessing on the original brain MRI medical image to obtain the preprocessed brain MRI medical image, and input the preprocessed brain MRI medical image into the trained 3D medical image segmentation model B in step S4 for prediction to obtain the segmentation result.

2. The method for brain tumor MRI image segmentation based on masked attention mechanism according to claim 1, characterized in that: In step S1, given the input feature X of the 3D Window based Multi-head Masked Self Attention layer, the input feature X of the 3D Window based Multi-head Masked Self Attention layer is operated by the 3D Window based Multi-head Masked Self Attention layer to obtain the output feature Y of the 3D Window based Multi-head Masked Self Attention layer; Among them, the operation process of the 3D Window based Multi-head Masked Self Attention layer is as follows: First, the input feature X of the 3D Window based Multi-head Masked Self Attention layer is evenly divided into N sets of feature blocks with a window size of w×w×w. Among them, the feature block has its spatial position coordinates denoted as (a i , b i , c i ); Then, perform the Multi-head Masked SelfAttention operation on each feature block in the set of feature blocks to obtain a set of transformed feature blocks Finally, concatenate all the transformed feature blocks in the set of transformed feature blocks , and place the transformed feature block at the spatial position (a i , b i , c i ) to obtain the output feature Y of the 3D Window based Multi-head Masked Self Attention layer; Among them, the calculation formula for the Multi-head Masked SelfAttention operation is: In the formula, and respectively represent the input feature of the Multi-head Masked Self Attention operation and the output feature of the Multi-head Masked Self Attention operation; f1, f2, and f3 are all linear transformation functions; M represents the three-dimensional background region mask map of the brain MRI medical image; d represents the number of channels; softmax represents the softmax function.

3. The method for brain tumor MRI image segmentation based on masked attention mechanism according to claim 1, characterized in that: In step S1, the input feature X of the 3D Shifted Window based Multi-head Masked Self Attention layer is given NS , the input feature X of the 3D Shifted Window based Multi-head Masked Self Attention layer NS is operated through the 3D Shifted Window based Multi-head Masked Self Attention layer to obtain the output feature Y of the 3D Shifted Window based Multi-head Masked Self Attention layer NS ; Among them, the operation process of the 3D Shifted Window based Multi-head Masked Self Attention layer is as follows: First, for the input feature X NS The shifted feature X is obtained through three cyclic shift operations in the positive directions of the three spatial axes S , where the step size of the shift is where w > 1, and the symbol represents rounding down the value in the symbol; then the input shifted feature X S is evenly divided into N sets of shifted feature blocks with a window size of w×w×w where the shifted feature block has its spatial position coordinates denoted as (a i , b i , c i ); then, a Multi-head Masked Self Attention operation is performed on each shifted feature block in the set of shifted feature blocks to obtain a set of shifted transformation feature blocks Then, all the shifted transformation feature blocks in the set of shifted transformation feature blocks are concatenated, and the shifted transformation feature block is placed at the spatial position (a i , b i , c i ) to obtain the shifted transformation feature Y S ; finally, the shifted transformation feature Y S is subjected to three cyclic shift operations in the reverse directions of the three spatial axes to obtain the output feature Y of the 3D Shifted Window based Multi-head Masked Self Attention layer NS .

4. The method for brain tumor MRI image segmentation based on masked attention mechanism according to claim 1, characterized in that: In step S2, the calculation formula of the 3D Patch Partition module is: P = LN(conv(K)) In the formula, K and P respectively represent the input feature of the 3D Patch Partition module and the output feature of the 3D Patch Partition module; conv represents the convolution operation; LN represents the Layer Normalization operation.

5. The method for brain tumor MRI image segmentation based on masked attention mechanism according to claim 1, characterized in that: In step S2, the calculation formula of the first 3D Masked Swin Transformer module is: Where F represents the input feature of the first 3D Masked Swin Transformer module; Z 1 and Z 2 represent the output features of the first 3D Swin Transformer module and the second 3D Swin Transformer module respectively; and represent the output features of the 3D Window based Multi-head Masked SelfAttention layer in 3D Masked Swin Transformer module A and the output features of the 3D Shifted Windowbased Multi-head Masked Self Attention layer in 3D Masked Swin Transformer module B respectively; 3DW-MMSA and 3DSW-MMSA represent the 3D Window based Multi-head Masked Self Attention layer and the 3D Shifted Window based Multi-head Masked Self Attention layer respectively; MLP 1 and MLP 2 represent the first Multi-Layer Perceptron layer and the second Multi-Layer Perceptron layer respectively; LN 1 and LN 2 and LN 3 and LN 4 represent the first Layer Normalization layer, the second Layer Normalization layer, the third Layer Normalization layer and the fourth Layer Normalization layer respectively; In step S2, the second 3D Masked Swin Transformer module, the third 3D Masked Swin Transformer module, and the fourth 3D Masked Swin Transformer module adopt the same calculation method as the first 3D Masked Swin Transformer module.

6. A method for segmenting brain tumor MRI images based on a masked attention mechanism according to claim 1, wherein: In step S2, the calculation formula of the first skip connection module is as follows: In the formula, represents the output feature of the Cross Attention block in the first skip connection module; X1 and X2 respectively represent the input features of the first input path in the first skip connection module and the input features of the second input path in the first skip connection module; Z represents the output feature of the first skip connection module; CA and Fusion respectively represent the Cross Attention block and the feature fusion block; In step S2, the second skip connection module and the third skip connection module adopt the same calculation method as the first skip connection module.

7. A method for segmenting brain tumor MRI images based on a masked attention mechanism according to claim 1, wherein: In step S2, the calculation formula of the Cross Attention block is as follows: Wherein, X1 and X2 respectively represent two input features of the Cross Attention block; represents the output feature of the Cross Attention block; f4, f5, and f6 are all linear transformation functions; d represents the number of channels of X2; softmax represents the softmax function.

8. A method for segmenting brain tumor MRI images based on a masked attention mechanism according to claim 1, wherein: In step S2, the calculation formula of the feature fusion block is as follows: O = ReLU(conv(cat(I1, I2))) In the formula, I1 and I2 respectively represent the first input feature of the feature fusion block and the second input feature of the feature fusion block; O represents the output feature of the feature fusion block; conv represents a convolution operation; cat represents the operation of concatenating the two input features in the channel dimension; ReLU represents the ReLU activation function; In step S2, the calculation formula of the first decoding module is as follows: D out = conv(ReLU(BN(conv(ReLU(BN(D in )))))) + D in Where D in and D out respectively represent the input feature of the first decoding module and the output feature of the first decoding module; conv represents a convolution operation; BN represents a batch normalization operation; ReLU represents the ReLU activation function; In step S2, the second decoding module and the third decoding module adopt the same calculation method as the first decoding module; In step S2, the calculation formula of the segmentation head is as follows: Seg = softmax(conv(upsample(L))) In the formula, L and Seg respectively represent the input feature of the segmentation head and the output result of the segmentation head; conv represents a convolution operation; upsample represents an upsampling operation; softmax represents the softmax function.

9. A method for segmenting brain tumor MRI images based on a masked attention mechanism according to claim 1, wherein: In steps S3 and S5, the original brain MRI medical image is preprocessed to obtain a preprocessed brain MRI medical image. The specific steps are as follows: (1) Multi-modal image merging: Perform multi-modal image merging on the original brain MRI medical image to obtain a multi-modal spliced image I raw , and the specific operation process of the multi-modal image merging is as follows: Select 4 modal images I t1 , I t2 , I t1ce , I flair in the original brain MRI medical image, and splice I t1 , I t2 , I t1ce , I flair in the channel dimension to obtain the multi-modal spliced image I raw ; (2) Data standardization: For the multi-modal spliced image I raw Perform data standardization to obtain the standardized image I z-score , and the calculation formula for the data standardization is as follows: Wherein, I raw and I z-score respectively represent the multi-modal mosaic image I raw and the normalized image I z-score ; μ represents the mean value of the multi-modal mosaic image I raw ; σ represents the standard deviation of the multi-modal mosaic image I raw ; (3) Background mask image extraction: For the normalized image I z-score perform background mask image extraction to obtain the background mask image I masked . The specific operation process of the background mask image extraction is as follows: First, slice the normalized image I z-score in the cross-sectional direction to obtain two-dimensional slice images . Then, perform two-dimensional background mask image extraction on all two-dimensional slice images respectively to obtain two-dimensional background mask images . Finally, splice all two-dimensional background mask images in the cross-sectional direction dimension according to the original slice order to obtain the background mask image I masked ; In the extraction of the background mask image, the specific operation process of the two-dimensional background mask image extraction is as follows: the two-dimensional slice image First, perform image binarization operation to obtain a binary image Then, the binary image Undergo morphological operations to obtain a binary image after morphological operations Finally, obtain the largest connected region of the binary image after morphological operations And mark it as the image background region, create a new initial two-dimensional background region mask image The size of which is the same as that of the binary image, set the pixel values of the initial two-dimensional background region mask image In the image background region to -999, and set the pixel values of the initial two-dimensional background region mask image In the non-background region of the image to 0 to obtain a two-dimensional background region mask image 10. A method for segmenting brain tumor MRI images based on a masked attention mechanism according to claim 1, wherein: In step S4, the optimizer for model training adopts the Adam optimizer. The number of training iterations e is set to 100000, the learning rate l is set to 0.01, and the loss function uses the BCEDiceLoss loss function; wherein, the calculation formula of the BCEDiceLoss is as follows: Wherein, I and J respectively represent the total number of three-dimensional image pixels and the number of prediction categories; G i,j represents the one-hot encoding of the true label in the j-th category of the i-th pixel, and Y i,j represents the one-hot encoding of the model prediction result in the j-th category of the i-th pixel; In step S4, the online data augmentation method includes random translation, random horizontal mirror flipping, random vertical mirror flipping, random image scaling, and image cropping.

Images