Multi-modal brain tumor segmentation method based on frequency domain channel attention

Abstract

The invention discloses a multi-modal brain tumor segmentation method based on frequency domain channel attention. The method comprises the following steps: obtaining a training data set; step 2, establishing a multi-modal brain tumor segmentation network which comprises two encoders and two decoders, the outputs of the first pooling layer of the two encoders being connected with a first FCMM module, the outputs of the second pooling layer being connected with a second FCMM module, the outputs of the third pooling layer being connected with a third FCMM module, the outputs of the fourth pooling layer being connected with a fourth FCMM module, and the outputs of the second pooling layer being connected with a second FCMM module; the fourth FCMM module is sequentially connected with fifth, sixth, seventh and eighth upper sampling layers; splicing the output feature of the eighth up-sampling layer with the output feature of the first convolutional layer in the two encoders, and then inputting the spliced output feature into a seventh convolutional layer to generate a prediction result; step 3, training is carried out; and step 4, calculating a loss function, carrying out back propagation, and updating network model parameters. According to the method, the high-frequency component and the low-frequency component are introduced into the channel dimension, so that complementary information and redundant information in the multi-modal data can be more effectively represented.

Description

Technical Field

[0001] The present invention relates to a multimodal brain tumor segmentation method based on frequency domain channel attention. Background Art

[0002] Brain tumors are among the most life-threatening malignancies, and timely diagnosis and treatment play a crucial role in prolonging patient survival and improving quality of life. Currently, magnetic resonance imaging (MRI) is widely used in the clinical diagnosis of brain tumors due to its ability to provide high-resolution anatomical structures, clear soft tissue contrast, and detailed information. Unlike traditional single-modality medical image segmentation, brain tumor segmentation involves four modalities: T1-weighted imaging (T1), contrast-enhanced imaging (T1ce), T2-weighted imaging (T2W), and fluid-attenuated inversion recovery (FLAIR). Different modalities have different representation capabilities for brain tumors, making traditional single-modality medical image segmentation methods often ineffective in achieving ideal results. Although multimodal MRI data provides rich information, effectively integrating the complementary information between these different modalities while avoiding information redundancy caused by the introduction of additional modalities has become a key challenge that needs to be addressed. When dealing with this problem, traditional methods often fail to fully utilize the synergy between various modalities, resulting in limited segmentation accuracy, especially poor performance in tumor boundary definition and sub-region division.

[0003] The patent application with application number 202311181827.2 discloses a multimodal MRI brain tumor semantic segmentation method based on a hierarchical fusion strategy. In the multimodal fusion process, the method uses AvgPool and MaxPool to represent average pooling respectively. First, the first splicing module in the method is used to splice the encoding results corresponding to the single modality input images of different stages to obtain multimodal feature maps of different stages; the multimodal feature maps of the current stage are input into the maximum pooling layer and the average pooling layer respectively, and the maximum pooling layer is used to collect the unique features in different modalities, and the average pooling layer is used to collect the shared feature information of different modalities, and the average pooling layer is used to collect the average pooling channel features M respectively. avg and the maximum pooling channel feature M max . The average pooled channel feature M avg and the maximum pooling channel feature M maxThe input is fed into a multilayer perceptron for combining feature map channel information to obtain a first feature map and a second feature map respectively; the first feature map and the second feature map are subjected to matrix addition operations to obtain a multimodal channel attention map, which is used as the input of a Gaussian distribution function to finally obtain a multimodal attention map. This method only uses the attention mechanism to learn the intrinsic correlation between multimodal information, and does not have a good strategy to characterize the complementary information and redundant information in multimodal data. Instead, the original data mixed with complementary information and redundant information is directly fed into the multimodal fusion module for learning, making it difficult to learn effective features. In addition, the framework of the first prior art also has defects. Specifically, the encoder of the framework of this method has different streams, each stream corresponding to a modality, but there is no corresponding mechanism in the decoder to supervise the feature extraction process, so the results of feature extraction are often not accurate enough.

[0004] "Yaru Cheng, Yuanjie Zheng, and Junxia Wang. Cfnet: Automatic multi-modal brain tumor segmentation through hierarchical coarse-to-fine fusion and feature communication. Biomed Signal Processing and Control, 99:106876, 2025." This paper proposes a novel fusion mechanism that leverages the potential correlations between multimodal images to promote cross-modal information fusion. This method first extracts features through two encoder branches, which are then input into a multimodal fusion module. The features extracted by the framework are first fed into the corresponding linear layers to obtain F1 and F2, which in turn yield Q1, V1, K1, and Q2, V2, K2. V1 and K1 are multiplied, softmaxed, multiplied with Q2, and added to F1 to obtain M1. Simultaneously, V2 and K2 are multiplied, softmaxed, multiplied with Q1, and added to F2 to obtain M2. M1 and M2 are then passed through a batchnormalization layer and concatenated together, followed by a convolutional layer with a Relu activation function and a batchnormalization layer to obtain the final result. This method improves the attention mechanism, but still fails to distinguish between redundancy and complementarity in multimodal information. It directly feeds raw data mixed with complementary and redundant information into the attention mechanism module for learning. This makes it difficult for the attention mechanism to learn useful features and the inherent connections between multimodal data. Furthermore, the framework of the second existing technique also has flaws: the feature extraction process is unsupervised, and its quality cannot be guaranteed.

[0005] In current multimodal brain tumor feature fusion technology, a common approach is to simply concatenate features extracted from different modalities and then feed them into a learning module for further processing using an attention mechanism. However, this approach has a significant drawback: it cannot effectively distinguish between complementary and redundant information in multimodal features. Because complementary and redundant information are intermingled, even when an attention mechanism is employed to capture regions of interest and avoid irrelevant areas, its effectiveness is significantly reduced, as it struggles to accurately identify which features constitute valuable complementary information and which are redundant information that should be ignored.

[0006] Furthermore, existing multimodal brain tumor segmentation network frameworks lack in-depth research. Most current technologies simply process modal data through a simple feature extraction layer, without deeply exploring how to design more effective modules to improve model performance. This limits the model's ability to understand the inherent connections between multimodal data, thus affecting the final segmentation accuracy and reliability. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multimodal brain tumor segmentation method based on frequency domain channel attention, which can more effectively characterize the complementary information and redundant information in multimodal data by introducing high-frequency and low-frequency components in the channel dimension.

[0008] The objective of the present invention is achieved through the following technical solution: a multimodal brain tumor segmentation method based on frequency domain channel attention, comprising the following steps:

[0009] Step 1: Obtain a training dataset: A publicly available brain tumor segmentation dataset is used as the training set. All images in the dataset are labeled. The T1-T1ce modality in the training set is used as one set of inputs, and the T2-Flair modality is used as another set of inputs. These two sets of inputs are fed into the two encoders of the multimodal brain tumor segmentation network model for processing.

[0010] Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, each encoder is connected to a decoder; each encoder includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer connected in sequence; each decoder includes a fifth convolutional layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a sixth convolutional layer connected in sequence; the fourth pooling layer is connected to the fifth convolutional layer;

[0011] The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected in sequence;

[0012] The output of the fourth FCMM module is connected to the fifth upsampling layer, the sixth upsampling layer, the seventh upsampling layer, and the eighth upsampling layer in sequence; the output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result prediction;

[0013] The output of the first convolutional layer is skip-connected to the output of the fourth upsampling layer, the output of the second convolutional layer is skip-connected to the output of the third upsampling layer, the output of the third convolutional layer is skip-connected to the output of the second upsampling layer, and the output of the fourth convolutional layer is skip-connected to the output of the first upsampling layer.

[0014] In the FCMM module, data processing includes the following steps:

[0015] Step 2.1: The features obtained after the two groups of modalities are processed by the convolution layer are recorded as and j represents the jth convolutional layer, j = 1, 2, 3, 4; after the pooling layer, the pooling layer feature is recorded as and The pooling layer features and Perform a partitioning operation on the channel dimension to obtain a set of two feature blocks;

[0016] Step 2.2: For each pixel in the divided feature block set, use discrete cosine transform to assign high-frequency and low-frequency components, and calculate the scalar

[0017]

[0018] Represents the scalar value calculated based on the frequency components u and v at the pixel point (h, w);

[0019] Step 2.3: Add the results of all pixels in each feature block to obtain a scalar used to characterize the feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks each feature is divided into in the channel dimension when performing the partitioning operation;

[0020]

[0021] Represents the value obtained by adding all the scalar results obtained after assigning frequency information to each pixel point of each channel block;

[0022] Step 2.4: All the obtained features used to identify the corresponding feature blocks Putting them together, we get Freq:

[0023]

[0024] Freq=Cat(Freq1,Freq2);

[0025] Where Cat represents the splicing operation;

[0026] Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j , the specific process is as follows: j =Sigmoid(fc(Freq)), where fc represents a fully connected layer;

[0027] Step 2.6: Fuse the input features according to the learned weights: j Assigned to P1 j , 1-ω j Assigned to The results are added together and passed through a layer of convolution to obtain the fused feature P of this layer. fuse The specific formula is as follows:

[0028] Where conv represents the convolutional layer;

[0029] Step 2.7: Determine whether the current FCMM module is the first FCMM module. If not, downsample the output features of the previous FCMM module, concatenate the fused features with the downsampled features, and then convolve the concatenated features to obtain the output of the FCMM module. This is expressed as:

[0030] M j =conv(Cat(P fuse ,DownUpsample(M j-1 )))

[0031] Where DownUpsample represents the downsampling operation;

[0032] If yes, the fused result is output as the FCMM module:

[0033] M1=Pfuse ;

[0034] Step 3: Use the training set to train the multimodal brain tumor segmentation network model; send the two sets of inputs to the two encoders respectively, extract features through the encoders, and then use the two decoders to obtain features of the same size as the input modality, and calculate the loss with the label to supervise the entire feature extraction process; the loss functions obtained by the two decoders are respectively recorded as and

[0035] Step 4: Calculate the loss between the prediction result and the label to get Will and The three loss functions are added together to obtain the total loss function, and backpropagation is performed to update the network model parameters.

[0036] The beneficial effects of the present invention are:

[0037] 1. This invention innovatively assigns high-frequency and low-frequency components to multimodal features. This allocation method aims to capture complementary information in multimodal data through high-frequency components, while using low-frequency components to represent redundant information. This allows the model to more accurately distinguish and process complementary and redundant information mixed in multimodal data, greatly improving the efficiency of feature learning and promoting in-depth exploration of the inherent connections between modalities.

[0038] 2. This paper designs an efficient information processing flow, in which features extracted by the pooling layer are fed into the FCMM module for fusion, while features extracted by the convolutional layer are directly passed to the decoder corresponding to each feature extraction stream via skip connections. This strategy not only enhances the expressiveness of features but also ensures that each stream can generate independent predictions and compare them with the true labels to calculate the loss value. This supervision mechanism enhances the accuracy of the feature extraction process, thereby ensuring that the final features are of higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the structure of the multimodal brain tumor segmentation network of the present invention;

[0040] Figure 2 Schematic diagram of the structure of the FCMM module of the present invention;

[0041] Figure 3 Flowchart of data processing in the FCMM module of the present invention. DETAILED DESCRIPTION

[0042] Definitions of Abbreviations and Key Terms

[0043] FCMM: Frequency Channel Multimodal Fusion Module. This is the multimodal fusion module designed in this invention.

[0044] MRI (Magnetic Resonance Imaging) is a diagnostic technique that uses strong magnetic fields and radiofrequency waves to image the interior of the human body. It produces detailed images of internal structures, particularly soft tissues such as the brain and muscles. In medicine, MRI is widely used for disease detection and diagnosis.

[0045] Sigmoid activation function: A commonly used activation function that maps real numbers to the range (0, 1). It is often used in the output layer of binary classification problems to predict the probability of an event.

[0046] This invention provides an innovative new multimodal brain tumor segmentation network, which provides a supervised framework for feature extraction. Furthermore, the multimodal fusion method of this invention effectively distinguishes complementary and redundant information in multimodal data, enabling the model to more accurately capture the inherent relationships between different modalities. The technical solutions of this invention are further illustrated below with reference to the accompanying figures.

[0047] The present invention provides a multimodal brain tumor segmentation method based on frequency domain channel attention, comprising the following steps:

[0048] Step 1. Obtain the training data set: Use a publicly available brain tumor segmentation data set as the training set; the present invention uses the BRATS2018 data set as the original data. The BRATS data set is one of the MICAI challenges for brain tumor segmentation. The present invention selects the 2018 challenge data set, namely BRATS2018. The images in this data set are all self-labeled. The BRATS2018 data set includes 285 3D cases, each of which provides four modes of imaging data: T1, T1ce, T2, and Flair. The present invention uses 2D slices of the axial plane of the 3D image and normalizes the 2D slices. At the same time, in order to reduce unnecessary background areas and reduce computational costs, slices of 224×224 size are cropped from the center of each normalized image as training samples.

[0049] The T1-T1ce modality in the training set is used as one group of inputs, and the T2-Flair modality is used as another group of inputs; the two groups of inputs are respectively sent to the two encoders of the multimodal brain tumor segmentation network model of the present invention for processing.

[0050] Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, each encoder is connected to a decoder; the network structure is as follows Figure 1 As shown. Each encoder includes the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer, the third pooling layer, the fourth convolution layer, and the fourth pooling layer connected in sequence; each decoder includes the fifth convolution layer, the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer, and the sixth convolution layer connected in sequence; the fourth pooling layer is connected to the fifth convolution layer;

[0051] The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected in sequence;

[0052] The output of the fourth FCMM module is connected to the fifth upsampling layer, the sixth upsampling layer, the seventh upsampling layer, and the eighth upsampling layer in sequence; the output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result prediction;

[0053] In each encoder and decoder connected to it, the output of the first convolutional layer is skip-connected to the output of the fourth upsampling layer, the output of the second convolutional layer is skip-connected to the output of the third upsampling layer, the output of the third convolutional layer is skip-connected to the output of the second upsampling layer, and the output of the fourth convolutional layer is skip-connected to the output of the first upsampling layer. In this embodiment, each upsampling operation uses an upsampling + convolution operation. Specifically, each upsampling layer is processed through a convolutional layer before being input into the next upsampling layer. The overall operation constitutes "Upsample."

[0054] In brain tumor segmentation, different segmentation methods provide different features and multi-dimensional information. Although combining multimodal data can enhance the understanding of complex tumors in certain areas, not all modal information is beneficial. For specific tumor areas, introducing additional modes may cause redundancy or conflict, potentially reducing performance. High-frequency information captures local details, edge features and rapidly changing signals within the tumor, enabling the detection of fine structures in the image. Since different modalities are more sensitive to these specific details, high-frequency information shows strong complementarity between modalities. In contrast, low-frequency information reflects the overall structure and provides the overall shape and distribution characteristics of the tumor. Since the various modes of brain tumor segmentation often have similar large-scale features, low-frequency information tends to become more and more redundant with the introduction of other modes. Therefore, the present invention proposes a new FCMM module. By introducing high-frequency and low-frequency components in the channel dimension, the complementary and redundant information in multimodal data can be more effectively represented. This helps the model learn the intrinsic correlation between different modalities, capture the region of interest, and effectively suppress the influence of irrelevant areas. The structure of the FCMM module is as follows Figure 2 shown.

[0055] In the FCMM module, the data processing flow is as follows Figure 3 As shown, the specific process includes the following:

[0056] Step 2.1: The features obtained after the two groups of modalities are processed by the convolution layer are recorded as and j represents the jth convolutional layer, j = 1, 2, 3, 4; the pooling layer feature obtained after the pooling layer is recorded as P1 j and The pooling layer feature P1 j and Perform a partitioning operation on the channel dimension to obtain a set of two feature blocks;

[0057] Step 2.2: For each pixel in the divided feature block set, use discrete cosine transform (DCT) to assign high-frequency and low-frequency components, and calculate the scalar

[0058]

[0059] Represents the scalar value calculated based on the frequency components u and v at the pixel point (h, w), where H and W are the height and width of the input feature respectively, and the total number of pixels in the image is H*W;

[0060] Different values ​​of u and v represent different frequencies, and the specific values ​​can be set according to the situation. For the specific method of discrete cosine transform, please refer to "Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. Fcanet: Frequency channel attention networks. In Proceedings of the IEEE / CVF international conference on computer vision, pages 783-792, 2021", which divides the two-dimensional DCT into 7×7 sizes, that is, the frequency span in each dimension is from 0 to 6. The closer to 0, the lower the frequency, and the closer to 6, the higher the frequency. And our input feature is 224×224, so the feature size obtained by the first pooling layer is 112×112, the feature size obtained by the second pooling layer is 56×56, the feature size obtained by the third pooling layer is 28×28, and the feature size obtained by the fourth pooling layer is 14×14. The feature size of each layer can be divided by 7.

[0061] u and v take different values. If u takes a large value, then u is high frequency and v is low frequency. If u takes a small value or even 0, then u is low frequency and v is high frequency. The values ​​of u and v in this embodiment are shown in Table 1.

[0062] Table 1

[0063]

[0064] In the table, N represents the number of feature blocks divided in step 2.1. 1, 2, 3, and 4 represent the first FCMM, the second FCMM, the third FCMM, and the fourth FCMM, respectively.

[0065] Step 2.3: Add the results of all pixels in each feature block to obtain a scalar used to characterize the feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks each feature is divided into in the channel dimension when performing the partitioning operation;

[0066]

[0067] Represents the value obtained by adding all the scalar results obtained after assigning frequency information to each pixel point of each channel block; represents the features of the (h, w) pixel obtained by the jth pooling layer in the i-th encoder; i = 1, 2; j = 1, 2, 3, 4. Compared with traditional global maximum pooling or global average pooling operations, this method can retain more information.

[0068] Step 2.4: All the obtained features used to identify the corresponding feature blocks Putting them together, we get Freq:

[0069]

[0070] Freq=Cat(Freq1,Freq2);

[0071] Where Cat represents the splicing operation;

[0072] Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j , the specific process is as follows: j =Sigmoid(fc(Freq)), where fc represents a fully connected layer;

[0073] Step 2.6: Fuse the input features according to the learned weights: j Assigned to P1 j , 1-ω j Assigned to The results are added together and passed through a layer of convolution to obtain the fused feature P of this layer. fuse The specific formula is as follows:

[0074] Where conv represents the convolutional layer;

[0075] Step 2.7: In order to capture the contextual information in the fused features obtained by the previous FCMM module, it is necessary to determine whether there is a result from the previous FCMM module, that is, to determine whether the current FCMM module is the first FCMM module;

[0076] If not, the output features of the previous FCMM module are downsampled, and then the fused features are concatenated with the downsampled features, and the concatenated features are convolved to obtain the output of the FCMM module; expressed as:

[0077] M j =conv(Cat(P fuse ,DownUpsample(M j-1 )))

[0078] Where DownUpsample represents the downsampling operation;

[0079] If yes, the fused result is output as the FCMM module:

[0080] M1=P fuse

[0081] M j Or M1 as the output of the FCMM module.

[0082] Step 3: Use the training set to train the multimodal brain tumor segmentation network model; send the two sets of inputs to the two encoders respectively, extract features through the encoders, and then use the decoders to obtain features of the same size as the input modality. The feature map obtained by the decoders is compared with the label to calculate the loss and supervise the entire feature extraction process; the loss functions obtained by the two decoders are respectively recorded as and

[0083] Calculate the loss between the prediction results of the entire network model and the label, and get

[0084] Will and The three loss functions are added together to obtain the total loss function. The total loss function is used for back propagation to update the network model parameters and obtain the optimal network model.

[0085] In this embodiment, and All three loss functions use the calculation method of adding dice loss and cross entropy loss; the specific calculation formula is:

[0086]

[0087] L stream =L Dice +L CE

[0088] where y i and y′ i Represent the label value and the predicted value respectively, N represents the total number of pixels, and p(.) represents the softmax function.

[0089] Step 4: Use the trained network model for multimodal brain tumor segmentation. The extracted T1-T1ce and T2-Flair modalities are fed into two encoders, processed by the encoders and then by the FCMM module. Finally, predictions are obtained through multiple upsampling and convolutional layers. The decoder is primarily used to calculate the loss function and supervise model training during training. In practical applications, only the encoder and FCMM modules can be retained in the network. When applying the model to actual multimodal brain tumor segmentation, the decoder can be retained as needed (retaining the decoder allows for subsequent model updates, while omitting it allows for a more streamlined network structure).

[0090] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multimodal brain tumor segmentation method based on frequency domain channel attention, characterized in that: The following steps are involved: Step 1: Obtain a training dataset: Use a publicly available brain tumor segmentation dataset as the training set. All images in the dataset come with labels. Use the T1-T1ce modality in the training set as one set of inputs, and the T2-Flair modality as another set of inputs. Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, each encoder is connected to a decoder; each encoder includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer connected in sequence; each decoder includes a fifth convolutional layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a sixth convolutional layer connected in sequence; the fourth pooling layer is connected to the fifth convolutional layer; The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected in sequence; The output of the fourth FCMM module is connected to the fifth upsampling layer, the sixth upsampling layer, the seventh upsampling layer, and the eighth upsampling layer in sequence; the output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result prediction; The output of the first convolutional layer is skip-connected to the output of the fourth upsampling layer, the output of the second convolutional layer is skip-connected to the output of the third upsampling layer, the output of the third convolutional layer is skip-connected to the output of the second upsampling layer, and the output of the fourth convolutional layer is skip-connected to the output of the first upsampling layer. In the FCMM module, data processing includes the following steps: Step 2.1: The features obtained after the two groups of modalities are processed by the convolution layer are recorded as and j represents the jth convolutional layer, j = 1, 2, 3, 4; after the pooling layer, the pooling layer feature is recorded as and The pooling layer features and Perform a partitioning operation on the channel dimension to obtain a set of two feature blocks; Step 2.2: For each pixel in the divided feature block set, use discrete cosine transform to assign high-frequency and low-frequency components, and calculate the scalar Represents the scalar value calculated based on the frequency components u and v at the pixel point (h, w); Step 2.3: Add the results of all pixels in each feature block to obtain a scalar used to characterize the feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks each feature is divided into in the channel dimension when performing the partitioning operation; Represents the value obtained by adding all the scalar results obtained after assigning frequency information to each pixel point of each channel block; Step 2.4: All the obtained features used to identify the corresponding feature blocks Putting them together, we get Freq: Freq=Cat(Freq1,Freq2); Where Cat represents the splicing operation; Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j , the specific process is as follows: j =Sigmoid(fc(Freq)), where fc represents a fully connected layer; Step 2.6: Fuse the input features according to the learned weights: j Assigned to 1-ω j Assigned to The results are added together and passed through a layer of convolution to obtain the fused feature P of this layer. fuse The specific formula is as follows: Where conv represents the convolutional layer; Step 2.7: Determine whether the current FCMM module is the first FCMM module. If not, downsample the output features of the previous FCMM module, concatenate the fused features with the downsampled features, and then convolve the concatenated features to obtain the output of the FCMM module. This is expressed as: M j =conv(Cat(P fuse ,DownUpsample(M j-1 ))) Where DownUpsample represents the downsampling operation; If yes, the fused result is output as the FCMM module: M1=P fuse ; Step 3: Use the training set to train the multimodal brain tumor segmentation network model; send the two sets of inputs to the two encoders respectively, extract features through the encoders, and then use the two decoders to obtain features of the same size as the input modality, and calculate the loss with the label. The loss functions obtained by the two decoders are respectively recorded as and Step 4: Calculate the loss between the prediction result and the label to get Will and The three loss functions are added together to obtain the total loss function, and backpropagation is performed to update the network model parameters.

Images