A multi-modal brain tumor segmentation method based on frequency domain channel attention

Abstract

The application discloses a kind of multi-modal brain tumor segmentation methods based on frequency domain channel attention, comprising the following steps: obtaining training dataset;Step 2, establish multi-modal brain tumor segmentation network, including two encoders and two decoders, the output of the first pooling layer of two encoders is connected with the first FCMM module, the output of the second pooling layer is connected with the second FCMM module, the output of the third pooling layer is connected with the third FCMM module, the output of the fourth pooling layer is connected with the fourth FCMM module, the fourth FCMM module is sequentially connected with the fifth, sixth, seventh and eighth up-sampling layer;The output feature of the eighth up-sampling layer is spliced with the output feature of the first convolutional layer in two encoders, then input seventh convolutional layer, generate prediction result;Step 3, training;Step 4, calculate loss function, carry out back propagation, update network model parameters.The application can more effectively represent complementary information and redundant information in multi-modal data by introducing high-frequency and low-frequency components in channel dimension.

Description

Technical Field

[0001] This invention relates to a multimodal brain tumor segmentation method based on frequency domain channel attention. Background Technology

[0002] Brain tumors are among the most life-threatening cancers, and timely diagnosis and treatment play a crucial role in prolonging patient survival and improving quality of life. Currently, magnetic resonance imaging (MRI) is widely used in the clinical diagnosis of brain tumors due to its ability to provide high-resolution anatomical structures, clear soft tissue contrast, and detailed information. Unlike traditional single-modal medical image segmentation, brain tumor segmentation involves four modalities: T1-weighted imaging (T1), contrast-enhanced T1-weighted imaging (T1ce), T2-weighted imaging (T2W), and fluid attenuated inversion recovery (FLAIR). Different modalities have different characterization capabilities for brain tumors; therefore, traditional single-modal medical image segmentation methods often fail to achieve ideal results in brain tumor segmentation tasks. Although multimodal MRI data provides rich information, effectively fusing the complementary information between these different modalities while avoiding information redundancy caused by introducing additional modalities has become a critical challenge that urgently needs to be addressed. Traditional methods often fail to fully utilize the synergistic effects between different modalities when dealing with this problem, resulting in limited segmentation accuracy, especially in tumor boundary delineation and subregion division.

[0003] Patent application number 202311181827.2 discloses a multimodal MRI brain tumor semantic segmentation method based on a hierarchical fusion strategy. In the multimodal fusion process, this method uses AvgPool and MaxPool to represent average pooling, respectively. First, the first concatenation module in the method concatenates the encoding results corresponding to the single modal input images at different stages to obtain multimodal feature maps at different stages. The multimodal feature map of the current stage is then input into the max pooling layer and the average pooling layer, respectively. The max pooling layer collects the unique features of different modalities, and the average pooling layer collects the shared feature information of different modalities, respectively, to obtain the average pooling channel features M. avg and maximum pooling channel features M max The average pooling channel feature M avg and maximum pooling channel features M maxThe input is fed into a multilayer perceptron that combines the channel information of the feature maps, resulting in a first feature map and a second feature map. A matrix addition operation is then performed on the first and second feature maps to obtain a multimodal channel attention map. This multimodal channel attention map is then used as the input to a Gaussian distribution function to finally obtain the multimodal attention map. This method only uses an attention mechanism to learn the intrinsic relationships between multimodal information, lacking a good strategy to represent complementary and redundant information in the multimodal data. Instead, it directly feeds the raw data, which contains a mixture of complementary and redundant information, into the multimodal fusion module for learning, making it difficult to learn effective features. Furthermore, the framework of the existing technique also has shortcomings. Specifically, the encoder of this framework has different streams, each corresponding to a modality, but the decoder lacks a corresponding mechanism to supervise the feature extraction process, resulting in inaccurate feature extraction results.

[0004] The paper, "Yaru Cheng, Yuanjie Zheng, and Junxia Wang. Cfnet: Automatic multi-modal brain tumor segmentation through hierarchical coarse-to-fine fusion and feature communication. Biomedullary Signal Processing and Control, 99:106876, 2025," proposes a novel fusion mechanism that leverages the latent correlations between multimodal images to promote cross-modal information fusion. The method first extracts features through two encoder branches, then inputs these two features into a multimodal fusion module. The features extracted from the framework are first fed into the corresponding linear layers to obtain F1 and F2, and then sequentially Q1,V1,K1 and Q2,V2,K2. V1K1 is multiplied and subjected to a softmax operation, then multiplied with Q2 and added to F1 to obtain M1. Simultaneously, V2K2 is multiplied and subjected to a softmax operation, then multiplied with Q1 and added to F2 to obtain M2. Next, M1 and M2 are processed through a batchnormalization layer and concatenated together. The final result is then obtained through a convolutional ReLU activation function and another batchnormalization layer. This method improves the attention mechanism, but it still doesn't distinguish between redundancy and complementarity in multimodal information. It directly inputs the raw data, with complementary and redundant information mixed together, into the attention mechanism module for learning. This makes it difficult for the attention mechanism to learn useful features and the inherent relationships between multimodal data. Furthermore, the framework of the existing technique two also has flaws; the feature extraction process is unsupervised, and the quality of feature extraction cannot be guaranteed.

[0005] In current multimodal brain tumor feature fusion techniques, a common approach is to simply concatenate features extracted from different modalities and then feed them into a learning module for further processing using an attention mechanism. However, this method has a significant drawback: it cannot effectively distinguish between complementary and redundant information in multimodal features. Because complementary and redundant information are mixed together, even with an attention mechanism to capture regions of interest and avoid irrelevant regions, the effectiveness of this mechanism is greatly reduced, as it struggles to accurately identify which features are valuable complementary information and which are redundant information that should be ignored.

[0006] Furthermore, existing multimodal brain tumor segmentation network frameworks lack in-depth research. Most current techniques simply process modal data through simple feature extraction layers without exploring how to design more effective modules to improve model performance. This limits the model's ability to understand the intrinsic relationships between multimodal data, thus affecting the final segmentation accuracy and reliability. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a frequency domain channel attention-based multimodal brain tumor segmentation method that can more effectively characterize complementary and redundant information in multimodal data by introducing high-frequency and low-frequency components in the channel dimension.

[0008] The objective of this invention is achieved through the following technical solution: a multimodal brain tumor segmentation method based on frequency domain channel attention, comprising the following steps:

[0009] Step 1: Obtain the training dataset: Use a publicly available brain tumor segmentation dataset as the training set. All images in the dataset are labeled. Use the T1-T1ce modality in the training set as one set of inputs and the T2-Flair modality as another set of inputs. Feed the two sets of inputs into the two encoders of the multimodal brain tumor segmentation network model for processing.

[0010] Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, with each encoder connected to one decoder. Each encoder includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer connected in sequence. Each decoder includes a fifth convolutional layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a sixth convolutional layer connected in sequence. The fourth pooling layer is connected to the fifth convolutional layer.

[0011] The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected sequentially.

[0012] The output of the fourth FCMM module is sequentially connected to the fifth, sixth, seventh, and eighth upsampling layers. The output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result.

[0013] The outputs of the first convolutional layer and the fourth upsampling layer are connected in a skip connection; the outputs of the second convolutional layer and the third upsampling layer are connected in a skip connection; the outputs of the third convolutional layer and the second upsampling layer are connected in a skip connection; the outputs of the fourth convolutional layer and the first upsampling layer are connected in a skip connection.

[0014] In the FCMM module, data processing includes the following steps:

[0015] Step 2.1: Denote the features obtained after processing the two sets of modes through the convolutional layer as follows: and j represents the j-th convolutional layer, j = 1, 2, 3, 4; the pooling layer features obtained after the pooling layer are denoted as... and Pooling layer features and Perform a partitioning operation on the channel dimension to obtain a set consisting of two feature blocks;

[0016] Step 2.2: For each pixel in the partitioned feature block set, use discrete cosine transform to assign high-frequency and low-frequency components, and calculate the scalar.

[0017]

[0018] This represents the scalar value calculated based on the frequency components u and v at pixel point (h,w);

[0019] Step 2.3: Sum the results obtained from all pixels in each feature block to obtain a scalar representing that feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks to be divided for each feature in the channel dimension when performing the partitioning operation;

[0020]

[0021] This represents the value obtained by summing all the scalar results after assigning frequency information to each pixel in each channel block.

[0022] Step 2.4: Collect all the obtained identifiers used to identify the corresponding feature blocks. By piecing them together, we get Freq:

[0023]

[0024] Freq = Cat(Freq1, Freq2);

[0025] Where Cat represents the splicing operation;

[0026] Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j The specific process is as follows: ω j =Sigmoid(fc(Freq)), where fc represents a fully connected layer;

[0027] Step 2.6: Fuse the input features based on the learned weights: ω j Assigned to P1 j , 1-ω j Assigned to The results are then summed and passed through a convolutional layer to obtain the fused feature P. fuse The specific formula is as follows:

[0028] Where conv represents a convolutional layer;

[0029] Step 2.7: Determine if the current FCMM module is the first FCMM module; if not, downsample the output features of the previous FCMM module, then concatenate the fused features with the downsampled features, and then convolve the concatenated features to obtain the output of the current FCMM module; represented as:

[0030] M j =conv(Cat(P) fuse DownUpsample(M j-1 )))

[0031] Where DownUpsample represents the downsampling operation;

[0032] If so, the fused result will be the output of the FCMM module:

[0033] M1 = Pfuse ;

[0034] Step 3: Train the multimodal brain tumor segmentation network model using the training set; feed the two sets of inputs into two encoders respectively, extract features through the encoders, and then pass them through two decoders to obtain features of the same size as the input modalities. Calculate the loss with the labels to supervise the entire feature extraction process; the loss functions obtained from the two decoders are denoted as follows: and

[0035] Step 4: Calculate the loss between the prediction result and the label, and obtain... Will and The three loss functions are added together to obtain the total loss function, and then backpropagation is performed to update the network model parameters.

[0036] The beneficial effects of this invention are:

[0037] 1. This invention innovatively assigns high-frequency and low-frequency components to multimodal features. This assignment method aims to capture complementary information in multimodal data through high-frequency components, while using low-frequency components to represent redundant information. In this way, the model can more accurately distinguish and process complementary and redundant information mixed in multimodal data, greatly improving the efficiency of feature learning and promoting in-depth exploration of the intrinsic relationships between different modalities.

[0038] 2. This invention designs an efficient information processing flow, in which features extracted by the pooling layer are fed into the FCMM module for fusion, while features extracted by the convolutional layer are directly passed to the decoder corresponding to each feature extraction stream through skip connections. This strategy not only enhances the expressive power of the features but also ensures that each stream can generate an independent prediction result, which is then compared with the true label to calculate the loss value. This supervision mechanism strengthens the accuracy of the feature extraction process, thereby ensuring that the final obtained features have higher quality. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of the multimodal brain tumor segmentation network of the present invention;

[0040] Figure 2 This is a schematic diagram of the FCMM module of the present invention;

[0041] Figure 3 This is a flowchart of data processing in the FCMM module of the present invention. Detailed Implementation

[0042] Definitions of abbreviations and key terms

[0043] FCMM: Frequency Channel Multimodal Fusion Module. It is the multimodal fusion module designed in this invention.

[0044] MRI (Magnetic Resonance Imaging) is a diagnostic technique that uses strong magnetic fields and radio frequency waves to image the interior of the human body. It can generate detailed images of internal structures, and is particularly suitable for imaging soft tissues such as the brain and muscles. In the medical field, MRI is widely used for disease detection and diagnosis.

[0045] The Sigmoid activation function is a commonly used activation function. It maps real numbers to the interval (0,1) and is often used in the output layer of binary classification problems to predict the probability of events occurring.

[0046] This invention provides an innovative new multimodal brain tumor segmentation network, which offers a supervised framework for the feature extraction process. Furthermore, the multimodal fusion method of this invention effectively distinguishes between complementary and redundant information in multimodal data, enabling the model to more accurately capture the intrinsic relationships between different modalities. The technical solution of this invention is further described below with reference to the accompanying drawings.

[0047] The present invention provides a multimodal brain tumor segmentation method based on frequency domain channel attention, comprising the following steps:

[0048] Step 1: Obtain the training dataset: A publicly available brain tumor segmentation dataset is used as the training set; this invention uses the brats2018 dataset as the original data. The Brats dataset is one of the miccai challenges for brain tumor segmentation. This invention selects the 2018 challenge dataset, namely brats2018, in which all images are self-labeled. The Brats2018 dataset includes 285 3D cases, each providing imaging data in four modes: T1, T1ce, T2, and Flair. This invention utilizes 2D slices of the body-axis plane of the 3D images and normalizes the 2D slices. Simultaneously, to reduce unnecessary background areas and lower computational costs, a 224×224 slice is cropped from the center of each normalized image as a training sample.

[0049] The T1-T1ce modalities in the training set are used as one set of inputs, and the T2-Flair modalities are used as another set of inputs. The two sets of inputs are respectively fed into the two encoders of the multimodal brain tumor segmentation network model of the present invention for processing.

[0050] Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, with each encoder connected to one decoder; the network structure is as follows. Figure 1 As shown. Each encoder includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer connected in sequence; each decoder includes a fifth convolutional layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a sixth convolutional layer connected in sequence; the fourth pooling layer is connected to the fifth convolutional layer;

[0051] The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected sequentially.

[0052] The output of the fourth FCMM module is sequentially connected to the fifth, sixth, seventh, and eighth upsampling layers. The output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result.

[0053] In each encoder and its connected decoder, the outputs of the first convolutional layer and the fourth upsampling layer are connected in a skip connection; the outputs of the second convolutional layer and the third upsampling layer are connected in a skip connection; the outputs of the third convolutional layer and the second upsampling layer are connected in a skip connection; and the output of the fourth convolutional layer and the first upsampling layer are connected in a skip connection. In this embodiment, each upsampling operation employs an upsampling + convolution operation. Specifically, each upsampled layer is processed by a convolutional layer before being input into the next upsampling layer; the entire operation constitutes "Upsample".

[0054] In brain tumor segmentation, different segmentation methods provide different features and multidimensional information. While combining multimodal data can enhance the understanding of complex tumors in certain regions, not all modal information is beneficial. For a specific tumor region, introducing additional modalities may lead to redundancy or conflict, potentially degrading performance. High-frequency information captures local details, edge features, and rapidly changing signals within the tumor, enabling the detection of fine structures in the image. Since different modalities are more sensitive to these specific details, high-frequency information exhibits strong complementarity between modalities. Conversely, low-frequency information reflects the overall structure, providing the overall shape and distribution characteristics of the tumor. Because various modalities in brain tumor segmentation often have similar large-scale features, low-frequency information tends to become increasingly redundant with the introduction of other modalities. Therefore, this invention proposes a novel FCMM module. By introducing high-frequency and low-frequency components in the channel dimension, complementary and redundant information in multimodal data can be more effectively characterized. This facilitates the model learning the intrinsic correlations between different modalities, capturing regions of interest, while effectively suppressing the influence of irrelevant regions. The structure of the FCMM module is as follows: Figure 2 As shown.

[0055] In the FCMM module, the data processing flow is as follows: Figure 3 As shown, the specific process includes the following:

[0056] Step 2.1: Denote the features obtained after processing the two sets of modes through the convolutional layer as follows: and j represents the j-th convolutional layer, j = 1, 2, 3, 4; the pooling layer feature obtained after the pooling layer is denoted as P1. j and Pooling layer feature P1 j and Perform a partitioning operation on the channel dimension to obtain a set consisting of two feature blocks;

[0057] Step 2.2: For each pixel in the partitioned feature block set, use Discrete Cosine Transform (DCT) to assign high-frequency and low-frequency components, and calculate the scalar.

[0058]

[0059] This represents the scalar value calculated based on the frequency components u and v at pixel (h,w), where H and W are the height and width of the input feature, respectively, and the total number of pixels in the image is H*W.

[0060] Different values ​​of u and v represent different frequencies, and the specific values ​​can be set according to the situation. For a detailed method of Discrete Cosine Transform (DCT), refer to "Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. Fcanet: Frequencychannel attention networks. In Proceedings of the IEEE / CVF international conference on computer vision, pages 783-792, 2021". Divide the two-dimensional DCT into a 7×7 size, meaning the frequency range in each dimension is from 0 to 6. The closer to 0, the lower the frequency; and the closer to 6, the higher the frequency. Our input feature size is 224×224. Thus, the feature size obtained by the first pooling layer is 112×112, the second pooling layer is 56×56, the third pooling layer is 28×28, and the fourth pooling layer is 14×14. The feature size of each layer is divisible by 7.

[0061] u and v take different values. If u takes a large value, then u is a high frequency and v is a low frequency; if u takes a small value or even 0, then u is a low frequency and v is a high frequency. The values ​​of u and v in this embodiment are shown in Table 1.

[0062] Table 1

[0063]

[0064] In the table, N represents the number of feature blocks divided in step 2.1. 1, 2, 3, and 4 represent the first FCMM, the second FCMM, the third FCMM, and the fourth FCMM, respectively.

[0065] Step 2.3: Sum the results obtained from all pixels in each feature block to obtain a scalar representing that feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks to be divided for each feature in the channel dimension when performing the partitioning operation;

[0066]

[0067] This represents the value obtained by summing all the scalar results after assigning frequency information to each pixel in each channel block. Let (h, w) represent the features of the (h, w) pixel obtained from the j-th pooling layer in the i-th encoder; i = 1, 2; j = 1, 2, 3, 4. Compared to traditional global max pooling or global average pooling operations, this method can retain more information.

[0068] Step 2.4: Collect all the obtained identifiers used to identify the corresponding feature blocks. By piecing them together, we get Freq:

[0069]

[0070] Freq = Cat(Freq1, Freq2);

[0071] Where Cat represents the splicing operation;

[0072] Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j The specific process is as follows: ω j =Sigmoid(fc(Freq)), where fc represents a fully connected layer;

[0073] Step 2.6: Fuse the input features based on the learned weights: ω j Assigned to P1 j , 1-ω j Assigned to The results are then summed and passed through a convolutional layer to obtain the fused feature P. fuse The specific formula is as follows:

[0074] Where conv represents a convolutional layer;

[0075] Step 2.7: In order to capture the context information in the fused features obtained by the previous FCMM module, it is necessary to determine whether there is a result from the previous FCMM module, that is, to determine whether the current FCMM module is the first FCMM module.

[0076] If not, the output features of the previous FCMM module are downsampled, then the fused features are concatenated with the downsampled features, and finally the concatenated features are convolved to obtain the output of the FCMM module; represented as:

[0077] M j =conv(Cat(P) fuse DownUpsample(M j-1 )))

[0078] Where DownUpsample represents the downsampling operation;

[0079] If so, the fused result will be the output of the FCMM module:

[0080] M1 = P fuse

[0081] M j Alternatively, M1 can be used as the output of the FCMM module.

[0082] Step 3: Train the multimodal brain tumor segmentation network model using the training set; feed the two sets of inputs into two encoders respectively, extract features through the encoders, and then obtain features of the same size as the input modality through the decoders. Calculate the loss between the feature maps obtained from the decoders and the labels to supervise the entire feature extraction process; the loss functions obtained from the two decoders are denoted as follows: and

[0083] Calculate the loss between the prediction results of the entire network model and the labels, and obtain...

[0084] Will and The three loss functions are added together to obtain the total loss function; the total loss function is then used for backpropagation to update the network model parameters and obtain the optimal network model.

[0085] In this embodiment, and All three loss functions are calculated by adding the Dice loss and the cross-entropy loss; the specific calculation formula is as follows:

[0086]

[0087] L stream =L Dice +L CE

[0088] Where y i and y′ i These represent the label value and the predicted value, respectively. N represents the total number of pixels, and p(.) represents the softmax function.

[0089] Step 4: Apply the trained network model to multimodal brain tumor segmentation. The extracted T1-T1ce and T2-Flair modalities are input into two encoders respectively. After processing by the encoders, they are further processed by the FCMM module. Finally, the prediction results are obtained through multiple upsampling layers and convolutional layers. The decoder is mainly used to calculate the loss function and supervise model training during the training process. In practical applications, the network can retain only the encoder and FCMM module. When applying the model to actual multimodal brain tumor segmentation, it is possible to choose whether to retain the decoder (retaining the decoder allows for subsequent model updates, while not retaining the decoder simplifies the network structure).

[0090] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multimodal brain tumor segmentation method based on frequency domain channel attention, characterized in that, Includes the following steps: Step 1: Obtain the training dataset: Use a publicly available brain tumor segmentation dataset as the training set. All images in the dataset are labeled. Use the T1-T1ce modalities in the training set as one set of inputs and the T2-Flair modalities as another set of inputs. Step 2: Establish a multimodal brain tumor segmentation network, including two encoders and two decoders, with each encoder connected to one decoder. Each encoder includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fourth pooling layer connected in sequence. Each decoder includes a fifth convolutional layer, a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a sixth convolutional layer connected in sequence. The fourth pooling layer is connected to the fifth convolutional layer. The outputs of the first pooling layers of the two encoders are connected to the first FCMM module, the outputs of the second pooling layers of the two encoders are connected to the second FCMM module, the outputs of the third pooling layers of the two encoders are connected to the third FCMM module, and the outputs of the fourth pooling layers of the two encoders are connected to the fourth FCMM module. The first FCMM module, the second FCMM module, the third FCMM module, and the fourth FCMM module are connected sequentially. The output of the fourth FCMM module is sequentially connected to the fifth, sixth, seventh, and eighth upsampling layers. The output features of the eighth upsampling layer are concatenated with the output features of the first convolutional layer in the two encoders, and then input into the seventh convolutional layer to generate the prediction result. The outputs of the first convolutional layer and the fourth upsampling layer are connected in a skip connection; the outputs of the second convolutional layer and the third upsampling layer are connected in a skip connection; the outputs of the third convolutional layer and the second upsampling layer are connected in a skip connection; the outputs of the fourth convolutional layer and the first upsampling layer are connected in a skip connection. In the FCMM module, data processing includes the following steps: Step 2.1: Denote the features obtained after processing the two sets of modes through the convolutional layer as follows: and j represents the j-th convolutional layer, j = 1, 2, 3, 4; the pooling layer features obtained after the pooling layer are denoted as... and Pooling layer features and Perform a partitioning operation on the channel dimension to obtain a set consisting of two feature blocks; Step 2.2: For each pixel in the partitioned feature block set, use discrete cosine transform to assign high-frequency and low-frequency components, and calculate the scalar. This represents the scalar value calculated based on the frequency components u and v at pixel point (h,w); Step 2.3: Sum the results obtained from all pixels in each feature block to obtain a scalar representing that feature block. Where i∈{1,2} represents the branch of the feature extraction flow; n∈{0,1,…,N-1} represents the number of blocks to be divided for each feature in the channel dimension when performing the partitioning operation; This represents the value obtained by summing all the scalar results after assigning frequency information to each pixel in each channel block. Step 2.4: Collect all the obtained identifiers used to identify the corresponding feature blocks. By piecing them together, we get Freq: Freq = Cat(Freq1, Freq2); Where Cat represents the splicing operation; Step 2.5: Input the obtained Freq into the fully connected layer for learning, and obtain the weight ω corresponding to the current layer through the Sigmoid activation function. j The specific process is as follows: ω j =Sigmoid(fc(Freq)), where fc represents a fully connected layer; Step 2.6: Fuse the input features based on the learned weights: ω j Assigned to 1-ω j Assigned to The results are then summed and passed through a convolutional layer to obtain the fused feature P. fuse The specific formula is as follows: Where conv represents a convolutional layer; Step 2.7: Determine if the current FCMM module is the first FCMM module; if not, downsample the output features of the previous FCMM module, then concatenate the fused features with the downsampled features, and then convolve the concatenated features to obtain the output of the current FCMM module; represented as: M j =conv(Cat(P fuse ,DownUpsample(M j-1 ))) Where DownUpsample represents the downsampling operation; If so, the fused result will be the output of the FCMM module: M1=P fuse ; Step 3: Train the multimodal brain tumor segmentation network model using the training set; feed the two sets of inputs into two encoders respectively, extract features through the encoders, and then pass them through two decoders to obtain features of the same size as the input modalities. Calculate the loss with the labels, and denote the loss functions obtained from the two decoders as follows: and Step 4: Calculate the loss between the prediction result and the label, and obtain... Will and The three loss functions are added together to obtain the total loss function, and then backpropagation is performed to update the network model parameters.

Images